test: pin the SLCC1/SLCC2 false positive from #208

The non-AI user agent list covers synthetic prefix/suffix variants
(NotCursor, CursorNot). It had no real-world agent that embeds a listed
name mid-string, which is what #208 actually reported: SLCC1 and SLCC2 in
older Internet Explorer and Trident agents span the listed agent LCC.

Verified the case is load-bearing: removing the word boundaries from
list_to_pcre fails this test with
  AssertionError: <re.Match object; span=(65, 68), match='LCC'> is not None

Also document in the FAQ that agent names are matched as whole words, for
anyone consuming robots.json directly and writing their own matcher.
This commit is contained in:
Saliu Jamiu Olamilekan 2026-08-18 20:28:19 -07:00
commit bcc4e408e2
2 changed files with 22 additions and 0 deletions

15
FAQ.md
View file

@ -32,6 +32,21 @@ Yes, provided the crawlers identify themselves and your application/hosting supp
Some crawlers — [such as Perplexity](https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/) — do not identify themselves via their user agent strings and, as such, are difficult to block.
## Can I use `robots.json` directly in my own tooling?
You're welcome to, with a caveat. `robots.json` isn't intended as a primary
deliverable of this project — the generated configuration files are. If you consume
it yourself, note that **the agent names are matched as whole words**, not as
substrings.
The generated configs wrap the list in `\b(...)\b` for exactly this reason. Without
word boundaries, short agent names match inside unrelated strings: the listed agent
`LCC` appears inside `SLCC1`, a Windows licensing component present in the user
agent of older Internet Explorer and Trident-based browsers. A naive substring match
would block those real visitors ([#208](https://github.com/ai-robots-txt/ai.robots.txt/issues/208)).
If you build your own matcher from `robots.json`, use word-boundary matching.
## What can we do if a bot doesn't respect `robots.txt`?
That depends on your stack.