test: pin the SLCC1/SLCC2 false positive from #208

The non-AI user agent list covers synthetic prefix/suffix variants
(NotCursor, CursorNot). It had no real-world agent that embeds a listed
name mid-string, which is what #208 actually reported: SLCC1 and SLCC2 in
older Internet Explorer and Trident agents span the listed agent LCC.

Verified the case is load-bearing: removing the word boundaries from
list_to_pcre fails this test with
  AssertionError: <re.Match object; span=(65, 68), match='LCC'> is not None

Also document in the FAQ that agent names are matched as whole words, for
anyone consuming robots.json directly and writing their own matcher.
This commit is contained in:
Saliu Jamiu Olamilekan 2026-08-18 20:28:19 -07:00
commit bcc4e408e2
2 changed files with 22 additions and 0 deletions

15
FAQ.md
View file

@ -32,6 +32,21 @@ Yes, provided the crawlers identify themselves and your application/hosting supp
Some crawlers — [such as Perplexity](https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/) — do not identify themselves via their user agent strings and, as such, are difficult to block.
## Can I use `robots.json` directly in my own tooling?
You're welcome to, with a caveat. `robots.json` isn't intended as a primary
deliverable of this project — the generated configuration files are. If you consume
it yourself, note that **the agent names are matched as whole words**, not as
substrings.
The generated configs wrap the list in `\b(...)\b` for exactly this reason. Without
word boundaries, short agent names match inside unrelated strings: the listed agent
`LCC` appears inside `SLCC1`, a Windows licensing component present in the user
agent of older Internet Explorer and Trident-based browsers. A naive substring match
would block those real visitors ([#208](https://github.com/ai-robots-txt/ai.robots.txt/issues/208)).
If you build your own matcher from `robots.json`, use word-boundary matching.
## What can we do if a bot doesn't respect `robots.txt`?
That depends on your stack.

View file

@ -126,6 +126,13 @@ class TestUserAgentPatternGeneration(unittest.TestCase):
"NotAmazonbot/1.0",
"NotApplebot/1.0",
"NotBytespider/1.0",
# Real-world user agents that embed a listed bot name mid-string.
# These older Internet Explorer / Trident agents contain "SLCC1" or
# "SLCC2" (a Windows licensing component), which spans the listed
# agent "LCC". Reported in #208 and fixed by the word boundaries
# added in #260; pinned here so the specific report cannot regress.
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30729)",
"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; Media Center PC 6.0)",
]
for ua in non_ai_user_agents: