mirror of
https://github.com/ai-robots-txt/ai.robots.txt.git
synced 2026-08-29 04:24:04 +02:00
test: pin the SLCC1/SLCC2 false positive from #208
The non-AI user agent list covers synthetic prefix/suffix variants (NotCursor, CursorNot). It had no real-world agent that embeds a listed name mid-string, which is what #208 actually reported: SLCC1 and SLCC2 in older Internet Explorer and Trident agents span the listed agent LCC. Verified the case is load-bearing: removing the word boundaries from list_to_pcre fails this test with AssertionError: <re.Match object; span=(65, 68), match='LCC'> is not None Also document in the FAQ that agent names are matched as whole words, for anyone consuming robots.json directly and writing their own matcher.
This commit is contained in:
parent
6a20f8dbc4
commit
bcc4e408e2
2 changed files with 22 additions and 0 deletions
15
FAQ.md
15
FAQ.md
|
|
@ -32,6 +32,21 @@ Yes, provided the crawlers identify themselves and your application/hosting supp
|
|||
|
||||
Some crawlers — [such as Perplexity](https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/) — do not identify themselves via their user agent strings and, as such, are difficult to block.
|
||||
|
||||
## Can I use `robots.json` directly in my own tooling?
|
||||
|
||||
You're welcome to, with a caveat. `robots.json` isn't intended as a primary
|
||||
deliverable of this project — the generated configuration files are. If you consume
|
||||
it yourself, note that **the agent names are matched as whole words**, not as
|
||||
substrings.
|
||||
|
||||
The generated configs wrap the list in `\b(...)\b` for exactly this reason. Without
|
||||
word boundaries, short agent names match inside unrelated strings: the listed agent
|
||||
`LCC` appears inside `SLCC1`, a Windows licensing component present in the user
|
||||
agent of older Internet Explorer and Trident-based browsers. A naive substring match
|
||||
would block those real visitors ([#208](https://github.com/ai-robots-txt/ai.robots.txt/issues/208)).
|
||||
|
||||
If you build your own matcher from `robots.json`, use word-boundary matching.
|
||||
|
||||
## What can we do if a bot doesn't respect `robots.txt`?
|
||||
|
||||
That depends on your stack.
|
||||
|
|
|
|||
|
|
@ -126,6 +126,13 @@ class TestUserAgentPatternGeneration(unittest.TestCase):
|
|||
"NotAmazonbot/1.0",
|
||||
"NotApplebot/1.0",
|
||||
"NotBytespider/1.0",
|
||||
# Real-world user agents that embed a listed bot name mid-string.
|
||||
# These older Internet Explorer / Trident agents contain "SLCC1" or
|
||||
# "SLCC2" (a Windows licensing component), which spans the listed
|
||||
# agent "LCC". Reported in #208 and fixed by the word boundaries
|
||||
# added in #260; pinned here so the specific report cannot regress.
|
||||
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30729)",
|
||||
"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; Media Center PC 6.0)",
|
||||
]
|
||||
|
||||
for ua in non_ai_user_agents:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue