From bcc4e408e2125b38d31492dff300af29b3fb8055 Mon Sep 17 00:00:00 2001 From: Saliu Jamiu Olamilekan Date: Tue, 18 Aug 2026 20:28:19 -0700 Subject: [PATCH] test: pin the SLCC1/SLCC2 false positive from #208 The non-AI user agent list covers synthetic prefix/suffix variants (NotCursor, CursorNot). It had no real-world agent that embeds a listed name mid-string, which is what #208 actually reported: SLCC1 and SLCC2 in older Internet Explorer and Trident agents span the listed agent LCC. Verified the case is load-bearing: removing the word boundaries from list_to_pcre fails this test with AssertionError: is not None Also document in the FAQ that agent names are matched as whole words, for anyone consuming robots.json directly and writing their own matcher. --- FAQ.md | 15 +++++++++++++++ code/tests.py | 7 +++++++ 2 files changed, 22 insertions(+) diff --git a/FAQ.md b/FAQ.md index 7264819..0c11841 100644 --- a/FAQ.md +++ b/FAQ.md @@ -32,6 +32,21 @@ Yes, provided the crawlers identify themselves and your application/hosting supp Some crawlers — [such as Perplexity](https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/) — do not identify themselves via their user agent strings and, as such, are difficult to block. +## Can I use `robots.json` directly in my own tooling? + +You're welcome to, with a caveat. `robots.json` isn't intended as a primary +deliverable of this project — the generated configuration files are. If you consume +it yourself, note that **the agent names are matched as whole words**, not as +substrings. + +The generated configs wrap the list in `\b(...)\b` for exactly this reason. Without +word boundaries, short agent names match inside unrelated strings: the listed agent +`LCC` appears inside `SLCC1`, a Windows licensing component present in the user +agent of older Internet Explorer and Trident-based browsers. A naive substring match +would block those real visitors ([#208](https://github.com/ai-robots-txt/ai.robots.txt/issues/208)). + +If you build your own matcher from `robots.json`, use word-boundary matching. + ## What can we do if a bot doesn't respect `robots.txt`? That depends on your stack. diff --git a/code/tests.py b/code/tests.py index 5ca0842..0006f5d 100755 --- a/code/tests.py +++ b/code/tests.py @@ -126,6 +126,13 @@ class TestUserAgentPatternGeneration(unittest.TestCase): "NotAmazonbot/1.0", "NotApplebot/1.0", "NotBytespider/1.0", + # Real-world user agents that embed a listed bot name mid-string. + # These older Internet Explorer / Trident agents contain "SLCC1" or + # "SLCC2" (a Windows licensing component), which spans the listed + # agent "LCC". Reported in #208 and fixed by the word boundaries + # added in #260; pinned here so the specific report cannot regress. + "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30729)", + "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; Media Center PC 6.0)", ] for ua in non_ai_user_agents: