Merge pull request #267 from jamiusaliu/test/slcc-real-world-false-positives

test: pin the SLCC1/SLCC2 false positive from #208, and document whole-word matching

FAQ.md

@@ -32,6 +32,16 @@ Yes, provided the crawlers identify themselves and your application/hosting supp
 
 Some crawlers — [such as Perplexity](https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/) — do not identify themselves via their user agent strings and, as such, are difficult to block.
 
+## Can I use `robots.json` directly in my own tooling?
+
+You're welcome to, with a caveat. `robots.json` isn't intended as a primary
+deliverable of this project — the generated configuration files are. If you consume
+it yourself, note that the agent names should be matched as whole words rather than
+substrings.
+
+The generated configs wrap items in `\b(...)\b` for this reason. Without
+word boundaries, agent names match inside unrelated strings (see [issue 208](https://github.com/ai-robots-txt/ai.robots.txt/issues/208) for an example).
+
 ## What can we do if a bot doesn't respect `robots.txt`?
 
 That depends on your stack.

code/tests.py

@@ -126,6 +126,13 @@ class TestUserAgentPatternGeneration(unittest.TestCase):
             "NotAmazonbot/1.0",
             "NotApplebot/1.0",
             "NotBytespider/1.0",
+            # Real-world user agents that embed a listed bot name mid-string.
+            # These older Internet Explorer / Trident agents contain "SLCC1" or
+            # "SLCC2" (a Windows licensing component), which spans the listed
+            # agent "LCC". Reported in #208 and fixed by the word boundaries
+            # added in #260; pinned here so the specific report cannot regress.
+            "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30729)",
+            "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; Media Center PC 6.0)",
         ]
 
         for ua in non_ai_user_agents: