mirror of
https://github.com/ai-robots-txt/ai.robots.txt.git
synced 2026-08-30 04:44:06 +02:00
bcc4e408e2
The non-AI user agent list covers synthetic prefix/suffix variants (NotCursor, CursorNot). It had no real-world agent that embeds a listed name mid-string, which is what #208 actually reported: SLCC1 and SLCC2 in older Internet Explorer and Trident agents span the listed agent LCC. Verified the case is load-bearing: removing the word boundaries from list_to_pcre fails this test with AssertionError: <re.Match object; span=(65, 68), match='LCC'> is not None Also document in the FAQ that agent names are matched as whole words, for anyone consuming robots.json directly and writing their own matcher. |
||
|---|---|---|
| .. | ||
| test_files | ||
| robots.py | ||
| tests.py | ||