remote/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2026-08-31 13:24:05 +02:00
Code Issues Projects Releases Packages Wiki Activity
717 commits 7 branches 53 tags 2 MiB
Commit graph

75 commits

Author SHA1 Message Date
Saliu Jamiu Olamilekan
bcc4e408e2 test: pin the SLCC1/SLCC2 false positive from #208 
The non-AI user agent list covers synthetic prefix/suffix variants
(NotCursor, CursorNot). It had no real-world agent that embeds a listed
name mid-string, which is what #208 actually reported: SLCC1 and SLCC2 in
older Internet Explorer and Trident agents span the listed agent LCC.

Verified the case is load-bearing: removing the word boundaries from
list_to_pcre fails this test with
  AssertionError: <re.Match object; span=(65, 68), match='LCC'> is not None

Also document in the FAQ that agent names are matched as whole words, for
anyone consuming robots.json directly and writing their own matcher.
 2026-08-18 20:28:19 -07:00
Ilyan321
4c9331ebbb Merge branch 'main' of https://github.com/ai-robots-txt/ai.robots.txt into fix/issue-257-regex-anchoring 2026-08-05 21:56:22 +05:00
girst
1177eba00a fix lighttpd config for 1.x 
single quotes only allowed in 2.x. follow-up to a94b2c9.
 2026-08-05 14:48:34 +02:00
Ilyan321
5658a6d726 test(code): expand false positive integration test coverage 2026-08-05 17:06:50 +05:00
Ilyan321
51a49494aa fix(code): use word boundaries \b in list_to_pcre to prevent false positive matches like NotCursor 2026-08-05 17:02:56 +05:00
Ilyan321
26510d4cd5 Merge branch 'main' of https://github.com/ai-robots-txt/ai.robots.txt into fix/issue-257-regex-anchoring 2026-08-05 16:58:22 +05:00
Ilyan321
7b1026e69c test(code): add integration test verifying regex does not match non-AI user agents 2026-08-05 16:50:35 +05:00
Ilyan321
b06780542b fix(code): unanchor PCRE patterns to allow matching real user-agent headers (#257) 2026-08-04 23:14:02 +05:00
girst
0beb4a2666 update test files to match new output 2026-08-04 19:36:25 +02:00
girst
a94b2c93e5 further minify regexps 
- nginx and lighttpd allow single quoted strings, so use python's
  built-in methods to escape quotes
- only apache requires parentheses on the outside (and only for obscure
  reasons)
- caddy expects double quoted strings. at least escape inner quotes
  directly (re.escape does not do this since python 3.7), should any
  appear
 2026-08-04 14:48:03 +02:00
girst
3bd200ba32 make nginx matches case-sensitive 
fixes #225.
 2026-08-04 14:18:17 +02:00
girst
c4b366740e do not needlessly escape hypens 
partially fixes #225.
 2026-08-04 14:13:52 +02:00
guest20
265ea03066
 		robots.py: duplciate parser = argparse.ArgumentParser(...) 
Two is just greedy
 2026-08-04 10:24:30 +02:00
Dhruv Maniya
49669178ef fix: configure exact user-agent matching 
Signed-off-by: Dhruv Maniya <dhruvmaniya1998@gmail.com>
 2026-07-29 21:47:42 +05:30
Dhruv Maniya
af31418609 Anchor the generic Spider user agent 
Signed-off-by: Dhruv Maniya <dhruvmaniya1998@gmail.com>
 2026-07-15 13:04:57 +05:30
Glyn Normington
12a15a3a28
 		Update code/robots.py 
For consistency.
 2026-07-09 01:40:00 +01:00
Glyn Normington
0b021b9e40
 		Update code/robots.py 
For consistency.
 2026-07-09 01:39:39 +01:00
Gavin King
dad7db5168 Fixed broken URLs 2026-07-08 20:38:48 -04:00
Gavin King
e1bf273b16 Update robots.py 2026-07-08 20:26:48 -04:00
Gavin King
c43c5afd07 More name changes 2026-07-08 15:43:07 -04:00
Gavin King
0c5de5b90b Updating the name 2026-07-08 15:30:25 -04:00
Glyn Normington
82df5f7659
 		Apply suggestions from code review 
Co-authored-by: Glyn Normington <work@underlap.org>
 2026-06-23 11:13:17 +01:00
guest20
ac207f9947
 		tests.py: ) 2026-06-23 04:17:14 +02:00
guest20
ffad9293c4
 		tests.py: consolidate 2026-06-23 04:13:17 +02:00
guest20
80357baf9c
 		robots.py: pass everything to consolodate 2026-06-21 23:51:32 +02:00
guest20
7dbe93295e
 		robots.py: pull consolodate out 2026-06-21 23:47:11 +02:00
guest20
3f965b292d
 		robots.py: keep new, non-default values 
Co-authored-by: Glyn Normington <work@underlap.org>
 2026-06-20 18:29:47 +02:00
guest20
25415e342c
 		robots.py: div class=description 2026-06-17 13:20:29 +02:00
guest20
8dd8e6c4e4
 		robots.py: possibly ruin the whole purpose of consolodate 
Replace an old non-default value with a new non-default value, so updates happen.
 2026-06-17 13:00:15 +02:00
guest20
78fd7b49d8
 		\robots.py 2026-06-17 12:23:10 +02:00
guest20
101d7ac0ad
 		robots.py: punked by auto-) again 2026-06-17 12:10:05 +02:00
guest20
f2147fd81c
 		robots.py: fix darkvistors link 2026-06-17 12:06:50 +02:00
guest20
bd3ee4702c
 		robots.py: agents/agents -> agents 2026-06-17 11:45:55 +02:00
Axel Leroy
6adc1f7b53
 		Add missing bot categories 2026-05-11 15:42:56 +02:00
Flynn Marquardt
790bff1817 Add reference file for lighttpd test. 2026-02-15 11:18:35 +01:00
Flynn Marquardt
742f9b9b5e Add lighttpd test. 2026-02-15 11:17:54 +01:00
Flynn Marquardt
70e83e6056 Add generation and output of lighttpd configuration fragment. 2026-02-13 19:18:46 +01:00
Adam Newbold
6d75f3c1c9
 		Update robots.py to address error on line 57 
Attempting to work around an error that prevents parsing the Dark Visitors site
 2025-12-01 20:18:29 -05:00
fiskhandlarn
a6cf6b204b test: update test nginx conf 2025-11-25 16:47:04 +01:00
fiskhandlarn
2679fcad34 feat: update nginx generator 2025-11-25 16:46:31 +01:00
Sebastian Davids
c7407e721d
 		Simplify htaccess rewrite rule 
https://httpd.apache.org/docs/2.4/rewrite/flags.html#flag_f

fix #159

Signed-off-by: Sebastian Davids <sdavids@gmx.de>
 2025-07-08 12:33:24 +02:00
Glyn Normington
ec995cd686 Fix Python syntax error 2025-05-01 11:27:40 +01:00
Crazyroostereye
1310dbae46
 		Added a Caddyfile converter (#110) 
Co-authored-by: Julian Beittel <julian@beittel.net>
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-05-01 11:21:32 +01:00
Rik Wijnen
a4a9f2ac2b Tests for HAProxy file added. 2025-04-28 09:30:26 +02:00
Rik Wijnen
50e739dd73 HAProxy converter added. 2025-04-28 08:51:02 +02:00
Joshua Sheard
d9f882a9b2
 		Include "AI Agents" from Dark Visitors 2025-04-14 15:46:01 +01:00
Frederic Barthelemy
c6f308cbd0
 		PR Feedback: log special-case, comment consistency 2025-04-05 09:01:52 -07:00
Frederic Barthelemy
5f5a89c38c
 		Fix html-mangled hyphen in Perplexity-Users 
Fixes: #99
 2025-04-04 17:34:14 -07:00
Frederic Barthelemy
6b0349f37d
 		fix python complaining about f-string syntax 
```
python code/tests.py
Traceback (most recent call last):
  File "/Users/fbarthelemy/Code/ai.robots.txt/code/tests.py", line 7, in <module>
    from robots import json_to_txt, json_to_table, json_to_htaccess, json_to_nginx
  File "/Users/fbarthelemy/Code/ai.robots.txt/code/robots.py", line 144
    return f"({"|".join(map(re.escape, lst))})"
                ^
SyntaxError: f-string: expecting '}'
```
 2025-04-04 15:20:30 -07:00
Thomas Leister
7c3b5a2cb2
 		Add tests for Nginx config generator 2025-03-27 18:28:21 +01:00