remote/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2026-08-31 05:14:06 +02:00
Code Issues Projects Releases Packages Wiki Activity
668 commits 7 branches 53 tags 2 MiB
Commit graph

668 commits

This branch
This branch
All branches
Author SHA1 Message Date
Glyn Normington
3db35be22d
 		Merge pull request #241 from iamdhrv/fix/exact-spider-user-agent v1.48 
Anchor generic Spider user-agent match
 2026-07-30 12:15:55 +01:00
Dhruv Maniya
49669178ef fix: configure exact user-agent matching 
Signed-off-by: Dhruv Maniya <dhruvmaniya1998@gmail.com>
 2026-07-29 21:47:42 +05:30
Dhruv Maniya
af31418609 Anchor the generic Spider user agent 
Signed-off-by: Dhruv Maniya <dhruvmaniya1998@gmail.com>
 2026-07-15 13:04:57 +05:30
Known Agents
a0fed45f0e Update from knownagents.com 2026-07-10 02:03:20 +00:00
ai.robots.txt
2a83168bbb Merge pull request #240 from bobmatyas/add/tabstack 
Add: Mozilla-Tabstack
 2026-07-09 15:01:12 +00:00
Glyn Normington
51f7fd5e3a
 		Merge pull request #240 from bobmatyas/add/tabstack 
Add: Mozilla-Tabstack
 2026-07-09 16:00:59 +01:00
Known Agents
a71b27a232 Update from knownagents.com 2026-07-09 02:04:28 +00:00
bob
c793fa0a8c Add: Mozilla-Tabstack. Closes #226 2026-07-08 21:16:52 -04:00
Glyn Normington
f030cbe956
 		Merge pull request #238 from ghking/rename-dark-visitors-to-known-agents v1.47 
Renaming Dark Visitors to Known Agents
 2026-07-09 01:41:52 +01:00
Glyn Normington
12a15a3a28
 		Update code/robots.py 
For consistency.
 2026-07-09 01:40:00 +01:00
Glyn Normington
5065ca6055
 		Update .github/workflows/ai_robots_update.yml 
For consistency.
 2026-07-09 01:39:50 +01:00
Glyn Normington
0b021b9e40
 		Update code/robots.py 
For consistency.
 2026-07-09 01:39:39 +01:00
Gavin King
dad7db5168 Fixed broken URLs 2026-07-08 20:38:48 -04:00
Gavin King
e1bf273b16 Update robots.py 2026-07-08 20:26:48 -04:00
Gavin King
a35e6f9a09 Update ai_robots_update.yml 2026-07-08 20:23:53 -04:00
Gavin King
c43c5afd07 More name changes 2026-07-08 15:43:07 -04:00
Gavin King
0c5de5b90b Updating the name 2026-07-08 15:30:25 -04:00
dark-visitors
a394130efd Update from Dark Visitors 2026-06-26 02:32:55 +00:00
Glyn Normington
ee4ae068c2
 		Merge pull request #236 from guest20/patch-2 
robots.py: fix darkvisiters descriptions and links
 2026-06-23 11:15:59 +01:00
Glyn Normington
82df5f7659
 		Apply suggestions from code review 
Co-authored-by: Glyn Normington <work@underlap.org>
 2026-06-23 11:13:17 +01:00
guest20
ac207f9947
 		tests.py: ) 2026-06-23 04:17:14 +02:00
guest20
ffad9293c4
 		tests.py: consolidate 2026-06-23 04:13:17 +02:00
guest20
80357baf9c
 		robots.py: pass everything to consolodate 2026-06-21 23:51:32 +02:00
guest20
7dbe93295e
 		robots.py: pull consolodate out 2026-06-21 23:47:11 +02:00
guest20
3f965b292d
 		robots.py: keep new, non-default values 
Co-authored-by: Glyn Normington <work@underlap.org>
 2026-06-20 18:29:47 +02:00
dark-visitors
5cb3ec765f Update from Dark Visitors 2026-06-17 11:21:12 +00:00
guest20
25415e342c
 		robots.py: div class=description 2026-06-17 13:20:29 +02:00
dark-visitors
118bb772c0 Update from Dark Visitors 2026-06-17 11:01:38 +00:00
guest20
8dd8e6c4e4
 		robots.py: possibly ruin the whole purpose of consolodate 
Replace an old non-default value with a new non-default value, so updates happen.
 2026-06-17 13:00:15 +02:00
guest20
78fd7b49d8
 		\robots.py 2026-06-17 12:23:10 +02:00
guest20
101d7ac0ad
 		robots.py: punked by auto-) again 2026-06-17 12:10:05 +02:00
guest20
f2147fd81c
 		robots.py: fix darkvistors link 2026-06-17 12:06:50 +02:00
guest20
bd3ee4702c
 		robots.py: agents/agents -> agents 2026-06-17 11:45:55 +02:00
dark-visitors
3091ad0a23 Update from Dark Visitors 2026-06-16 02:52:42 +00:00
dark-visitors
f420408eee Update from Dark Visitors 2026-06-04 02:48:49 +00:00
ai.robots.txt
000552c2df Merge pull request #232 from justinribeiro/main 
add: AgentTimes bot
 2026-06-03 08:56:38 +00:00
Glyn Normington
ef50defc1a
 		Merge pull request #232 from justinribeiro/main 
add: AgentTimes bot
 2026-06-03 09:56:26 +01:00
Justin Ribeiro
7673437733
 		add: AgentTimes bot 2026-06-02 11:26:06 -07:00
dark-visitors
ee3a96ee5c Update from Dark Visitors 2026-05-22 02:33:30 +00:00
Glyn Normington
1fbf7a06ac
 		Merge pull request #231 from cfcryptotrader-rgb/cfcryptotrader-sync-generated-robots v1.46 
Keep generated robot files synced after Dark Visitors update
 2026-05-13 07:44:48 +01:00
cfcryptotrader-rgb
8d5a081a16 Keep generated robot files in dark visitors update 2026-05-13 00:44:06 -03:00
dark-visitors
aea7db9e34 Update from Dark Visitors 2026-05-13 02:24:20 +00:00
Glyn Normington
767fcc6f15
 		Merge pull request #229 from axeleroy/main 
Add missing bot categories
 2026-05-12 04:33:19 +01:00
Axel Leroy
6adc1f7b53
 		Add missing bot categories 2026-05-11 15:42:56 +02:00
dark-visitors
851ce068cf Update from Dark Visitors 2026-05-03 02:01:44 +00:00
ai.robots.txt
8b0bc322f4 Merge pull request #227 from gsauthof/naget-news 
Add NagetBot and newsai user-agents
 2026-05-02 15:05:09 +00:00
Glyn Normington
7244a7e870
 		Merge pull request #227 from gsauthof/naget-news 
Add NagetBot and newsai user-agents
 2026-05-02 16:04:59 +01:00
Georg Sauthoff
a9f686064f Add NagetBot and newsai user-agents 
User-agent header values seen in the wild over the last days:

    Mozilla/5.0 (compatible; NagetBot/1.0; +https://naget.ai/bot)
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 newsai/1.0 Safari/537.36
 2026-05-02 13:04:51 +02:00
ai.robots.txt
198653b59a Update from Dark Visitors 2026-03-27 01:26:26 +00:00
dark-visitors
ad47cafc97 Update from Dark Visitors 2026-03-26 01:26:23 +00:00