remote/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2026-09-21 15:34:06 +02:00
Code Issues Projects Releases Packages Wiki Activity
668 commits 7 branches 54 tags 2.3 MiB
Commit graph

157 commits

Author SHA1 Message Date
Dhruv Maniya
49669178ef fix: configure exact user-agent matching 
Signed-off-by: Dhruv Maniya <dhruvmaniya1998@gmail.com>
 2026-07-29 21:47:42 +05:30
Known Agents
a0fed45f0e Update from knownagents.com 2026-07-10 02:03:20 +00:00
Glyn Normington
51f7fd5e3a
 		Merge pull request #240 from bobmatyas/add/tabstack 
Add: Mozilla-Tabstack
 2026-07-09 16:00:59 +01:00
Known Agents
a71b27a232 Update from knownagents.com 2026-07-09 02:04:28 +00:00
bob
c793fa0a8c Add: Mozilla-Tabstack. Closes #226 2026-07-08 21:16:52 -04:00
Gavin King
dad7db5168 Fixed broken URLs 2026-07-08 20:38:48 -04:00
Gavin King
0c5de5b90b Updating the name 2026-07-08 15:30:25 -04:00
dark-visitors
a394130efd Update from Dark Visitors 2026-06-26 02:32:55 +00:00
dark-visitors
5cb3ec765f Update from Dark Visitors 2026-06-17 11:21:12 +00:00
dark-visitors
118bb772c0 Update from Dark Visitors 2026-06-17 11:01:38 +00:00
dark-visitors
3091ad0a23 Update from Dark Visitors 2026-06-16 02:52:42 +00:00
dark-visitors
f420408eee Update from Dark Visitors 2026-06-04 02:48:49 +00:00
Justin Ribeiro
7673437733
 		add: AgentTimes bot 2026-06-02 11:26:06 -07:00
dark-visitors
ee3a96ee5c Update from Dark Visitors 2026-05-22 02:33:30 +00:00
dark-visitors
aea7db9e34 Update from Dark Visitors 2026-05-13 02:24:20 +00:00
dark-visitors
851ce068cf Update from Dark Visitors 2026-05-03 02:01:44 +00:00
Georg Sauthoff
a9f686064f Add NagetBot and newsai user-agents 
User-agent header values seen in the wild over the last days:

    Mozilla/5.0 (compatible; NagetBot/1.0; +https://naget.ai/bot)
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 newsai/1.0 Safari/537.36
 2026-05-02 13:04:51 +02:00
dark-visitors
ad47cafc97 Update from Dark Visitors 2026-03-26 01:26:23 +00:00
dark-visitors
2d2ec4ce8d Update from Dark Visitors 2026-03-25 01:20:42 +00:00
dark-visitors
3de8865f00 Update from Dark Visitors 2026-03-06 01:21:01 +00:00
dark-visitors
a762015e58 Update from Dark Visitors 2026-02-13 01:23:17 +00:00
dark-visitors
83485effdb Update from Dark Visitors 2025-12-20 00:58:49 +00:00
dark-visitors
f1c752ef12 Update from Dark Visitors 2025-12-05 01:00:44 +00:00
dark-visitors
45b071b29f Update from Dark Visitors 2025-12-04 01:00:27 +00:00
dark-visitors
f61b3496f7 Update from Dark Visitors 2025-12-03 01:00:32 +00:00
dark-visitors
8363d4fdd4 Update from Dark Visitors 2025-12-02 01:25:24 +00:00
Adam Newbold
f5d7ccb243
 		Fixed invalid JSON 2025-11-28 14:10:38 -05:00
Adam Newbold
30d719a09a
 		Adding LAIONDownloader 2025-11-28 14:08:18 -05:00
Glyn Normington
ab24e41106 Clarify that DeepSeekBot does not respect robots.txt 
Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/198
 2025-11-26 12:02:42 +00:00
Glyn Normington
efb4d260da delete extraneous hyphen 
Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/193
 2025-11-20 10:23:48 +00:00
Glyn Normington
663d030f96
 		Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo 
chore: adds NotebookLM and KlaviyoAIBot agents
 2025-11-14 21:01:13 +00:00
Cory Dransfeldt
28b45ea08d
 		chore: adds NotebookLM and KlaviyoAIBot agents 2025-11-14 11:03:03 -08:00
Glyn Normington
60b6a0829d
 		Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots 
chore(robots.json): add AmazonBuyForMe and atlassian-bot
 2025-11-05 23:50:48 +00:00
Cory Dransfeldt
00bf2b0e13
 		chore(robots.json): add AmazonBuyForMe and atlassian-bot 2025-11-05 14:02:37 -08:00
Cory Dransfeldt
808451055c
 		chore: adds BuddyBot 2025-11-03 13:39:07 -08:00
Cory Dransfeldt
192b0a2eef
 		chore: adds Linguee Bot 2025-10-24 10:35:04 -07:00
dark-visitors
97e19445ce Update from Dark Visitors 2025-10-24 00:53:30 +00:00
Glyn Normington
b89a9eae6a Use third party in text, rather than first 2025-10-21 01:11:32 +01:00
Glyn Normington
d22b8dfd7b
 		Description is third party, not first 2025-10-21 01:08:42 +01:00
Cory Dransfeldt
19c1d346c3
 		chore: adds IbouBot 2025-10-20 09:28:31 -07:00
Glyn Normington
0874a92503
 		Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare 
chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot
 2025-10-20 15:34:30 +01:00
Cory Dransfeldt
28d2d09633
 		chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot 2025-10-19 15:57:28 -07:00
dark-visitors
91bf905fa9 Update from Dark Visitors 2025-10-15 00:56:30 +00:00
dark-visitors
56d03d46fb Update from Dark Visitors 2025-09-26 00:54:30 +00:00
Cory Dransfeldt
bf347bdf91
 		chore(robots.json): add DeepSeekBot 2025-09-24 09:42:01 -07:00
dark-visitors
2fd93029ca Update from Dark Visitors 2025-09-11 00:55:06 +00:00
Simon Ser
50870ba911 Add TerraCotta 
Example server log:

    X.X.X.X - - [10/Sep/2025:00:06:17 +0000] "GET /archives/dri-devel/2023-November/430875.html HTTP/1.1" 200 7388 "-" "TerraCotta https://github.com/CeramicTeam/CeramicTerracotta"
 2025-09-10 12:04:08 +02:00
dark-visitors
dd391bf960 Update from Dark Visitors 2025-09-10 00:53:56 +00:00
László Károlyi
ec508ab434
 		Update Brightbot operator and details; add meta-webindexer entry 
- Update Brightbot operator to https://brightdata.com/brightbot.
- Change Brightbot frequency to "At least one per minute."
- Expand Brightbot description with disguise tactics link.
- Add new entry for meta-webindexer under Meta operator.
- Set meta-webindexer respect to "Unclear at this time."
- Define meta-webindexer function as "AI Assistants."
- Set meta-webindexer frequency to "Unhinged, more than 1 per second."
- Include meta-webindexer description on improving Meta AI search.
 2025-09-08 21:42:06 +02:00
dark-visitors
0ed29412c9 Update from Dark Visitors 2025-08-28 00:55:47 +00:00