Commit graph

136 commits

Author SHA1 Message Date
dark-visitors
83485effdb Update from Dark Visitors 2025-12-20 00:58:49 +00:00
dark-visitors
f1c752ef12 Update from Dark Visitors 2025-12-05 01:00:44 +00:00
dark-visitors
45b071b29f Update from Dark Visitors 2025-12-04 01:00:27 +00:00
dark-visitors
f61b3496f7 Update from Dark Visitors 2025-12-03 01:00:32 +00:00
dark-visitors
8363d4fdd4 Update from Dark Visitors 2025-12-02 01:25:24 +00:00
Adam Newbold
f5d7ccb243
Fixed invalid JSON 2025-11-28 14:10:38 -05:00
Adam Newbold
30d719a09a
Adding LAIONDownloader 2025-11-28 14:08:18 -05:00
Glyn Normington
ab24e41106 Clarify that DeepSeekBot does not respect robots.txt
Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/198
2025-11-26 12:02:42 +00:00
Glyn Normington
efb4d260da delete extraneous hyphen
Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/193
2025-11-20 10:23:48 +00:00
Glyn Normington
663d030f96
Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo
chore: adds NotebookLM and KlaviyoAIBot agents
2025-11-14 21:01:13 +00:00
Cory Dransfeldt
28b45ea08d
chore: adds NotebookLM and KlaviyoAIBot agents 2025-11-14 11:03:03 -08:00
Glyn Normington
60b6a0829d
Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots
chore(robots.json): add AmazonBuyForMe and atlassian-bot
2025-11-05 23:50:48 +00:00
Cory Dransfeldt
00bf2b0e13
chore(robots.json): add AmazonBuyForMe and atlassian-bot 2025-11-05 14:02:37 -08:00
Cory Dransfeldt
808451055c
chore: adds BuddyBot 2025-11-03 13:39:07 -08:00
Cory Dransfeldt
192b0a2eef
chore: adds Linguee Bot 2025-10-24 10:35:04 -07:00
dark-visitors
97e19445ce Update from Dark Visitors 2025-10-24 00:53:30 +00:00
Glyn Normington
b89a9eae6a Use third party in text, rather than first 2025-10-21 01:11:32 +01:00
Glyn Normington
d22b8dfd7b
Description is third party, not first 2025-10-21 01:08:42 +01:00
Cory Dransfeldt
19c1d346c3
chore: adds IbouBot 2025-10-20 09:28:31 -07:00
Glyn Normington
0874a92503
Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare
chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot
2025-10-20 15:34:30 +01:00
Cory Dransfeldt
28d2d09633
chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot 2025-10-19 15:57:28 -07:00
dark-visitors
91bf905fa9 Update from Dark Visitors 2025-10-15 00:56:30 +00:00
dark-visitors
56d03d46fb Update from Dark Visitors 2025-09-26 00:54:30 +00:00
Cory Dransfeldt
bf347bdf91
chore(robots.json): add DeepSeekBot 2025-09-24 09:42:01 -07:00
dark-visitors
2fd93029ca Update from Dark Visitors 2025-09-11 00:55:06 +00:00
Simon Ser
50870ba911 Add TerraCotta
Example server log:

    X.X.X.X - - [10/Sep/2025:00:06:17 +0000] "GET /archives/dri-devel/2023-November/430875.html HTTP/1.1" 200 7388 "-" "TerraCotta https://github.com/CeramicTeam/CeramicTerracotta"
2025-09-10 12:04:08 +02:00
dark-visitors
dd391bf960 Update from Dark Visitors 2025-09-10 00:53:56 +00:00
László Károlyi
ec508ab434
Update Brightbot operator and details; add meta-webindexer entry
- Update Brightbot operator to https://brightdata.com/brightbot.
- Change Brightbot frequency to "At least one per minute."
- Expand Brightbot description with disguise tactics link.
- Add new entry for meta-webindexer under Meta operator.
- Set meta-webindexer respect to "Unclear at this time."
- Define meta-webindexer function as "AI Assistants."
- Set meta-webindexer frequency to "Unhinged, more than 1 per second."
- Include meta-webindexer description on improving Meta AI search.
2025-09-08 21:42:06 +02:00
dark-visitors
0ed29412c9 Update from Dark Visitors 2025-08-28 00:55:47 +00:00
Cory Dransfeldt
1a8edfa84a
chore: add Google-Firebase and ShapBot 2025-08-26 19:39:42 -07:00
dark-visitors
cf073d49f2 Update from Dark Visitors 2025-08-15 01:01:42 +00:00
Cory Dransfeldt
0e687a5b58
chore: adds OpenAI but user agent 2025-08-13 09:01:55 -07:00
dark-visitors
146a229662 Update from Dark Visitors 2025-07-31 01:05:21 +00:00
dark-visitors
085dd1071e Update from Dark Visitors 2025-07-29 01:11:26 +00:00
Glyn Normington
8869442615
Merge pull request #166 from ai-robots-txt/summaly-bot
fix: remove summaly
2025-07-28 20:19:06 +01:00
Glyn Normington
12c5368e04
Fix syntax 2025-07-28 19:29:19 +01:00
Glitz Smarter
c15065544a
Rewriting function in YaK
Rewrote the function to make it clear the quote is according to the page on the Clearwater website.
2025-07-28 12:54:25 -04:00
Cory Dransfeldt
9171625db6
fix: remove summaly 2025-07-28 08:56:21 -07:00
Gregory
8b188d0612 Added YaK user-agent to list of robots
Added Meltwater's AI to list of bots that use AI and scrape websites.
2025-07-26 23:50:11 -04:00
dark-visitors
9843c31303 Update from Dark Visitors 2025-07-16 01:03:43 +00:00
dark-visitors
f6f504012f Update from Dark Visitors 2025-07-08 01:01:25 +00:00
Cory Dransfeldt
c5dd4e98b6
chore(robots.json): add SummalyBot + Thinkbot 2025-07-07 09:49:26 -07:00
dark-visitors
70ff0c9fbd Update from Dark Visitors 2025-07-03 01:00:49 +00:00
Cory Dransfeldt
91ad302d68
Merge pull request #156 from nisbet-hubbard/patch-9
Update funtion and description for SemrushBot
2025-07-02 08:58:08 -07:00
dark-visitors
79bc22523f Update from Dark Visitors 2025-07-02 01:01:05 +00:00
dark-visitors
7eb2099d3f Update from Dark Visitors 2025-06-28 00:58:57 +00:00
nisbet-hubbard
accc48f327
Update funtion and description for SemrushBot 2025-06-21 08:48:58 +08:00
dark-visitors
4ed17b8e4a Update from Dark Visitors 2025-06-17 01:00:21 +00:00
paulrudy
7535893aec re-add facebookexternalhit 2025-06-15 16:49:07 -07:00
Cory Dransfeldt
26a46c409d
Merge pull request #153 from sergiospagnuolo/Poseidon
Update robots.json with new crawler
2025-06-14 07:03:52 -07:00