136 commits

Author	SHA1	Message	Date
dark-visitors	83485effdb	Update from Dark Visitors	2025-12-20 00:58:49 +00:00
dark-visitors	f1c752ef12	Update from Dark Visitors	2025-12-05 01:00:44 +00:00
dark-visitors	45b071b29f	Update from Dark Visitors	2025-12-04 01:00:27 +00:00
dark-visitors	f61b3496f7	Update from Dark Visitors	2025-12-03 01:00:32 +00:00
dark-visitors	8363d4fdd4	Update from Dark Visitors	2025-12-02 01:25:24 +00:00
Adam Newbold	f5d7ccb243	Fixed invalid JSON	2025-11-28 14:10:38 -05:00
Adam Newbold	30d719a09a	Adding LAIONDownloader	2025-11-28 14:08:18 -05:00
Glyn Normington	ab24e41106	Clarify that DeepSeekBot does not respect robots.txt ... Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/198	2025-11-26 12:02:42 +00:00
Glyn Normington	efb4d260da	delete extraneous hyphen ... Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/193	2025-11-20 10:23:48 +00:00
Glyn Normington	663d030f96	Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo ... chore: adds NotebookLM and KlaviyoAIBot agents	2025-11-14 21:01:13 +00:00
Cory Dransfeldt	28b45ea08d	chore: adds NotebookLM and KlaviyoAIBot agents	2025-11-14 11:03:03 -08:00
Glyn Normington	60b6a0829d	Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots ... chore(robots.json): add AmazonBuyForMe and atlassian-bot	2025-11-05 23:50:48 +00:00
Cory Dransfeldt	00bf2b0e13	chore(robots.json): add AmazonBuyForMe and atlassian-bot	2025-11-05 14:02:37 -08:00
Cory Dransfeldt	808451055c	chore: adds BuddyBot	2025-11-03 13:39:07 -08:00
Cory Dransfeldt	192b0a2eef	chore: adds Linguee Bot	2025-10-24 10:35:04 -07:00
dark-visitors	97e19445ce	Update from Dark Visitors	2025-10-24 00:53:30 +00:00
Glyn Normington	b89a9eae6a	Use third party in text, rather than first	2025-10-21 01:11:32 +01:00
Glyn Normington	d22b8dfd7b	Description is third party, not first	2025-10-21 01:08:42 +01:00
Cory Dransfeldt	19c1d346c3	chore: adds IbouBot	2025-10-20 09:28:31 -07:00
Glyn Normington	0874a92503	Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare ... chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot	2025-10-20 15:34:30 +01:00
Cory Dransfeldt	28d2d09633	chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot	2025-10-19 15:57:28 -07:00
dark-visitors	91bf905fa9	Update from Dark Visitors	2025-10-15 00:56:30 +00:00
dark-visitors	56d03d46fb	Update from Dark Visitors	2025-09-26 00:54:30 +00:00
Cory Dransfeldt	bf347bdf91	chore(robots.json): add DeepSeekBot	2025-09-24 09:42:01 -07:00
dark-visitors	2fd93029ca	Update from Dark Visitors	2025-09-11 00:55:06 +00:00
Simon Ser	50870ba911	Add TerraCotta ... Example server log: X.X.X.X - - [10/Sep/2025:00:06:17 +0000] "GET /archives/dri-devel/2023-November/430875.html HTTP/1.1" 200 7388 "-" "TerraCotta https://github.com/CeramicTeam/CeramicTerracotta"	2025-09-10 12:04:08 +02:00
dark-visitors	dd391bf960	Update from Dark Visitors	2025-09-10 00:53:56 +00:00
László Károlyi	ec508ab434	Update Brightbot operator and details; add meta-webindexer entry ... - Update Brightbot operator to https://brightdata.com/brightbot. - Change Brightbot frequency to "At least one per minute." - Expand Brightbot description with disguise tactics link. - Add new entry for meta-webindexer under Meta operator. - Set meta-webindexer respect to "Unclear at this time." - Define meta-webindexer function as "AI Assistants." - Set meta-webindexer frequency to "Unhinged, more than 1 per second." - Include meta-webindexer description on improving Meta AI search.	2025-09-08 21:42:06 +02:00
dark-visitors	0ed29412c9	Update from Dark Visitors	2025-08-28 00:55:47 +00:00
Cory Dransfeldt	1a8edfa84a	chore: add Google-Firebase and ShapBot	2025-08-26 19:39:42 -07:00
dark-visitors	cf073d49f2	Update from Dark Visitors	2025-08-15 01:01:42 +00:00
Cory Dransfeldt	0e687a5b58	chore: adds OpenAI but user agent	2025-08-13 09:01:55 -07:00
dark-visitors	146a229662	Update from Dark Visitors	2025-07-31 01:05:21 +00:00
dark-visitors	085dd1071e	Update from Dark Visitors	2025-07-29 01:11:26 +00:00
Glyn Normington	8869442615	Merge pull request #166 from ai-robots-txt/summaly-bot ... fix: remove summaly	2025-07-28 20:19:06 +01:00
Glyn Normington	12c5368e04	Fix syntax	2025-07-28 19:29:19 +01:00
Glitz Smarter	c15065544a	Rewriting function in YaK ... Rewrote the function to make it clear the quote is according to the page on the Clearwater website.	2025-07-28 12:54:25 -04:00
Cory Dransfeldt	9171625db6	fix: remove summaly	2025-07-28 08:56:21 -07:00
Gregory	8b188d0612	Added YaK user-agent to list of robots ... Added Meltwater's AI to list of bots that use AI and scrape websites.	2025-07-26 23:50:11 -04:00
dark-visitors	9843c31303	Update from Dark Visitors	2025-07-16 01:03:43 +00:00
dark-visitors	f6f504012f	Update from Dark Visitors	2025-07-08 01:01:25 +00:00
Cory Dransfeldt	c5dd4e98b6	chore(robots.json): add SummalyBot + Thinkbot	2025-07-07 09:49:26 -07:00
dark-visitors	70ff0c9fbd	Update from Dark Visitors	2025-07-03 01:00:49 +00:00
Cory Dransfeldt	91ad302d68	Merge pull request #156 from nisbet-hubbard/patch-9 ... Update funtion and description for SemrushBot	2025-07-02 08:58:08 -07:00
dark-visitors	79bc22523f	Update from Dark Visitors	2025-07-02 01:01:05 +00:00
dark-visitors	7eb2099d3f	Update from Dark Visitors	2025-06-28 00:58:57 +00:00
nisbet-hubbard	accc48f327	Update funtion and description for SemrushBot	2025-06-21 08:48:58 +08:00
dark-visitors	4ed17b8e4a	Update from Dark Visitors	2025-06-17 01:00:21 +00:00
paulrudy	7535893aec	re-add facebookexternalhit	2025-06-15 16:49:07 -07:00
Cory Dransfeldt	26a46c409d	Merge pull request #153 from sergiospagnuolo/Poseidon ... Update robots.json with new crawler	2025-06-14 07:03:52 -07:00