114 commits

Author	SHA1	Message	Date
dark-visitors	56d03d46fb	Update from Dark Visitors	2025-09-26 00:54:30 +00:00
Cory Dransfeldt	bf347bdf91	chore(robots.json): add DeepSeekBot	2025-09-24 09:42:01 -07:00
dark-visitors	2fd93029ca	Update from Dark Visitors	2025-09-11 00:55:06 +00:00
Simon Ser	50870ba911	Add TerraCotta ... Example server log: X.X.X.X - - [10/Sep/2025:00:06:17 +0000] "GET /archives/dri-devel/2023-November/430875.html HTTP/1.1" 200 7388 "-" "TerraCotta https://github.com/CeramicTeam/CeramicTerracotta"	2025-09-10 12:04:08 +02:00
dark-visitors	dd391bf960	Update from Dark Visitors	2025-09-10 00:53:56 +00:00
László Károlyi	ec508ab434	Update Brightbot operator and details; add meta-webindexer entry ... - Update Brightbot operator to https://brightdata.com/brightbot. - Change Brightbot frequency to "At least one per minute." - Expand Brightbot description with disguise tactics link. - Add new entry for meta-webindexer under Meta operator. - Set meta-webindexer respect to "Unclear at this time." - Define meta-webindexer function as "AI Assistants." - Set meta-webindexer frequency to "Unhinged, more than 1 per second." - Include meta-webindexer description on improving Meta AI search.	2025-09-08 21:42:06 +02:00
dark-visitors	0ed29412c9	Update from Dark Visitors	2025-08-28 00:55:47 +00:00
Cory Dransfeldt	1a8edfa84a	chore: add Google-Firebase and ShapBot	2025-08-26 19:39:42 -07:00
dark-visitors	cf073d49f2	Update from Dark Visitors	2025-08-15 01:01:42 +00:00
Cory Dransfeldt	0e687a5b58	chore: adds OpenAI but user agent	2025-08-13 09:01:55 -07:00
dark-visitors	146a229662	Update from Dark Visitors	2025-07-31 01:05:21 +00:00
dark-visitors	085dd1071e	Update from Dark Visitors	2025-07-29 01:11:26 +00:00
Glyn Normington	8869442615	Merge pull request #166 from ai-robots-txt/summaly-bot ... fix: remove summaly	2025-07-28 20:19:06 +01:00
Glyn Normington	12c5368e04	Fix syntax	2025-07-28 19:29:19 +01:00
Glitz Smarter	c15065544a	Rewriting function in YaK ... Rewrote the function to make it clear the quote is according to the page on the Clearwater website.	2025-07-28 12:54:25 -04:00
Cory Dransfeldt	9171625db6	fix: remove summaly	2025-07-28 08:56:21 -07:00
Gregory	8b188d0612	Added YaK user-agent to list of robots ... Added Meltwater's AI to list of bots that use AI and scrape websites.	2025-07-26 23:50:11 -04:00
dark-visitors	9843c31303	Update from Dark Visitors	2025-07-16 01:03:43 +00:00
dark-visitors	f6f504012f	Update from Dark Visitors	2025-07-08 01:01:25 +00:00
Cory Dransfeldt	c5dd4e98b6	chore(robots.json): add SummalyBot + Thinkbot	2025-07-07 09:49:26 -07:00
dark-visitors	70ff0c9fbd	Update from Dark Visitors	2025-07-03 01:00:49 +00:00
Cory Dransfeldt	91ad302d68	Merge pull request #156 from nisbet-hubbard/patch-9 ... Update funtion and description for SemrushBot	2025-07-02 08:58:08 -07:00
dark-visitors	79bc22523f	Update from Dark Visitors	2025-07-02 01:01:05 +00:00
dark-visitors	7eb2099d3f	Update from Dark Visitors	2025-06-28 00:58:57 +00:00
nisbet-hubbard	accc48f327	Update funtion and description for SemrushBot	2025-06-21 08:48:58 +08:00
dark-visitors	4ed17b8e4a	Update from Dark Visitors	2025-06-17 01:00:21 +00:00
paulrudy	7535893aec	re-add facebookexternalhit	2025-06-15 16:49:07 -07:00
Cory Dransfeldt	26a46c409d	Merge pull request #153 from sergiospagnuolo/Poseidon ... Update robots.json with new crawler	2025-06-14 07:03:52 -07:00
dark-visitors	2b68568ac2	Update from Dark Visitors	2025-06-14 00:58:11 +00:00
Sérgio Spagnuolo	b05f2fee00	Update robots.json with new crawler ... Update with Poseidon Research Crawler as found in nytimes.com/robots.txt	2025-06-13 17:15:13 -03:00
Glyn Normington	20e327e74e	Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ... chore(robots.json): adds MyCentralAIScraperBot	2025-06-13 10:28:32 +01:00
Glyn Normington	8f17718e76	Fix typo	2025-06-13 10:28:12 +01:00
Cory Dransfeldt	d760f9216f	chore(robots.json): adds MyCentralAIScraperBot	2025-06-12 13:08:29 -07:00
Cory Dransfeldt	14d68f05ba	chore(robots.json): adds additional SemrushBot user agents	2025-06-11 13:50:53 -07:00
dark-visitors	cf598b6b71	Update from Dark Visitors	2025-06-10 01:00:37 +00:00
Cory Dransfeldt	7867c3e26c	chore(robots.json): adds EchoboxBot (#148)	2025-06-09 16:44:25 +01:00
dark-visitors	e21f6ae1b6	Update from Dark Visitors	2025-06-06 00:59:25 +00:00
Glyn Normington	81747e6772	Merge pull request #145 from ai-robots-txt/aws-bedrockbot ... chore(robots.json): adds bedrockbot	2025-06-05 17:51:03 +01:00
Cory Dransfeldt	528d77bf07	chore(robots.json): adds bedrockbot	2025-06-05 09:14:23 -07:00
dark-visitors	77393df5aa	Update from Dark Visitors	2025-06-05 00:59:28 +00:00
Glyn Normington	2fca1ddcf1	Merge pull request #143 from ai-robots-txt/panscient ... chore(robots.json): adds Panscient	2025-06-04 19:03:53 +01:00
Cory Dransfeldt	4568d69b0e	chore(robots.json): adds Panscient	2025-06-04 10:54:14 -07:00
Cory Dransfeldt	03831a7eb5	chore(robots.json): adds Quillbot	2025-06-04 10:46:58 -07:00
dark-visitors	2b5a59a303	Update from Dark Visitors	2025-06-04 01:00:07 +00:00
Ivan Chupin	8f75f4a2f5	Add SBIntuitionsBot	2025-06-04 03:48:42 +05:00
Cory Dransfeldt	3187fd8a32	chore(robots.json): adds YandexAdditional crawlers	2025-06-03 12:41:57 -07:00
dark-visitors	87016d1504	Update from Dark Visitors	2025-06-03 01:00:29 +00:00
Cory Dransfeldt	1dd66b6969	Revert "chore(robots.json): adds imgproxy crawler" ... This reverts commit b65f45e408.	2025-06-02 11:53:06 -07:00
Cory Dransfeldt	268922f8f2	Update robots.json	2025-05-31 16:02:05 -07:00
Cory Dransfeldt	4259b25ccc	Update robots.json	2025-05-31 16:01:09 -07:00