105 commits

Author	SHA1	Message	Date
Cory Dransfeldt	0e687a5b58	chore: adds OpenAI but user agent	2025-08-13 09:01:55 -07:00
dark-visitors	146a229662	Update from Dark Visitors	2025-07-31 01:05:21 +00:00
dark-visitors	085dd1071e	Update from Dark Visitors	2025-07-29 01:11:26 +00:00
Glyn Normington	8869442615	Merge pull request #166 from ai-robots-txt/summaly-bot ... fix: remove summaly	2025-07-28 20:19:06 +01:00
Glyn Normington	12c5368e04	Fix syntax	2025-07-28 19:29:19 +01:00
Glitz Smarter	c15065544a	Rewriting function in YaK ... Rewrote the function to make it clear the quote is according to the page on the Clearwater website.	2025-07-28 12:54:25 -04:00
Cory Dransfeldt	9171625db6	fix: remove summaly	2025-07-28 08:56:21 -07:00
Gregory	8b188d0612	Added YaK user-agent to list of robots ... Added Meltwater's AI to list of bots that use AI and scrape websites.	2025-07-26 23:50:11 -04:00
dark-visitors	9843c31303	Update from Dark Visitors	2025-07-16 01:03:43 +00:00
dark-visitors	f6f504012f	Update from Dark Visitors	2025-07-08 01:01:25 +00:00
Cory Dransfeldt	c5dd4e98b6	chore(robots.json): add SummalyBot + Thinkbot	2025-07-07 09:49:26 -07:00
dark-visitors	70ff0c9fbd	Update from Dark Visitors	2025-07-03 01:00:49 +00:00
Cory Dransfeldt	91ad302d68	Merge pull request #156 from nisbet-hubbard/patch-9 ... Update funtion and description for SemrushBot	2025-07-02 08:58:08 -07:00
dark-visitors	79bc22523f	Update from Dark Visitors	2025-07-02 01:01:05 +00:00
dark-visitors	7eb2099d3f	Update from Dark Visitors	2025-06-28 00:58:57 +00:00
nisbet-hubbard	accc48f327	Update funtion and description for SemrushBot	2025-06-21 08:48:58 +08:00
dark-visitors	4ed17b8e4a	Update from Dark Visitors	2025-06-17 01:00:21 +00:00
paulrudy	7535893aec	re-add facebookexternalhit	2025-06-15 16:49:07 -07:00
Cory Dransfeldt	26a46c409d	Merge pull request #153 from sergiospagnuolo/Poseidon ... Update robots.json with new crawler	2025-06-14 07:03:52 -07:00
dark-visitors	2b68568ac2	Update from Dark Visitors	2025-06-14 00:58:11 +00:00
Sérgio Spagnuolo	b05f2fee00	Update robots.json with new crawler ... Update with Poseidon Research Crawler as found in nytimes.com/robots.txt	2025-06-13 17:15:13 -03:00
Glyn Normington	20e327e74e	Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ... chore(robots.json): adds MyCentralAIScraperBot	2025-06-13 10:28:32 +01:00
Glyn Normington	8f17718e76	Fix typo	2025-06-13 10:28:12 +01:00
Cory Dransfeldt	d760f9216f	chore(robots.json): adds MyCentralAIScraperBot	2025-06-12 13:08:29 -07:00
Cory Dransfeldt	14d68f05ba	chore(robots.json): adds additional SemrushBot user agents	2025-06-11 13:50:53 -07:00
dark-visitors	cf598b6b71	Update from Dark Visitors	2025-06-10 01:00:37 +00:00
Cory Dransfeldt	7867c3e26c	chore(robots.json): adds EchoboxBot (#148)	2025-06-09 16:44:25 +01:00
dark-visitors	e21f6ae1b6	Update from Dark Visitors	2025-06-06 00:59:25 +00:00
Glyn Normington	81747e6772	Merge pull request #145 from ai-robots-txt/aws-bedrockbot ... chore(robots.json): adds bedrockbot	2025-06-05 17:51:03 +01:00
Cory Dransfeldt	528d77bf07	chore(robots.json): adds bedrockbot	2025-06-05 09:14:23 -07:00
dark-visitors	77393df5aa	Update from Dark Visitors	2025-06-05 00:59:28 +00:00
Glyn Normington	2fca1ddcf1	Merge pull request #143 from ai-robots-txt/panscient ... chore(robots.json): adds Panscient	2025-06-04 19:03:53 +01:00
Cory Dransfeldt	4568d69b0e	chore(robots.json): adds Panscient	2025-06-04 10:54:14 -07:00
Cory Dransfeldt	03831a7eb5	chore(robots.json): adds Quillbot	2025-06-04 10:46:58 -07:00
dark-visitors	2b5a59a303	Update from Dark Visitors	2025-06-04 01:00:07 +00:00
Ivan Chupin	8f75f4a2f5	Add SBIntuitionsBot	2025-06-04 03:48:42 +05:00
Cory Dransfeldt	3187fd8a32	chore(robots.json): adds YandexAdditional crawlers	2025-06-03 12:41:57 -07:00
dark-visitors	87016d1504	Update from Dark Visitors	2025-06-03 01:00:29 +00:00
Cory Dransfeldt	1dd66b6969	Revert "chore(robots.json): adds imgproxy crawler" ... This reverts commit b65f45e408.	2025-06-02 11:53:06 -07:00
Cory Dransfeldt	268922f8f2	Update robots.json	2025-05-31 16:02:05 -07:00
Cory Dransfeldt	4259b25ccc	Update robots.json	2025-05-31 16:01:09 -07:00
Cory Dransfeldt	d22b9ec51a	Update robots.json	2025-05-31 16:00:13 -07:00
imp	3e8edd083e	Add AndiBot and PhindBot ... Fixes #75	2025-05-23 13:03:49 +01:00
dark-visitors	7bf7f9164d	Update from Dark Visitors	2025-05-22 00:58:45 +00:00
Glyn Normington	851eabe059	Merge pull request #133 from ai-robots-txt/wpbot ... chore(robots.json): adds wpbot	2025-05-21 22:05:51 +01:00
Cory Dransfeldt	af597586b6	Merge pull request #98 from kylebuckingham/main ... Updating Claude Bots	2025-05-21 12:00:11 -07:00
Cory Dransfeldt	b1d9a60a38	chore(robots.json): adds wpbot	2025-05-21 11:40:33 -07:00
Glyn Normington	202d3c3b9a	Merge pull request #126 from ai-robots-txt/mistral-bot ... chore(robots.json): adds MistralAI-User/1.0 crawler	2025-05-21 16:27:14 +01:00
dark-visitors	7a2e6cba52	Update from Dark Visitors	2025-05-17 00:57:28 +00:00
Cory Dransfeldt	ca918a963f	chore(robots.json): adds Google-CloudVertexBot	2025-05-15 21:16:49 -07:00