99 commits

Author	SHA1	Message	Date
Glitz Smarter	c15065544a	Rewriting function in YaK ... Rewrote the function to make it clear the quote is according to the page on the Clearwater website.	2025-07-28 12:54:25 -04:00
Gregory	8b188d0612	Added YaK user-agent to list of robots ... Added Meltwater's AI to list of bots that use AI and scrape websites.	2025-07-26 23:50:11 -04:00
dark-visitors	9843c31303	Update from Dark Visitors	2025-07-16 01:03:43 +00:00
dark-visitors	f6f504012f	Update from Dark Visitors	2025-07-08 01:01:25 +00:00
Cory Dransfeldt	c5dd4e98b6	chore(robots.json): add SummalyBot + Thinkbot	2025-07-07 09:49:26 -07:00
dark-visitors	70ff0c9fbd	Update from Dark Visitors	2025-07-03 01:00:49 +00:00
Cory Dransfeldt	91ad302d68	Merge pull request #156 from nisbet-hubbard/patch-9 ... Update funtion and description for SemrushBot	2025-07-02 08:58:08 -07:00
dark-visitors	79bc22523f	Update from Dark Visitors	2025-07-02 01:01:05 +00:00
dark-visitors	7eb2099d3f	Update from Dark Visitors	2025-06-28 00:58:57 +00:00
nisbet-hubbard	accc48f327	Update funtion and description for SemrushBot	2025-06-21 08:48:58 +08:00
dark-visitors	4ed17b8e4a	Update from Dark Visitors	2025-06-17 01:00:21 +00:00
paulrudy	7535893aec	re-add facebookexternalhit	2025-06-15 16:49:07 -07:00
Cory Dransfeldt	26a46c409d	Merge pull request #153 from sergiospagnuolo/Poseidon ... Update robots.json with new crawler	2025-06-14 07:03:52 -07:00
dark-visitors	2b68568ac2	Update from Dark Visitors	2025-06-14 00:58:11 +00:00
Sérgio Spagnuolo	b05f2fee00	Update robots.json with new crawler ... Update with Poseidon Research Crawler as found in nytimes.com/robots.txt	2025-06-13 17:15:13 -03:00
Glyn Normington	20e327e74e	Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ... chore(robots.json): adds MyCentralAIScraperBot	2025-06-13 10:28:32 +01:00
Glyn Normington	8f17718e76	Fix typo	2025-06-13 10:28:12 +01:00
Cory Dransfeldt	d760f9216f	chore(robots.json): adds MyCentralAIScraperBot	2025-06-12 13:08:29 -07:00
Cory Dransfeldt	14d68f05ba	chore(robots.json): adds additional SemrushBot user agents	2025-06-11 13:50:53 -07:00
dark-visitors	cf598b6b71	Update from Dark Visitors	2025-06-10 01:00:37 +00:00
Cory Dransfeldt	7867c3e26c	chore(robots.json): adds EchoboxBot (#148)	2025-06-09 16:44:25 +01:00
dark-visitors	e21f6ae1b6	Update from Dark Visitors	2025-06-06 00:59:25 +00:00
Glyn Normington	81747e6772	Merge pull request #145 from ai-robots-txt/aws-bedrockbot ... chore(robots.json): adds bedrockbot	2025-06-05 17:51:03 +01:00
Cory Dransfeldt	528d77bf07	chore(robots.json): adds bedrockbot	2025-06-05 09:14:23 -07:00
dark-visitors	77393df5aa	Update from Dark Visitors	2025-06-05 00:59:28 +00:00
Glyn Normington	2fca1ddcf1	Merge pull request #143 from ai-robots-txt/panscient ... chore(robots.json): adds Panscient	2025-06-04 19:03:53 +01:00
Cory Dransfeldt	4568d69b0e	chore(robots.json): adds Panscient	2025-06-04 10:54:14 -07:00
Cory Dransfeldt	03831a7eb5	chore(robots.json): adds Quillbot	2025-06-04 10:46:58 -07:00
dark-visitors	2b5a59a303	Update from Dark Visitors	2025-06-04 01:00:07 +00:00
Ivan Chupin	8f75f4a2f5	Add SBIntuitionsBot	2025-06-04 03:48:42 +05:00
Cory Dransfeldt	3187fd8a32	chore(robots.json): adds YandexAdditional crawlers	2025-06-03 12:41:57 -07:00
dark-visitors	87016d1504	Update from Dark Visitors	2025-06-03 01:00:29 +00:00
Cory Dransfeldt	1dd66b6969	Revert "chore(robots.json): adds imgproxy crawler" ... This reverts commit b65f45e408.	2025-06-02 11:53:06 -07:00
Cory Dransfeldt	268922f8f2	Update robots.json	2025-05-31 16:02:05 -07:00
Cory Dransfeldt	4259b25ccc	Update robots.json	2025-05-31 16:01:09 -07:00
Cory Dransfeldt	d22b9ec51a	Update robots.json	2025-05-31 16:00:13 -07:00
imp	3e8edd083e	Add AndiBot and PhindBot ... Fixes #75	2025-05-23 13:03:49 +01:00
dark-visitors	7bf7f9164d	Update from Dark Visitors	2025-05-22 00:58:45 +00:00
Glyn Normington	851eabe059	Merge pull request #133 from ai-robots-txt/wpbot ... chore(robots.json): adds wpbot	2025-05-21 22:05:51 +01:00
Cory Dransfeldt	af597586b6	Merge pull request #98 from kylebuckingham/main ... Updating Claude Bots	2025-05-21 12:00:11 -07:00
Cory Dransfeldt	b1d9a60a38	chore(robots.json): adds wpbot	2025-05-21 11:40:33 -07:00
Glyn Normington	202d3c3b9a	Merge pull request #126 from ai-robots-txt/mistral-bot ... chore(robots.json): adds MistralAI-User/1.0 crawler	2025-05-21 16:27:14 +01:00
dark-visitors	7a2e6cba52	Update from Dark Visitors	2025-05-17 00:57:28 +00:00
Cory Dransfeldt	ca918a963f	chore(robots.json): adds Google-CloudVertexBot	2025-05-15 21:16:49 -07:00
Cory Dransfeldt	5fba0b746d	chore(robots.json): adds MistralAI-User/1.0 crawler	2025-05-15 20:45:20 -07:00
dark-visitors	16d1de7094	Update from Dark Visitors	2025-05-16 00:59:08 +00:00
Joe Hoyle	0c56b96fd9	Fix JSON syntax error	2025-05-15 11:26:47 -04:00
Cory Dransfeldt	9539256cb3	chore(robots.json): adds QualifiedBot crawler	2025-05-15 07:16:07 -07:00
dark-visitors	934ac7b318	Update from Dark Visitors	2025-04-25 00:56:57 +00:00
maia	9d846ced45	Update robots.json ... Lowercase meta-external* as that was not technically the UA for the bots, also removed a period in the "respect" for consistency	2025-04-24 04:08:20 +02:00