81 commits

Author	SHA1	Message	Date
Cory Dransfeldt	d760f9216f	chore(robots.json): adds MyCentralAIScraperBot	2025-06-12 13:08:29 -07:00
dark-visitors	cf598b6b71	Update from Dark Visitors	2025-06-10 01:00:37 +00:00
Cory Dransfeldt	7867c3e26c	chore(robots.json): adds EchoboxBot (#148)	2025-06-09 16:44:25 +01:00
dark-visitors	e21f6ae1b6	Update from Dark Visitors	2025-06-06 00:59:25 +00:00
Glyn Normington	81747e6772	Merge pull request #145 from ai-robots-txt/aws-bedrockbot ... chore(robots.json): adds bedrockbot	2025-06-05 17:51:03 +01:00
Cory Dransfeldt	528d77bf07	chore(robots.json): adds bedrockbot	2025-06-05 09:14:23 -07:00
dark-visitors	77393df5aa	Update from Dark Visitors	2025-06-05 00:59:28 +00:00
Glyn Normington	2fca1ddcf1	Merge pull request #143 from ai-robots-txt/panscient ... chore(robots.json): adds Panscient	2025-06-04 19:03:53 +01:00
Cory Dransfeldt	4568d69b0e	chore(robots.json): adds Panscient	2025-06-04 10:54:14 -07:00
Cory Dransfeldt	03831a7eb5	chore(robots.json): adds Quillbot	2025-06-04 10:46:58 -07:00
dark-visitors	2b5a59a303	Update from Dark Visitors	2025-06-04 01:00:07 +00:00
Ivan Chupin	8f75f4a2f5	Add SBIntuitionsBot	2025-06-04 03:48:42 +05:00
Cory Dransfeldt	3187fd8a32	chore(robots.json): adds YandexAdditional crawlers	2025-06-03 12:41:57 -07:00
dark-visitors	87016d1504	Update from Dark Visitors	2025-06-03 01:00:29 +00:00
Cory Dransfeldt	1dd66b6969	Revert "chore(robots.json): adds imgproxy crawler" ... This reverts commit b65f45e408.	2025-06-02 11:53:06 -07:00
Cory Dransfeldt	268922f8f2	Update robots.json	2025-05-31 16:02:05 -07:00
Cory Dransfeldt	4259b25ccc	Update robots.json	2025-05-31 16:01:09 -07:00
Cory Dransfeldt	d22b9ec51a	Update robots.json	2025-05-31 16:00:13 -07:00
imp	3e8edd083e	Add AndiBot and PhindBot ... Fixes #75	2025-05-23 13:03:49 +01:00
dark-visitors	7bf7f9164d	Update from Dark Visitors	2025-05-22 00:58:45 +00:00
Glyn Normington	851eabe059	Merge pull request #133 from ai-robots-txt/wpbot ... chore(robots.json): adds wpbot	2025-05-21 22:05:51 +01:00
Cory Dransfeldt	af597586b6	Merge pull request #98 from kylebuckingham/main ... Updating Claude Bots	2025-05-21 12:00:11 -07:00
Cory Dransfeldt	b1d9a60a38	chore(robots.json): adds wpbot	2025-05-21 11:40:33 -07:00
Glyn Normington	202d3c3b9a	Merge pull request #126 from ai-robots-txt/mistral-bot ... chore(robots.json): adds MistralAI-User/1.0 crawler	2025-05-21 16:27:14 +01:00
dark-visitors	7a2e6cba52	Update from Dark Visitors	2025-05-17 00:57:28 +00:00
Cory Dransfeldt	ca918a963f	chore(robots.json): adds Google-CloudVertexBot	2025-05-15 21:16:49 -07:00
Cory Dransfeldt	5fba0b746d	chore(robots.json): adds MistralAI-User/1.0 crawler	2025-05-15 20:45:20 -07:00
dark-visitors	16d1de7094	Update from Dark Visitors	2025-05-16 00:59:08 +00:00
Joe Hoyle	0c56b96fd9	Fix JSON syntax error	2025-05-15 11:26:47 -04:00
Cory Dransfeldt	9539256cb3	chore(robots.json): adds QualifiedBot crawler	2025-05-15 07:16:07 -07:00
dark-visitors	934ac7b318	Update from Dark Visitors	2025-04-25 00:56:57 +00:00
maia	9d846ced45	Update robots.json ... Lowercase meta-external* as that was not technically the UA for the bots, also removed a period in the "respect" for consistency	2025-04-24 04:08:20 +02:00
dark-visitors	8d25a424d9	Update from Dark Visitors	2025-04-23 00:56:52 +00:00
Dennis Lee	33c5ce1326	Update robots.json ... Updated robots list with five new proposed AI bots: aiHitBot Cotoyogi Factset_spyderbot FirecrawlAgent TikTokSpider	2025-04-21 18:55:11 +01:00
Kyle Buckingham	fd41de8522	Update robots.json ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-04-16 16:43:03 -07:00
Kyle Buckingham	4a6f37d727	Update robots.json ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-04-16 16:42:58 -07:00
dark-visitors	a96e330989	Update from Dark Visitors	2025-04-15 00:57:01 +00:00
dark-visitors	305188b2e7	Update from Dark Visitors	2025-04-11 00:55:52 +00:00
Cory Dransfeldt	b65f45e408	chore(robots.json): adds imgproxy crawler	2025-04-10 10:12:51 -07:00
Frederic Barthelemy	5f5a89c38c	Fix html-mangled hyphen in Perplexity-Users ... Fixes: #99	2025-04-04 17:34:14 -07:00
Kyle Buckingham	ae8f74c10c	Update robots.json	2025-04-01 15:22:04 -07:00
dark-visitors	c249de99a3	Update from Dark Visitors	2025-03-28 00:54:28 +00:00
deyigifts	6ecfcdfcbf	Update perplexity bot ... Update based on perplexity bot docs	2025-03-24 14:16:57 +08:00
dark-visitors	abfd6dfcd1	Update from Dark Visitors	2025-02-17 00:53:32 +00:00
Cory Dransfeldt	a9ec4ffa6f	chore: add Brightbot 1.0	2025-02-16 13:36:39 -08:00
dark-visitors	bebffccc0c	Update from Dark Visitors	2025-02-02 00:52:50 +00:00
nisbet-hubbard	05b79b8a58	Update robots.json	2025-01-27 19:41:03 +08:00
dark-visitors	9c060dee1c	Update from Dark Visitors	2025-01-21 00:49:22 +00:00
Joshua Sheard	7427d96bac	Update robots.json ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-01-20 10:59:02 +00:00
Joshua Sheard	5aa08bc002	Add Crawlspace	2025-01-19 22:03:50 +00:00