53 commits

Author	SHA1	Message	Date
dark-visitors	7a2e6cba52	Update from Dark Visitors	2025-05-17 00:57:28 +00:00
Cory Dransfeldt	ca918a963f	chore(robots.json): adds Google-CloudVertexBot	2025-05-15 21:16:49 -07:00
dark-visitors	16d1de7094	Update from Dark Visitors	2025-05-16 00:59:08 +00:00
Joe Hoyle	0c56b96fd9	Fix JSON syntax error	2025-05-15 11:26:47 -04:00
Cory Dransfeldt	9539256cb3	chore(robots.json): adds QualifiedBot crawler	2025-05-15 07:16:07 -07:00
dark-visitors	934ac7b318	Update from Dark Visitors	2025-04-25 00:56:57 +00:00
maia	9d846ced45	Update robots.json ... Lowercase meta-external* as that was not technically the UA for the bots, also removed a period in the "respect" for consistency	2025-04-24 04:08:20 +02:00
dark-visitors	8d25a424d9	Update from Dark Visitors	2025-04-23 00:56:52 +00:00
Dennis Lee	33c5ce1326	Update robots.json ... Updated robots list with five new proposed AI bots: aiHitBot Cotoyogi Factset_spyderbot FirecrawlAgent TikTokSpider	2025-04-21 18:55:11 +01:00
dark-visitors	a96e330989	Update from Dark Visitors	2025-04-15 00:57:01 +00:00
dark-visitors	305188b2e7	Update from Dark Visitors	2025-04-11 00:55:52 +00:00
Cory Dransfeldt	b65f45e408	chore(robots.json): adds imgproxy crawler	2025-04-10 10:12:51 -07:00
Frederic Barthelemy	5f5a89c38c	Fix html-mangled hyphen in Perplexity-Users ... Fixes: #99	2025-04-04 17:34:14 -07:00
dark-visitors	c249de99a3	Update from Dark Visitors	2025-03-28 00:54:28 +00:00
deyigifts	6ecfcdfcbf	Update perplexity bot ... Update based on perplexity bot docs	2025-03-24 14:16:57 +08:00
dark-visitors	abfd6dfcd1	Update from Dark Visitors	2025-02-17 00:53:32 +00:00
Cory Dransfeldt	a9ec4ffa6f	chore: add Brightbot 1.0	2025-02-16 13:36:39 -08:00
dark-visitors	bebffccc0c	Update from Dark Visitors	2025-02-02 00:52:50 +00:00
nisbet-hubbard	05b79b8a58	Update robots.json	2025-01-27 19:41:03 +08:00
dark-visitors	9c060dee1c	Update from Dark Visitors	2025-01-21 00:49:22 +00:00
Joshua Sheard	7427d96bac	Update robots.json ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-01-20 10:59:02 +00:00
Joshua Sheard	5aa08bc002	Add Crawlspace	2025-01-19 22:03:50 +00:00
Jordan Atwood	143f8f2285	Block SemrushBot	2025-01-06 12:34:38 -08:00
dark-visitors	2036a68c1f	Update from Dark Visitors	2024-12-04 00:55:50 +00:00
dark-visitors	37065f9118	Update from Dark Visitors	2024-11-24 00:57:05 +00:00
Glyn Normington	80002f5e17	Allow facebookexternalhit ... At the time of writing, this crawler does not appear to be for the purpose of AI. See: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ (accessed on 19 November 2024). Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/40	2024-11-19 03:33:45 +00:00
dark-visitors	bc0a0ad0e9	Update from Dark Visitors	2024-10-29 00:52:12 +00:00
dark-visitors	fe5f407673	Update from Dark Visitors	2024-10-27 00:54:47 +00:00
Glyn Normington	38a388097c	Fix typo and trigger rerun of main job	2024-10-19 04:42:27 +01:00
dark-visitors	faf81efb12	Daily update from Dark Visitors	2024-10-19 01:17:15 +00:00
dark-visitors	b1491d2694	Daily update from Dark Visitors	2024-10-09 01:17:37 +00:00
Laker Turner	dc15afe847	Update robots.json with Claude respect link	2024-10-07 17:38:01 +01:00
Cory Dransfeldt	9c2394f23b	chore: add ISSCyberRiskCrawler	2024-09-30 16:25:20 -07:00
Cory Dransfeldt	6a988be27f	chore: add sidetrade bot	2024-09-28 13:58:00 -07:00
dark-visitors	7851cea4fd	Daily update from Dark Visitors	2024-09-27 01:18:04 +00:00
Greg Lindahl	a6de89e6bd	feat: make CCBot entry more accurate	2024-09-26 21:41:28 +00:00
dark-visitors	5963cbf9f7	Daily update from Dark Visitors	2024-09-08 01:19:31 +00:00
Cory Dransfeldt	8373294404	chore: add iaskspider/2.0	2024-09-06 19:05:26 -07:00
Cory Dransfeldt	0f8723558f	chore: add ai2bot	2024-08-28 20:07:32 -07:00
dark-visitors	7bfc1647a8	Daily update from Dark Visitors	2024-08-22 01:11:43 +00:00
dark-visitors	5937434aff	Daily update from Dark Visitors	2024-08-15 01:07:15 +00:00
dark-visitors	6a275366be	Daily update from Dark Visitors	2024-08-07 10:50:45 +00:00
Chenghao Mou	944bee0f56	call main after update	2024-08-07 11:31:58 +01:00
dark-visitors	cebf809391	Daily update from Dark Visitors	2024-08-07 00:14:26 +00:00
Chenghao Mou	4cf82b703f	restore original robots.json	2024-08-06 19:50:38 +01:00
dark-visitors	63c7e742c3	Daily update from Dark Visitors	2024-08-06 16:54:29 +00:00
dark-visitors	fdd261dad4	Daily update from Dark Visitors	2024-08-06 16:27:02 +00:00
Joshua Sheard	146fd4ffba	Fix Imagesift user agent	2024-08-04 21:33:04 +01:00
Cory Dransfeldt	1ca936ce11	chore: restore FriendlyCrawler + ImageSift	2024-08-04 12:28:48 -07:00
Mirium999	5826c18909	Add ICC-Crawler	2024-08-04 10:11:25 +09:00