266 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Glyn Normington	4d47b17c45	Merge pull request #47 from fabianegli/fabianegli-patch-1 ... log the diff in the update actions	2024-10-19 02:58:05 +01:00
dark-visitors	faf81efb12	Daily update from Dark Visitors	2024-10-19 01:17:15 +00:00
Fabian Egli	25adc6b802	log git repository status	2024-10-19 00:28:41 +02:00
Fabian Egli	b584f613cd	add some signposts to the log	2024-10-19 00:13:09 +02:00
Fabian Egli	b3068a8d90	add some signposts	2024-10-19 00:12:25 +02:00
Fabian Egli	a46d06d436	log changes made by the action in main.yml	2024-10-19 00:04:15 +02:00
Fabian Egli	cfaade6e2f	log the diff in the update action daily_update.yml	2024-10-19 00:01:15 +02:00
Cory Dransfeldt	04f630f7f8	Merge pull request #45 from glyn/faq-update ... Update the FAQ	2024-10-18 06:35:47 -07:00
Glyn Normington	898c8ab82d	Merge pull request #46 from isagalaev/case-insensitive-sorting ... Sort the content of robots.json by keys, case-insensitively	2024-10-18 07:57:56 +01:00
Ivan Sagalaev	7bb5efd462	Sort the content case-insensitively before dumping to JSON	2024-10-17 21:08:43 -04:00
Glyn Normington	e6bb7cae9e	Augment the "why" FAQ ... Ref: https://github.com/ai-robots-txt/ai.robots.txt/issues/40#issuecomment-2419078796	2024-10-17 12:27:05 +01:00
Glyn Normington	b229f5b936	Re-order the FAQ ... The "why" question should come first.	2024-10-17 12:25:54 +01:00
dark-visitors	b1491d2694	Daily update from Dark Visitors	2024-10-09 01:17:37 +00:00
ai.robots.txt	9be286626d	Merge pull request #43 from lxjv/main ... Update robots.json with Claude respect link	2024-10-08 02:30:17 +00:00
Glyn Normington	01993b98c3	Merge pull request #43 from lxjv/main ... Update robots.json with Claude respect link	2024-10-08 03:30:07 +01:00
Laker Turner	dc15afe847	Update robots.json with Claude respect link	2024-10-07 17:38:01 +01:00
ai.robots.txt	6da804e826	chore: add ISSCyberRiskCrawler v1.18	2024-09-30 23:50:18 +00:00
Cory Dransfeldt	9c2394f23b	chore: add ISSCyberRiskCrawler	2024-09-30 16:25:20 -07:00
ai.robots.txt	6d9ce1d62a	chore: add sidetrade bot v1.17	2024-09-28 20:58:18 +00:00
Cory Dransfeldt	6a988be27f	chore: add sidetrade bot	2024-09-28 13:58:00 -07:00
ai.robots.txt	632e9d6510	Daily update from Dark Visitors	2024-09-28 01:17:19 +00:00
dark-visitors	7851cea4fd	Daily update from Dark Visitors	2024-09-27 01:18:04 +00:00
Glyn Normington	75343c790e	Merge pull request #38 from urvish-p80/main ... Add an additional resource - README.md	2024-09-27 01:26:04 +01:00
ai.robots.txt	44d975c799	Merge pull request #42 from commoncrawl/main ... feat: make CCBot entry more accurate	2024-09-27 00:21:49 +00:00
Glyn Normington	2f67e77ddb	Merge pull request #42 from commoncrawl/main ... feat: make CCBot entry more accurate	2024-09-27 01:21:37 +01:00
Greg Lindahl	a6de89e6bd	feat: make CCBot entry more accurate	2024-09-26 21:41:28 +00:00
Cory Dransfeldt	60bdfa7eb3	Merge pull request #41 from cityrolr/patch-1 ... Update README.md	2024-09-24 12:53:52 -07:00
Julian Mair	af05890b07	Update README.md ... For people who don't use or don't want to use RSS for this, I've added a little explanation of how to subscribe to releases via GitHub.	2024-09-23 23:27:27 +02:00
Urvish Patel	0106d4b15a	Add additional resource - README.md ... A detailed blogpost to - See the live dashboard showing the websites that are blocking AI Bots such as GPTBot, CCBot, Google-extended and ByteSpider from crawling and scraping the content on their website. Learn which AI crawlers / scrapers do what and how to block them using Robots.txt.	2024-09-23 08:19:27 -04:00
ai.robots.txt	6b8d7f5890	Daily update from Dark Visitors	2024-09-09 01:16:21 +00:00
dark-visitors	5963cbf9f7	Daily update from Dark Visitors	2024-09-08 01:19:31 +00:00
Glyn Normington	b15b8062ce	Merge pull request #36 from cramforce/patch-1 ... Add instructions for AI bot blocking on Vercel	2024-09-08 01:26:07 +01:00
Malte Ubl	809851ae88	Add instructions for AI bot blocking on Vercel	2024-09-07 15:59:25 -07:00
ai.robots.txt	1c1b423684	chore: add iaskspider/2.0 v1.16	2024-09-07 02:05:43 +00:00
Cory Dransfeldt	8373294404	chore: add iaskspider/2.0	2024-09-06 19:05:26 -07:00
Cory Dransfeldt	b30ca5f193	Merge pull request #35 from nisbet-hubbard/patch-7 ... Improve main workflow	2024-09-02 18:40:57 -07:00
ai.robots.txt	fb5c995243	Daily update from Dark Visitors	2024-09-03 01:12:57 +00:00
ai.robots.txt	7151f6c569	Removing previously generated files	2024-09-03 01:12:56 +00:00
nisbet-hubbard	cc18b8617c	Update main.yml	2024-09-03 07:48:48 +08:00
ai.robots.txt	c9325c9e18	Daily update from Dark Visitors	2024-09-02 01:15:07 +00:00
ai.robots.txt	567bd00aec	Removing previously generated files	2024-09-02 01:15:07 +00:00
ai.robots.txt	543e993b08	Daily update from Dark Visitors	2024-09-01 01:24:53 +00:00
ai.robots.txt	01589718df	Removing previously generated files	2024-09-01 01:24:52 +00:00
ai.robots.txt	9a7f556d87	Daily update from Dark Visitors	2024-08-31 01:13:04 +00:00
ai.robots.txt	9a4ebb57ee	Removing previously generated files	2024-08-31 01:13:04 +00:00
ai.robots.txt	054c97ad4f	Daily update from Dark Visitors	2024-08-30 01:13:29 +00:00
ai.robots.txt	b2970316d8	Removing previously generated files	2024-08-30 01:13:29 +00:00
ai.robots.txt	008a34ceb4	chore: add ai2bot v1.15	2024-08-29 03:07:52 +00:00
ai.robots.txt	3bce634e4a	Removing previously generated files	2024-08-29 03:07:51 +00:00
Cory Dransfeldt	0f8723558f	chore: add ai2bot	2024-08-28 20:07:32 -07:00