650 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Glyn Normington	ee4ae068c2	Merge pull request #236 from guest20/patch-2 ... robots.py: fix darkvisiters descriptions and links	2026-06-23 11:15:59 +01:00
Glyn Normington	82df5f7659	Apply suggestions from code review ... Co-authored-by: Glyn Normington <work@underlap.org>	2026-06-23 11:13:17 +01:00
guest20	ac207f9947	tests.py: )	2026-06-23 04:17:14 +02:00
guest20	ffad9293c4	tests.py: consolidate	2026-06-23 04:13:17 +02:00
guest20	80357baf9c	robots.py: pass everything to consolodate	2026-06-21 23:51:32 +02:00
guest20	7dbe93295e	robots.py: pull consolodate out	2026-06-21 23:47:11 +02:00
guest20	3f965b292d	robots.py: keep new, non-default values ... Co-authored-by: Glyn Normington <work@underlap.org>	2026-06-20 18:29:47 +02:00
dark-visitors	5cb3ec765f	Update from Dark Visitors	2026-06-17 11:21:12 +00:00
guest20	25415e342c	robots.py: div class=description	2026-06-17 13:20:29 +02:00
dark-visitors	118bb772c0	Update from Dark Visitors	2026-06-17 11:01:38 +00:00
guest20	8dd8e6c4e4	robots.py: possibly ruin the whole purpose of consolodate ... Replace an old non-default value with a new non-default value, so updates happen.	2026-06-17 13:00:15 +02:00
guest20	78fd7b49d8	\robots.py	2026-06-17 12:23:10 +02:00
guest20	101d7ac0ad	robots.py: punked by auto-) again	2026-06-17 12:10:05 +02:00
guest20	f2147fd81c	robots.py: fix darkvistors link	2026-06-17 12:06:50 +02:00
guest20	bd3ee4702c	robots.py: agents/agents -> agents	2026-06-17 11:45:55 +02:00
dark-visitors	3091ad0a23	Update from Dark Visitors	2026-06-16 02:52:42 +00:00
dark-visitors	f420408eee	Update from Dark Visitors	2026-06-04 02:48:49 +00:00
ai.robots.txt	000552c2df	Merge pull request #232 from justinribeiro/main ... add: AgentTimes bot	2026-06-03 08:56:38 +00:00
Glyn Normington	ef50defc1a	Merge pull request #232 from justinribeiro/main ... add: AgentTimes bot	2026-06-03 09:56:26 +01:00
Justin Ribeiro	7673437733	add: AgentTimes bot	2026-06-02 11:26:06 -07:00
dark-visitors	ee3a96ee5c	Update from Dark Visitors	2026-05-22 02:33:30 +00:00
Glyn Normington	1fbf7a06ac	Merge pull request #231 from cfcryptotrader-rgb/cfcryptotrader-sync-generated-robots ... v1.46 Keep generated robot files synced after Dark Visitors update	2026-05-13 07:44:48 +01:00
cfcryptotrader-rgb	8d5a081a16	Keep generated robot files in dark visitors update	2026-05-13 00:44:06 -03:00
dark-visitors	aea7db9e34	Update from Dark Visitors	2026-05-13 02:24:20 +00:00
Glyn Normington	767fcc6f15	Merge pull request #229 from axeleroy/main ... Add missing bot categories	2026-05-12 04:33:19 +01:00
Axel Leroy	6adc1f7b53	Add missing bot categories	2026-05-11 15:42:56 +02:00
dark-visitors	851ce068cf	Update from Dark Visitors	2026-05-03 02:01:44 +00:00
ai.robots.txt	8b0bc322f4	Merge pull request #227 from gsauthof/naget-news ... Add NagetBot and newsai user-agents	2026-05-02 15:05:09 +00:00
Glyn Normington	7244a7e870	Merge pull request #227 from gsauthof/naget-news ... Add NagetBot and newsai user-agents	2026-05-02 16:04:59 +01:00
Georg Sauthoff	a9f686064f	Add NagetBot and newsai user-agents ... User-agent header values seen in the wild over the last days: Mozilla/5.0 (compatible; NagetBot/1.0; +https://naget.ai/bot) Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 newsai/1.0 Safari/537.36	2026-05-02 13:04:51 +02:00
ai.robots.txt	198653b59a	Update from Dark Visitors	2026-03-27 01:26:26 +00:00
dark-visitors	ad47cafc97	Update from Dark Visitors	2026-03-26 01:26:23 +00:00
dark-visitors	2d2ec4ce8d	Update from Dark Visitors	2026-03-25 01:20:42 +00:00
ai.robots.txt	86d582b11c	Update from Dark Visitors	2026-03-07 01:12:48 +00:00
dark-visitors	3de8865f00	Update from Dark Visitors	2026-03-06 01:21:01 +00:00
Glyn Normington	243ec6b67d	Merge pull request #216 from flymarq/lighttpd ... v1.45 Lighttpd	2026-02-17 17:28:34 +00:00
Flynn Marquardt	793b0454f2	Use adverb.	2026-02-15 11:20:34 +01:00
Flynn Marquardt	790bff1817	Add reference file for lighttpd test.	2026-02-15 11:18:35 +01:00
Flynn Marquardt	742f9b9b5e	Add lighttpd test.	2026-02-15 11:17:54 +01:00
ai.robots.txt	48bfefbc5f	Update from Dark Visitors	2026-02-14 01:16:06 +00:00
Flynn Marquardt	5f0dd97ccb	Add lighttpd sample file.	2026-02-13 19:23:19 +01:00
Flynn Marquardt	55d404c115	Fix typos.	2026-02-13 19:21:31 +01:00
Flynn Marquardt	deae8eed2d	Update with lighttpd instructions.	2026-02-13 19:20:12 +01:00
Flynn Marquardt	70e83e6056	Add generation and output of lighttpd configuration fragment.	2026-02-13 19:18:46 +01:00
dark-visitors	a762015e58	Update from Dark Visitors	2026-02-13 01:23:17 +00:00
ai.robots.txt	aa8519ec10	Update from Dark Visitors	2025-12-21 01:07:06 +00:00
dark-visitors	83485effdb	Update from Dark Visitors	2025-12-20 00:58:49 +00:00
ai.robots.txt	8b8bf9da5d	Update from Dark Visitors	2025-12-06 00:58:16 +00:00
dark-visitors	f1c752ef12	Update from Dark Visitors	2025-12-05 01:00:44 +00:00
Adam Newbold	51afa7113a	Update ai_robots_update.yml with rebase command to fix scheduled run	2025-12-03 20:15:10 -05:00