677 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
ai.robots.txt	24c010e668	Merge pull request #251 from glyn/delete-IbouBot ... v1.49 Delete IbouBot	2026-07-31 15:01:58 +00:00
Glyn Normington	1be8fa6220	Merge pull request #251 from glyn/delete-IbouBot ... Delete IbouBot	2026-07-31 16:01:41 +01:00
Glyn Normington	69a87566de	Delete IbouBot ... Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/250	2026-07-31 13:04:31 +01:00
Glyn Normington	042d00a5ab	Merge pull request #249 from ai-robots-txt/test-ci ... Empty commit to test CI	2026-07-31 03:35:15 +01:00
Glyn Normington	c96a9916cc	Empty commit to test CI	2026-07-31 03:34:01 +01:00
Glyn Normington	80c19fcad5	Revert "Fill operator and respect from primary docs for 6 entries" ... This reverts commit e62a95bb8a. Reverted because of CI failure noted in PR.	2026-07-31 03:30:04 +01:00
Glyn Normington	bbe0579a23	Merge pull request #248 from INXPRNCD/fill-unclear-operator-and-respect-from-primary-docs ... Fill "Unclear at this time." from primary operator docs for 6 entries	2026-07-31 03:20:58 +01:00
Known Agents	2155906baa	Update from knownagents.com	2026-07-31 01:57:31 +00:00
Erdinc Özden	e62a95bb8a	Fill operator and respect from primary docs for 6 entries ... Applebot, Meta-ExternalAgent, Meta-ExternalFetcher, meta-externalfetcher, DuckAssistBot and Google-Agent carried 'Unclear at this time.' in both operator and respect, but each operator documents the behaviour on its own site. Values are linked to the primary source inline, matching the style already used by Applebot-Extended and meta-externalagent. robots.txt output is unchanged — no keys added or renamed.	2026-07-30 22:05:49 +02:00
Glyn Normington	3db35be22d	Merge pull request #241 from iamdhrv/fix/exact-spider-user-agent ... v1.48 Anchor generic Spider user-agent match	2026-07-30 12:15:55 +01:00
Dhruv Maniya	49669178ef	fix: configure exact user-agent matching ... Signed-off-by: Dhruv Maniya <dhruvmaniya1998@gmail.com>	2026-07-29 21:47:42 +05:30
Dhruv Maniya	af31418609	Anchor the generic Spider user agent ... Signed-off-by: Dhruv Maniya <dhruvmaniya1998@gmail.com>	2026-07-15 13:04:57 +05:30
Known Agents	a0fed45f0e	Update from knownagents.com	2026-07-10 02:03:20 +00:00
ai.robots.txt	2a83168bbb	Merge pull request #240 from bobmatyas/add/tabstack ... Add: Mozilla-Tabstack	2026-07-09 15:01:12 +00:00
Glyn Normington	51f7fd5e3a	Merge pull request #240 from bobmatyas/add/tabstack ... Add: Mozilla-Tabstack	2026-07-09 16:00:59 +01:00
Known Agents	a71b27a232	Update from knownagents.com	2026-07-09 02:04:28 +00:00
bob	c793fa0a8c	Add: Mozilla-Tabstack. Closes #226	2026-07-08 21:16:52 -04:00
Glyn Normington	f030cbe956	Merge pull request #238 from ghking/rename-dark-visitors-to-known-agents ... v1.47 Renaming Dark Visitors to Known Agents	2026-07-09 01:41:52 +01:00
Glyn Normington	12a15a3a28	Update code/robots.py ... For consistency.	2026-07-09 01:40:00 +01:00
Glyn Normington	5065ca6055	Update .github/workflows/ai_robots_update.yml ... For consistency.	2026-07-09 01:39:50 +01:00
Glyn Normington	0b021b9e40	Update code/robots.py ... For consistency.	2026-07-09 01:39:39 +01:00
Gavin King	dad7db5168	Fixed broken URLs	2026-07-08 20:38:48 -04:00
Gavin King	e1bf273b16	Update robots.py	2026-07-08 20:26:48 -04:00
Gavin King	a35e6f9a09	Update ai_robots_update.yml	2026-07-08 20:23:53 -04:00
Gavin King	c43c5afd07	More name changes	2026-07-08 15:43:07 -04:00
Gavin King	0c5de5b90b	Updating the name	2026-07-08 15:30:25 -04:00
dark-visitors	a394130efd	Update from Dark Visitors	2026-06-26 02:32:55 +00:00
Glyn Normington	ee4ae068c2	Merge pull request #236 from guest20/patch-2 ... robots.py: fix darkvisiters descriptions and links	2026-06-23 11:15:59 +01:00
Glyn Normington	82df5f7659	Apply suggestions from code review ... Co-authored-by: Glyn Normington <work@underlap.org>	2026-06-23 11:13:17 +01:00
guest20	ac207f9947	tests.py: )	2026-06-23 04:17:14 +02:00
guest20	ffad9293c4	tests.py: consolidate	2026-06-23 04:13:17 +02:00
guest20	80357baf9c	robots.py: pass everything to consolodate	2026-06-21 23:51:32 +02:00
guest20	7dbe93295e	robots.py: pull consolodate out	2026-06-21 23:47:11 +02:00
guest20	3f965b292d	robots.py: keep new, non-default values ... Co-authored-by: Glyn Normington <work@underlap.org>	2026-06-20 18:29:47 +02:00
dark-visitors	5cb3ec765f	Update from Dark Visitors	2026-06-17 11:21:12 +00:00
guest20	25415e342c	robots.py: div class=description	2026-06-17 13:20:29 +02:00
dark-visitors	118bb772c0	Update from Dark Visitors	2026-06-17 11:01:38 +00:00
guest20	8dd8e6c4e4	robots.py: possibly ruin the whole purpose of consolodate ... Replace an old non-default value with a new non-default value, so updates happen.	2026-06-17 13:00:15 +02:00
guest20	78fd7b49d8	\robots.py	2026-06-17 12:23:10 +02:00
guest20	101d7ac0ad	robots.py: punked by auto-) again	2026-06-17 12:10:05 +02:00
guest20	f2147fd81c	robots.py: fix darkvistors link	2026-06-17 12:06:50 +02:00
guest20	bd3ee4702c	robots.py: agents/agents -> agents	2026-06-17 11:45:55 +02:00
dark-visitors	3091ad0a23	Update from Dark Visitors	2026-06-16 02:52:42 +00:00
dark-visitors	f420408eee	Update from Dark Visitors	2026-06-04 02:48:49 +00:00
ai.robots.txt	000552c2df	Merge pull request #232 from justinribeiro/main ... add: AgentTimes bot	2026-06-03 08:56:38 +00:00
Glyn Normington	ef50defc1a	Merge pull request #232 from justinribeiro/main ... add: AgentTimes bot	2026-06-03 09:56:26 +01:00
Justin Ribeiro	7673437733	add: AgentTimes bot	2026-06-02 11:26:06 -07:00
dark-visitors	ee3a96ee5c	Update from Dark Visitors	2026-05-22 02:33:30 +00:00
Glyn Normington	1fbf7a06ac	Merge pull request #231 from cfcryptotrader-rgb/cfcryptotrader-sync-generated-robots ... v1.46 Keep generated robot files synced after Dark Visitors update	2026-05-13 07:44:48 +01:00
cfcryptotrader-rgb	8d5a081a16	Keep generated robot files in dark visitors update	2026-05-13 00:44:06 -03:00