60 commits

Author	SHA1	Message	Date
Dhruv Maniya	49669178ef	fix: configure exact user-agent matching ... Signed-off-by: Dhruv Maniya <dhruvmaniya1998@gmail.com>	2026-07-29 21:47:42 +05:30
Dhruv Maniya	af31418609	Anchor the generic Spider user agent ... Signed-off-by: Dhruv Maniya <dhruvmaniya1998@gmail.com>	2026-07-15 13:04:57 +05:30
ai.robots.txt	2a83168bbb	Merge pull request #240 from bobmatyas/add/tabstack ... Add: Mozilla-Tabstack	2026-07-09 15:01:12 +00:00
Known Agents	a71b27a232	Update from knownagents.com	2026-07-09 02:04:28 +00:00
dark-visitors	a394130efd	Update from Dark Visitors	2026-06-26 02:32:55 +00:00
dark-visitors	3091ad0a23	Update from Dark Visitors	2026-06-16 02:52:42 +00:00
dark-visitors	f420408eee	Update from Dark Visitors	2026-06-04 02:48:49 +00:00
ai.robots.txt	000552c2df	Merge pull request #232 from justinribeiro/main ... add: AgentTimes bot	2026-06-03 08:56:38 +00:00
dark-visitors	ee3a96ee5c	Update from Dark Visitors	2026-05-22 02:33:30 +00:00
cfcryptotrader-rgb	8d5a081a16	Keep generated robot files in dark visitors update	2026-05-13 00:44:06 -03:00
ai.robots.txt	8b0bc322f4	Merge pull request #227 from gsauthof/naget-news ... Add NagetBot and newsai user-agents	2026-05-02 15:05:09 +00:00
ai.robots.txt	198653b59a	Update from Dark Visitors	2026-03-27 01:26:26 +00:00
ai.robots.txt	86d582b11c	Update from Dark Visitors	2026-03-07 01:12:48 +00:00
ai.robots.txt	48bfefbc5f	Update from Dark Visitors	2026-02-14 01:16:06 +00:00
ai.robots.txt	aa8519ec10	Update from Dark Visitors	2025-12-21 01:07:06 +00:00
ai.robots.txt	8b8bf9da5d	Update from Dark Visitors	2025-12-06 00:58:16 +00:00
ai.robots.txt	7598d77e4a	Update from Dark Visitors	2025-12-04 01:10:02 +00:00
ai.robots.txt	3fadc88a23	Merge pull request #206 from newbold/main ... Adding LAIONDownloader	2025-11-29 10:00:52 +00:00
ai.robots.txt	be4d74412c	Merge pull request #194 from glyn/193-kendra ... delete extraneous hyphen	2025-11-21 16:32:11 +00:00
ai.robots.txt	e2726ac160	Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo ... chore: adds NotebookLM and KlaviyoAIBot agents	2025-11-14 21:01:25 +00:00
ai.robots.txt	443dd27527	Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots ... chore(robots.json): add AmazonBuyForMe and atlassian-bot	2025-11-05 23:50:59 +00:00
ai.robots.txt	e87eb706e3	Merge pull request #188 from ai-robots-txt/cdransf/buddybot ... chore: adds BuddyBot	2025-11-04 05:16:28 +00:00
ai.robots.txt	5cad0ee389	Merge pull request #187 from ai-robots-txt/cdransf/add-Linguee-Bot ... chore: adds Linguee Bot	2025-10-24 18:05:01 +00:00
ai.robots.txt	646ab08e15	Merge pull request #185 from ai-robots-txt/cdransf/add-IbouBot ... chore: adds IbouBot	2025-10-21 00:09:24 +00:00
ai.robots.txt	2fa0e9119c	Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare ... chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot	2025-10-20 14:34:43 +00:00
ai.robots.txt	260f5029fe	Update from Dark Visitors	2025-10-16 00:56:33 +00:00
ai.robots.txt	38d60b928c	Merge pull request #179 from ai-robots-txt/deepseekbot ... chore(robots.json): add DeepSeekBot	2025-09-25 10:17:38 +00:00
ai.robots.txt	8906f6b447	Update from Dark Visitors	2025-09-12 00:53:06 +00:00
ai.robots.txt	b6338ddc73	Merge pull request #176 from emersion/TerraCotta ... Add TerraCotta	2025-09-10 14:11:16 +00:00
ai.robots.txt	229d1b4dbc	Merge pull request #174 from karolyi/master ... Update Brightbot operator and details; add meta-webindexer entry	2025-09-09 02:45:32 +00:00
ai.robots.txt	2ad1c3e831	Merge pull request #168 from ai-robots-txt/google-firebase-shapbot ... chore: add Google-Firebase and ShapBot	2025-08-27 10:13:17 +00:00
ai.robots.txt	6d3f3e1712	Merge pull request #167 from ai-robots-txt/OpenAI-bot ... chore: adds OpenAI but user agent	2025-08-14 01:45:05 +00:00
ai.robots.txt	ff9fc26404	Update from Dark Visitors	2025-08-01 01:12:53 +00:00
ai.robots.txt	64f9d6ce9c	Update from Dark Visitors	2025-07-30 01:05:21 +00:00
ai.robots.txt	9565c11d4c	Merge pull request #166 from ai-robots-txt/summaly-bot ... fix: remove summaly	2025-07-28 19:19:15 +00:00
ai.robots.txt	27420d6fed	Merge pull request #164 from GlitzSmarter/main ... Adding YaK	2025-07-28 18:29:57 +00:00
ai.robots.txt	47769c8429	Update from Dark Visitors	2025-07-17 01:04:20 +00:00
ai.robots.txt	567e94a6cf	Merge pull request #160 from sdavids/users/sdavids/feat/htaccess ... Simplify htaccess rewrite rule	2025-07-08 16:00:16 +00:00
ai.robots.txt	63f1f56307	Merge pull request #158 from ai-robots-txt/thinkbot-summaly ... chore(robots.json): add SummalyBot + Thinkbot	2025-07-07 16:55:31 +00:00
ai.robots.txt	20a242d390	Merge pull request #156 from nisbet-hubbard/patch-9 ... Update funtion and description for SemrushBot	2025-07-02 15:58:19 +00:00
ai.robots.txt	2bfcee794d	Update from Dark Visitors	2025-06-29 01:07:47 +00:00
ai.robots.txt	5326c202b5	Merge pull request #154 from paulrudy/main ... re-add facebookexternalhit	2025-06-16 15:12:42 +00:00
ai.robots.txt	eb05f2f527	Merge pull request #153 from sergiospagnuolo/Poseidon ... Update robots.json with new crawler	2025-06-14 14:04:03 +00:00
ai.robots.txt	e53d81c66d	Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ... chore(robots.json): adds MyCentralAIScraperBot	2025-06-13 09:28:41 +00:00
ai.robots.txt	842e2256e8	Merge pull request #150 from ai-robots-txt/semrush-bots ... chore(robots.json): adds additional SemrushBot user agents	2025-06-12 07:12:00 +00:00
ai.robots.txt	3759a6bf14	chore(robots.json): adds EchoboxBot (#148)	2025-06-09 15:44:36 +00:00
ai.robots.txt	ac7ed17e71	Merge pull request #145 from ai-robots-txt/aws-bedrockbot ... chore(robots.json): adds bedrockbot	2025-06-05 16:51:17 +00:00
ai.robots.txt	75ea75a95b	Merge pull request #143 from ai-robots-txt/panscient ... chore(robots.json): adds Panscient	2025-06-04 18:04:06 +00:00
ai.robots.txt	9c28c63a0c	Merge pull request #142 from ai-robots-txt/quillbot ... chore(robots.json): adds Quillbot	2025-06-04 17:54:57 +00:00
ai.robots.txt	3efabc603d	Merge pull request #141 from Ivan-Chupin/patch-1 ... Add SBIntuitionsBot	2025-06-03 23:28:48 +00:00