46 commits

Author	SHA1	Message	Date
ai.robots.txt	aa8519ec10	Update from Dark Visitors	2025-12-21 01:07:06 +00:00
ai.robots.txt	8b8bf9da5d	Update from Dark Visitors	2025-12-06 00:58:16 +00:00
ai.robots.txt	7598d77e4a	Update from Dark Visitors	2025-12-04 01:10:02 +00:00
ai.robots.txt	3fadc88a23	Merge pull request #206 from newbold/main ... Adding LAIONDownloader	2025-11-29 10:00:52 +00:00
ai.robots.txt	be4d74412c	Merge pull request #194 from glyn/193-kendra ... delete extraneous hyphen	2025-11-21 16:32:11 +00:00
ai.robots.txt	e2726ac160	Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo ... chore: adds NotebookLM and KlaviyoAIBot agents	2025-11-14 21:01:25 +00:00
ai.robots.txt	443dd27527	Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots ... chore(robots.json): add AmazonBuyForMe and atlassian-bot	2025-11-05 23:50:59 +00:00
ai.robots.txt	e87eb706e3	Merge pull request #188 from ai-robots-txt/cdransf/buddybot ... chore: adds BuddyBot	2025-11-04 05:16:28 +00:00
ai.robots.txt	5cad0ee389	Merge pull request #187 from ai-robots-txt/cdransf/add-Linguee-Bot ... chore: adds Linguee Bot	2025-10-24 18:05:01 +00:00
ai.robots.txt	646ab08e15	Merge pull request #185 from ai-robots-txt/cdransf/add-IbouBot ... chore: adds IbouBot	2025-10-21 00:09:24 +00:00
ai.robots.txt	2fa0e9119c	Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare ... chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot	2025-10-20 14:34:43 +00:00
ai.robots.txt	260f5029fe	Update from Dark Visitors	2025-10-16 00:56:33 +00:00
ai.robots.txt	38d60b928c	Merge pull request #179 from ai-robots-txt/deepseekbot ... chore(robots.json): add DeepSeekBot	2025-09-25 10:17:38 +00:00
ai.robots.txt	8906f6b447	Update from Dark Visitors	2025-09-12 00:53:06 +00:00
ai.robots.txt	b6338ddc73	Merge pull request #176 from emersion/TerraCotta ... Add TerraCotta	2025-09-10 14:11:16 +00:00
ai.robots.txt	229d1b4dbc	Merge pull request #174 from karolyi/master ... Update Brightbot operator and details; add meta-webindexer entry	2025-09-09 02:45:32 +00:00
ai.robots.txt	2ad1c3e831	Merge pull request #168 from ai-robots-txt/google-firebase-shapbot ... chore: add Google-Firebase and ShapBot	2025-08-27 10:13:17 +00:00
ai.robots.txt	6d3f3e1712	Merge pull request #167 from ai-robots-txt/OpenAI-bot ... chore: adds OpenAI but user agent	2025-08-14 01:45:05 +00:00
ai.robots.txt	ff9fc26404	Update from Dark Visitors	2025-08-01 01:12:53 +00:00
ai.robots.txt	64f9d6ce9c	Update from Dark Visitors	2025-07-30 01:05:21 +00:00
ai.robots.txt	9565c11d4c	Merge pull request #166 from ai-robots-txt/summaly-bot ... fix: remove summaly	2025-07-28 19:19:15 +00:00
ai.robots.txt	27420d6fed	Merge pull request #164 from GlitzSmarter/main ... Adding YaK	2025-07-28 18:29:57 +00:00
ai.robots.txt	47769c8429	Update from Dark Visitors	2025-07-17 01:04:20 +00:00
ai.robots.txt	567e94a6cf	Merge pull request #160 from sdavids/users/sdavids/feat/htaccess ... Simplify htaccess rewrite rule	2025-07-08 16:00:16 +00:00
ai.robots.txt	63f1f56307	Merge pull request #158 from ai-robots-txt/thinkbot-summaly ... chore(robots.json): add SummalyBot + Thinkbot	2025-07-07 16:55:31 +00:00
ai.robots.txt	20a242d390	Merge pull request #156 from nisbet-hubbard/patch-9 ... Update funtion and description for SemrushBot	2025-07-02 15:58:19 +00:00
ai.robots.txt	2bfcee794d	Update from Dark Visitors	2025-06-29 01:07:47 +00:00
ai.robots.txt	5326c202b5	Merge pull request #154 from paulrudy/main ... re-add facebookexternalhit	2025-06-16 15:12:42 +00:00
ai.robots.txt	eb05f2f527	Merge pull request #153 from sergiospagnuolo/Poseidon ... Update robots.json with new crawler	2025-06-14 14:04:03 +00:00
ai.robots.txt	e53d81c66d	Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ... chore(robots.json): adds MyCentralAIScraperBot	2025-06-13 09:28:41 +00:00
ai.robots.txt	842e2256e8	Merge pull request #150 from ai-robots-txt/semrush-bots ... chore(robots.json): adds additional SemrushBot user agents	2025-06-12 07:12:00 +00:00
ai.robots.txt	3759a6bf14	chore(robots.json): adds EchoboxBot (#148)	2025-06-09 15:44:36 +00:00
ai.robots.txt	ac7ed17e71	Merge pull request #145 from ai-robots-txt/aws-bedrockbot ... chore(robots.json): adds bedrockbot	2025-06-05 16:51:17 +00:00
ai.robots.txt	75ea75a95b	Merge pull request #143 from ai-robots-txt/panscient ... chore(robots.json): adds Panscient	2025-06-04 18:04:06 +00:00
ai.robots.txt	9c28c63a0c	Merge pull request #142 from ai-robots-txt/quillbot ... chore(robots.json): adds Quillbot	2025-06-04 17:54:57 +00:00
ai.robots.txt	3efabc603d	Merge pull request #141 from Ivan-Chupin/patch-1 ... Add SBIntuitionsBot	2025-06-03 23:28:48 +00:00
ai.robots.txt	080946c360	Merge pull request #140 from ai-robots-txt/yandex-bots ... chore(robots.json): adds YandexAdditional crawlers	2025-06-03 19:51:25 +00:00
ai.robots.txt	d239e7e5ad	Merge pull request #139 from ai-robots-txt/workflow-fix ... chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard	2025-06-03 01:52:35 +00:00
ai.robots.txt	093ab81d78	Update from Dark Visitors	2025-05-23 00:58:57 +00:00
ai.robots.txt	fedb658cc0	Merge pull request #133 from ai-robots-txt/wpbot ... chore(robots.json): adds wpbot	2025-05-21 21:06:05 +00:00
ai.robots.txt	7c5389f4a0	Merge pull request #98 from kylebuckingham/main ... Updating Claude Bots	2025-05-21 19:00:23 +00:00
ai.robots.txt	1c2acd75b7	Merge pull request #126 from ai-robots-txt/mistral-bot ... chore(robots.json): adds MistralAI-User/1.0 crawler	2025-05-21 15:27:26 +00:00
ai.robots.txt	dd1ed174b7	Merge pull request #129 from ai-robots-txt/google-cloudvertexbot ... chore(robots.json): adds Google-CloudVertexBot	2025-05-16 11:35:15 +00:00
ai.robots.txt	1c470babbe	Merge pull request #123 from joehoyle/patch-1 ... Fix JSON syntax error	2025-05-15 16:12:30 +00:00
ai.robots.txt	678380727e	Merge pull request #115 from glyn/syntax ... Fix Python syntax error	2025-05-01 10:29:06 +00:00
Crazyroostereye	1310dbae46	Added a Caddyfile converter (#110) ... Co-authored-by: Julian Beittel <julian@beittel.net> Co-authored-by: Glyn Normington <work@underlap.org>	2025-05-01 11:21:32 +01:00