remote/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2026-09-10 01:54:08 +02:00
Code Issues Projects Releases Packages Wiki Activity
746 commits 7 branches 54 tags 2.3 MiB
Commit graph

73 commits

Author SHA1 Message Date
ai.robots.txt
197f156dd9 Merge pull request #286 from SamHartleyFixes/case-insensitive-matching 
Match generated nginx/lighttpd/Caddy configs case-insensitively
 2026-09-08 05:19:31 +00:00
ai.robots.txt
2acefa38cc Merge pull request #274 from andyman01/add-qwen-ernie-doubao-mistralindex 
Add four AI-lab crawlers named by Dutch news publishers
 2026-09-07 03:41:26 +00:00
ai.robots.txt
f76020b4e4 Merge pull request #277 from andyman01/add-kimi-mistral-training 
Add KimiBot, Kimi-SearchBot and MistralAI-Training
 2026-09-06 03:57:23 +00:00
ai.robots.txt
da24dd28d1 Merge pull request #281 from andyman01/add-oai-adsbot 
Add OAI-AdsBot
 2026-09-06 03:56:13 +00:00
andyman01
ed6af83530 Add Diffbot-User 
Diffbot documents two separate user agents. The list currently has only
the proactive crawler; this adds the user-triggered one.

From Diffbot's own documentation:

  Diffbot-User — This is used by requests originating on behalf of a
  human user browsing a URL using Diffbot software, in response to
  their input.

Source: https://docs.diffbot.com/docs/does-crawl-respect-robotstxt

Same distinction as GPTBot / ChatGPT-User and PerplexityBot /
Perplexity-User, so it is classified as an AI Assistant fetching only
when prompted by a user.

Observed in the robots.txt of five independent publishers in four
countries: nytimes.com, bbc.co.uk, lefigaro.fr, corriere.it and
usatoday.com.

Generated files updated with code/robots.py --convert.
 2026-08-29 20:34:05 +02:00
ai.robots.txt
738c80df21 Merge pull request #269 from henriquejsza/add/reflectionbot 
Add Reflectionbot to robots.json
 2026-08-22 09:33:25 +00:00
ai.robots.txt
ff88a7fcb6 Merge pull request #266 from MLuc24/add-exasearchbot 
Add ExaSearchBot to robots.json
 2026-08-17 16:39:01 +00:00
ai.robots.txt
c7c888c60e Merge pull request #263 from acook/patch-1 
add Lightpanda user agent
 2026-08-17 02:39:02 +00:00
Ilyan321
51a49494aa fix(code): use word boundaries \b in list_to_pcre to prevent false positive matches like NotCursor 2026-08-05 17:02:56 +05:00
Ilyan321
26510d4cd5 Merge branch 'main' of https://github.com/ai-robots-txt/ai.robots.txt into fix/issue-257-regex-anchoring 2026-08-05 16:58:22 +05:00
ai.robots.txt
5994ca0b33 Merge pull request #259 from fork-graveyard/main 
do not needlessly escape hypens, make nginx matches case-sensitive, further minify regexps
 2026-08-05 11:33:22 +00:00
Ilyan321
b06780542b fix(code): unanchor PCRE patterns to allow matching real user-agent headers (#257) 2026-08-04 23:14:02 +05:00
ai.robots.txt
24c010e668 Merge pull request #251 from glyn/delete-IbouBot 
Delete IbouBot
 2026-07-31 15:01:58 +00:00
Dhruv Maniya
49669178ef fix: configure exact user-agent matching 
Signed-off-by: Dhruv Maniya <dhruvmaniya1998@gmail.com>
 2026-07-29 21:47:42 +05:30
Dhruv Maniya
af31418609 Anchor the generic Spider user agent 
Signed-off-by: Dhruv Maniya <dhruvmaniya1998@gmail.com>
 2026-07-15 13:04:57 +05:30
ai.robots.txt
2a83168bbb Merge pull request #240 from bobmatyas/add/tabstack 
Add: Mozilla-Tabstack
 2026-07-09 15:01:12 +00:00
Known Agents
a71b27a232 Update from knownagents.com 2026-07-09 02:04:28 +00:00
dark-visitors
a394130efd Update from Dark Visitors 2026-06-26 02:32:55 +00:00
dark-visitors
3091ad0a23 Update from Dark Visitors 2026-06-16 02:52:42 +00:00
dark-visitors
f420408eee Update from Dark Visitors 2026-06-04 02:48:49 +00:00
ai.robots.txt
000552c2df Merge pull request #232 from justinribeiro/main 
add: AgentTimes bot
 2026-06-03 08:56:38 +00:00
dark-visitors
ee3a96ee5c Update from Dark Visitors 2026-05-22 02:33:30 +00:00
cfcryptotrader-rgb
8d5a081a16 Keep generated robot files in dark visitors update 2026-05-13 00:44:06 -03:00
ai.robots.txt
8b0bc322f4 Merge pull request #227 from gsauthof/naget-news 
Add NagetBot and newsai user-agents
 2026-05-02 15:05:09 +00:00
ai.robots.txt
198653b59a Update from Dark Visitors 2026-03-27 01:26:26 +00:00
ai.robots.txt
86d582b11c Update from Dark Visitors 2026-03-07 01:12:48 +00:00
ai.robots.txt
48bfefbc5f Update from Dark Visitors 2026-02-14 01:16:06 +00:00
ai.robots.txt
aa8519ec10 Update from Dark Visitors 2025-12-21 01:07:06 +00:00
ai.robots.txt
8b8bf9da5d Update from Dark Visitors 2025-12-06 00:58:16 +00:00
ai.robots.txt
7598d77e4a Update from Dark Visitors 2025-12-04 01:10:02 +00:00
ai.robots.txt
3fadc88a23 Merge pull request #206 from newbold/main 
Adding LAIONDownloader
 2025-11-29 10:00:52 +00:00
ai.robots.txt
be4d74412c Merge pull request #194 from glyn/193-kendra 
delete extraneous hyphen
 2025-11-21 16:32:11 +00:00
ai.robots.txt
e2726ac160 Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo 
chore: adds NotebookLM and KlaviyoAIBot agents
 2025-11-14 21:01:25 +00:00
ai.robots.txt
443dd27527 Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots 
chore(robots.json): add AmazonBuyForMe and atlassian-bot
 2025-11-05 23:50:59 +00:00
ai.robots.txt
e87eb706e3 Merge pull request #188 from ai-robots-txt/cdransf/buddybot 
chore: adds BuddyBot
 2025-11-04 05:16:28 +00:00
ai.robots.txt
5cad0ee389 Merge pull request #187 from ai-robots-txt/cdransf/add-Linguee-Bot 
chore: adds Linguee Bot
 2025-10-24 18:05:01 +00:00
ai.robots.txt
646ab08e15 Merge pull request #185 from ai-robots-txt/cdransf/add-IbouBot 
chore: adds IbouBot
 2025-10-21 00:09:24 +00:00
ai.robots.txt
2fa0e9119c Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare 
chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot
 2025-10-20 14:34:43 +00:00
ai.robots.txt
260f5029fe Update from Dark Visitors 2025-10-16 00:56:33 +00:00
ai.robots.txt
38d60b928c Merge pull request #179 from ai-robots-txt/deepseekbot 
chore(robots.json): add DeepSeekBot
 2025-09-25 10:17:38 +00:00
ai.robots.txt
8906f6b447 Update from Dark Visitors 2025-09-12 00:53:06 +00:00
ai.robots.txt
b6338ddc73 Merge pull request #176 from emersion/TerraCotta 
Add TerraCotta
 2025-09-10 14:11:16 +00:00
ai.robots.txt
229d1b4dbc Merge pull request #174 from karolyi/master 
Update Brightbot operator and details; add meta-webindexer entry
 2025-09-09 02:45:32 +00:00
ai.robots.txt
2ad1c3e831 Merge pull request #168 from ai-robots-txt/google-firebase-shapbot 
chore: add Google-Firebase and ShapBot
 2025-08-27 10:13:17 +00:00
ai.robots.txt
6d3f3e1712 Merge pull request #167 from ai-robots-txt/OpenAI-bot 
chore: adds OpenAI but user agent
 2025-08-14 01:45:05 +00:00
ai.robots.txt
ff9fc26404 Update from Dark Visitors 2025-08-01 01:12:53 +00:00
ai.robots.txt
64f9d6ce9c Update from Dark Visitors 2025-07-30 01:05:21 +00:00
ai.robots.txt
9565c11d4c Merge pull request #166 from ai-robots-txt/summaly-bot 
fix: remove summaly
 2025-07-28 19:19:15 +00:00
ai.robots.txt
27420d6fed Merge pull request #164 from GlitzSmarter/main 
Adding YaK
 2025-07-28 18:29:57 +00:00
ai.robots.txt
47769c8429 Update from Dark Visitors 2025-07-17 01:04:20 +00:00