ai.robots.txt

remote/ai.robots.txt

Fork 0

mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2026-10-25 17:56:58 +01:00

Commit graph

cfa3031df9

Merge c02a9627e1 into 3ef52d9032 Damien Sorel 2026-10-25 13:30:22 +00:00
c02a9627e1 Add ElevenlabsCrawler Damien Sorel 2026-10-25 14:27:30 +01:00
3ef52d9032 Update from knownagents.com main Known Agents 2026-10-24 02:37:28 +00:00
ff1c94c914

Merge 4b1b128249 into 1c4694a3ac jamiezseo 2026-10-24 00:26:41 +07:00
4b1b128249 Add State of AI Crawler Access to Related jamiezseo 2026-10-24 00:26:26 +07:00
1c4694a3ac Update from knownagents.com Known Agents 2026-10-23 02:39:54 +00:00
0366df0b3a Update from knownagents.com Known Agents 2026-10-22 02:35:23 +00:00
1e457c6f2b Update from knownagents.com Known Agents 2026-10-21 02:37:38 +00:00
a446ec974c Update from knownagents.com Known Agents 2026-10-20 02:43:51 +00:00
6abe630599 Update from knownagents.com Known Agents 2026-10-18 02:15:30 +00:00
da755f8052 Merge pull request #300 from sbulen/several_adds v2.1 ai.robots.txt 2026-10-17 02:56:34 +00:00
c8cf4f97b3

Merge pull request #300 from sbulen/several_adds Glyn Normington 2026-10-17 03:56:26 +01:00
d0b9f69bec Add YiBot, xAI-Grok, Hunyuan, MoonshotBot Shawn Bulen 2026-10-16 16:59:05 -07:00
0467ca550a Update from knownagents.com Known Agents 2026-10-15 02:28:42 +00:00
cd94d6c4c3 Merge pull request #299 from TriHydera/main ai.robots.txt 2026-10-14 03:43:09 +00:00
3a39226140

Merge pull request #299 from TriHydera/main Glyn Normington 2026-10-14 04:43:00 +01:00
ceae764ab0 ExaSearchBot, Grok and NBot TriHydera 2026-10-13 18:38:06 +01:00
ee24bcfeb7

Enhance README with file links and notes Glyn Normington 2026-10-09 12:21:53 +01:00
9ad8a47e23 Update from knownagents.com Known Agents 2026-10-03 03:02:07 +00:00
e670bcafe6

Merge pull request #294 from Hitesh-XS/add-keenablebot v2.0 Glyn Normington 2026-10-03 00:52:11 +01:00
341d25e239 feat: add KeenableBot SAD LIFE 2026-10-02 10:33:53 +05:30
987266f3c5 Update from knownagents.com Known Agents 2026-09-26 02:41:07 +00:00
5631dee581

Merge pull request #289 from dunn/faq-entries Glyn Normington 2026-09-22 19:04:41 +01:00
1ecb571700 FAQ: add references subheading, convert to unordered list alexandra catalina 2026-09-21 10:54:24 -07:00
57ed273dc7 FAQ: add entries on labor and military use alexandra catalina 2026-09-19 11:13:03 -07:00
6c9c1a3327 Improve wording Glyn Normington 2026-09-19 15:47:08 +01:00
fab855244f

Merge pull request #288 from ai-robots-txt/noAI Glyn Normington 2026-09-19 15:45:13 +01:00
e4c5d5e206 Do not permit AI contributions Glyn Normington 2026-09-19 15:43:27 +01:00
425d1a6207

Merge pull request #287 from flober81/patch-1 Glyn Normington 2026-09-17 15:25:50 +01:00
9f039df105

Shorten entry: drop dated figures, spell out the file types Florian Berger 2026-09-17 14:01:37 +02:00
8136c08d6a

Add AI Discovery Radar to related resources Florian Berger 2026-09-17 10:47:31 +02:00
0e111dcc24

Merge pull request #285 from SamHartleyFixes/match-agent-names-case-insensitively Glyn Normington 2026-09-08 06:20:04 +01:00
197f156dd9 Merge pull request #286 from SamHartleyFixes/case-insensitive-matching ai.robots.txt 2026-09-08 05:19:31 +00:00
ebc94b445f

Merge pull request #286 from SamHartleyFixes/case-insensitive-matching Glyn Normington 2026-09-08 06:19:20 +01:00
86a2e1fc67 Regenerate Caddyfile test fixture for case-insensitive match Sam Hartley 2026-09-07 20:08:07 -07:00
3adc0cb1db Regenerate lighttpd test fixture for case-insensitive match Sam Hartley 2026-09-07 20:08:06 -07:00
aa623355c1 Regenerate nginx test fixture for case-insensitive match Sam Hartley 2026-09-07 20:08:06 -07:00
447bb15a22 Make the nginx/lighttpd/Caddy configs match case-insensitively Sam Hartley 2026-09-07 20:08:05 -07:00
e028137f46 Resolve agent names case-insensitively when ingesting knownagents.com SamHartleyFixes 2026-09-07 18:08:23 -07:00
db83172548

Merge pull request #283 from SamHartleyFixes/ai-crawler-census Glyn Normington 2026-09-07 18:08:07 +01:00
ee9805ef9f Add AI Crawler Census to Related Sam Hartley 2026-09-07 08:09:25 -07:00
2acefa38cc Merge pull request #274 from andyman01/add-qwen-ernie-doubao-mistralindex v1.52 ai.robots.txt 2026-09-07 03:41:26 +00:00
a595115206

Merge pull request #274 from andyman01/add-qwen-ernie-doubao-mistralindex Glyn Normington 2026-09-07 04:41:12 +01:00
48abe91f99 Add four AI-lab crawlers named by Dutch news publishers andyman01 2026-09-06 09:37:40 +02:00
f76020b4e4 Merge pull request #277 from andyman01/add-kimi-mistral-training ai.robots.txt 2026-09-06 03:57:23 +00:00
b509c62249

Merge pull request #277 from andyman01/add-kimi-mistral-training Glyn Normington 2026-09-06 04:57:15 +01:00
da24dd28d1 Merge pull request #281 from andyman01/add-oai-adsbot ai.robots.txt 2026-09-06 03:56:13 +00:00
397d77cd74

Merge pull request #281 from andyman01/add-oai-adsbot Glyn Normington 2026-09-06 04:56:05 +01:00
d37bb86786 Add KimiBot, Kimi-SearchBot and MistralAI-Training andyman01 2026-09-05 15:01:02 +02:00
2d7931778c Add OAI-AdsBot andyman01 2026-09-05 15:05:22 +02:00
f44d4386fc

Merge pull request #279 from saar-twito/main Glyn Normington 2026-09-04 17:00:28 +01:00
a28a34b049

Merge pull request #280 from ai-robots-txt/instructions Glyn Normington 2026-09-04 16:58:06 +01:00
984382f00f Disambiguate contribution instructions Glyn Normington 2026-09-04 16:56:43 +01:00
282ef53f61

Merge pull request #276 from andyman01/add-diffbot-user Glyn Normington 2026-09-04 16:45:26 +01:00
538fc8de8f

Add AI Access Checker to additional resources Saar Twito 2026-09-04 12:11:49 +03:00
ed6af83530 Add Diffbot-User andyman01 2026-08-29 20:34:05 +02:00
ef259563f0 Add 46 AI agent-ecosystem crawlers observed on a public MCP + x402 endpoint roblouw2nd 2026-08-27 23:52:48 +02:00
7cd4c92343

Merge pull request #272 from peppe1337/de-access-index v1.51 Glyn Normington 2026-08-26 23:51:59 +01:00
5e73986001 Apply maintainer's suggestion: trim entry to avoid stale figures Ideenschmiede CEO 2026-08-26 16:02:12 +00:00
3b67c9c54e Add KI-Zugangsindex to Related resources Ideenschmiede CEO 2026-08-25 12:15:02 +00:00
0037a9e2c5 Update from knownagents.com Known Agents 2026-08-25 00:43:16 +00:00
5692a202f0 Merge pull request #271 from lyrenth/correct-aiwebindex-metadata ai.robots.txt 2026-08-24 14:38:04 +00:00
d8e26238a7

Merge pull request #271 from lyrenth/correct-aiwebindex-metadata Glyn Normington 2026-08-24 15:37:53 +01:00
763f194129 Correct AIWebIndex metadata Lyrenth 2026-08-24 18:27:59 +04:00
738c80df21 Merge pull request #269 from henriquejsza/add/reflectionbot ai.robots.txt 2026-08-22 09:33:25 +00:00
0a871b9e62

Merge pull request #269 from henriquejsza/add/reflectionbot Glyn Normington 2026-08-22 10:33:13 +01:00
cfd62dbc3b

Add Reflectionbot to robots.json henriquejsza 2026-08-22 03:58:31 -03:00
2d1d8972c9

Merge pull request #267 from jamiusaliu/test/slcc-real-world-false-positives Glyn Normington 2026-08-20 16:05:40 +01:00
66a8329a61 docs: trim the FAQ entry per review Saliu Jamiu Olamilekan 2026-08-18 21:13:10 -07:00
bcc4e408e2 test: pin the SLCC1/SLCC2 false positive from #208 Saliu Jamiu Olamilekan 2026-08-18 20:28:19 -07:00
6a20f8dbc4 Update from knownagents.com Known Agents 2026-08-18 00:41:50 +00:00
ff88a7fcb6 Merge pull request #266 from MLuc24/add-exasearchbot ai.robots.txt 2026-08-17 16:39:01 +00:00
8e366f564b

Merge pull request #266 from MLuc24/add-exasearchbot Glyn Normington 2026-08-17 17:38:50 +01:00
024c9c2d6b Add ExaSearchBot to robots.json MLuc24 2026-08-17 10:30:41 +07:00
c7c888c60e Merge pull request #263 from acook/patch-1 v1.50 ai.robots.txt 2026-08-17 02:39:02 +00:00
a2858c0255

Merge pull request #263 from acook/patch-1 Glyn Normington 2026-08-17 03:38:51 +01:00
dd30b585f4

Merge pull request #262 from farrelldan/add-bot-ledger-related Glyn Normington 2026-08-16 04:02:44 +01:00
a5d17db8b8

add Lightpanda user agent Anthony M. Cook 2026-08-07 11:22:16 -05:00
abe839e4e0 docs(README): add Bot Ledger to Related resources Dan 2026-08-07 08:12:59 -06:00
2f5d7ccf39

Merge pull request #260 from Ilyan321/fix/issue-257-regex-anchoring Glyn Normington 2026-08-05 18:18:57 +01:00
4c9331ebbb Merge branch 'main' of https://github.com/ai-robots-txt/ai.robots.txt into fix/issue-257-regex-anchoring Ilyan321 2026-08-05 21:56:22 +05:00
6f3054bcf0 Merge pull request #261 from fork-graveyard/main ai.robots.txt 2026-08-05 16:07:54 +00:00
dbd9aa78d4

Merge pull request #261 from fork-graveyard/main Glyn Normington 2026-08-05 17:07:44 +01:00
1177eba00a fix lighttpd config for 1.x girst 2026-08-05 14:43:39 +02:00
5658a6d726 test(code): expand false positive integration test coverage Ilyan321 2026-08-05 17:06:50 +05:00
51a49494aa fix(code): use word boundaries \b in list_to_pcre to prevent false positive matches like NotCursor Ilyan321 2026-08-05 17:02:56 +05:00
26510d4cd5 Merge branch 'main' of https://github.com/ai-robots-txt/ai.robots.txt into fix/issue-257-regex-anchoring Ilyan321 2026-08-05 16:58:22 +05:00
7b1026e69c test(code): add integration test verifying regex does not match non-AI user agents Ilyan321 2026-08-05 16:50:35 +05:00
5994ca0b33 Merge pull request #259 from fork-graveyard/main ai.robots.txt 2026-08-05 11:33:22 +00:00
d93db7b81d

Merge pull request #259 from fork-graveyard/main Glyn Normington 2026-08-05 12:33:05 +01:00
b06780542b fix(code): unanchor PCRE patterns to allow matching real user-agent headers (#257) Ilyan321 2026-08-04 23:14:02 +05:00
0beb4a2666 update test files to match new output girst 2026-08-04 19:25:12 +02:00
8ab5f96c66

robots.py: list_to_prcre ]) -> ] guest20 2026-08-04 14:58:04 +02:00
d98c220be8

robots.py: list_to_pcre join a list of patterns isntead guest20 2026-08-04 14:56:33 +02:00
eed7c67963

robots.py: list_to_prcre - joining strings joins charaacters guest20 2026-08-04 14:48:57 +02:00
a94b2c93e5 further minify regexps girst 2026-08-04 14:44:41 +02:00
9d3a286796

robots.py: list_to_pcre don't overwrite patterns guest20 2026-08-04 14:46:01 +02:00
eaf22ce5bb

robots.py: list_to_pcre gets support for sends_full_name guest20 2026-08-04 14:36:25 +02:00
3bd200ba32 make nginx matches case-sensitive girst 2026-08-04 14:14:14 +02:00
c4b366740e do not needlessly escape hypens girst 2026-08-04 14:13:52 +02:00