remote/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2026-08-31 13:24:05 +02:00
Code Issues Projects Releases Packages Wiki Activity

Commit graph

  • 2299cddfd5
         Merge ed6af83530 into 7cd4c92343 andyman01 2026-08-29 20:34:48 +02:00
  • ed6af83530 Add Diffbot-User andyman01 2026-08-29 20:34:05 +02:00
  • 416621bf53
         Merge ef259563f0 into 7cd4c92343 roblouw2nd 2026-08-27 23:53:22 +02:00
  • ef259563f0 Add 46 AI agent-ecosystem crawlers observed on a public MCP + x402 endpoint roblouw2nd 2026-08-27 23:52:48 +02:00
  • 23b0e7b3b9
         Merge 79ef41e4ee into 7cd4c92343 andyman01 2026-08-27 22:47:44 +02:00
  • 79ef41e4ee Add four AI-lab crawlers named by Dutch news publishers Lars Anderson 2026-08-27 22:38:09 +02:00
  • 7cd4c92343
         Merge pull request #272 from peppe1337/de-access-index main v1.51 Glyn Normington 2026-08-26 23:51:59 +01:00
  • 5e73986001 Apply maintainer's suggestion: trim entry to avoid stale figures Ideenschmiede CEO 2026-08-26 16:02:12 +00:00
  • 3b67c9c54e Add KI-Zugangsindex to Related resources Ideenschmiede CEO 2026-08-25 12:15:02 +00:00
  • 0037a9e2c5 Update from knownagents.com Known Agents 2026-08-25 00:43:16 +00:00
  • 5692a202f0 Merge pull request #271 from lyrenth/correct-aiwebindex-metadata ai.robots.txt 2026-08-24 14:38:04 +00:00
  • d8e26238a7
         Merge pull request #271 from lyrenth/correct-aiwebindex-metadata Glyn Normington 2026-08-24 15:37:53 +01:00
  • 763f194129 Correct AIWebIndex metadata Lyrenth 2026-08-24 18:27:59 +04:00
  • 738c80df21 Merge pull request #269 from henriquejsza/add/reflectionbot ai.robots.txt 2026-08-22 09:33:25 +00:00
  • 0a871b9e62
         Merge pull request #269 from henriquejsza/add/reflectionbot Glyn Normington 2026-08-22 10:33:13 +01:00
  • cfd62dbc3b
         Add Reflectionbot to robots.json henriquejsza 2026-08-22 03:58:31 -03:00
  • 2d1d8972c9
         Merge pull request #267 from jamiusaliu/test/slcc-real-world-false-positives Glyn Normington 2026-08-20 16:05:40 +01:00
  • 66a8329a61 docs: trim the FAQ entry per review Saliu Jamiu Olamilekan 2026-08-18 21:13:10 -07:00
  • bcc4e408e2 test: pin the SLCC1/SLCC2 false positive from #208 Saliu Jamiu Olamilekan 2026-08-18 20:28:19 -07:00
  • 6a20f8dbc4 Update from knownagents.com Known Agents 2026-08-18 00:41:50 +00:00
  • ff88a7fcb6 Merge pull request #266 from MLuc24/add-exasearchbot ai.robots.txt 2026-08-17 16:39:01 +00:00
  • 8e366f564b
         Merge pull request #266 from MLuc24/add-exasearchbot Glyn Normington 2026-08-17 17:38:50 +01:00
  • 024c9c2d6b Add ExaSearchBot to robots.json MLuc24 2026-08-17 10:30:41 +07:00
  • c7c888c60e Merge pull request #263 from acook/patch-1 v1.50 ai.robots.txt 2026-08-17 02:39:02 +00:00
  • a2858c0255
         Merge pull request #263 from acook/patch-1 Glyn Normington 2026-08-17 03:38:51 +01:00
  • dd30b585f4
         Merge pull request #262 from farrelldan/add-bot-ledger-related Glyn Normington 2026-08-16 04:02:44 +01:00
  • a5d17db8b8
         add Lightpanda user agent Anthony M. Cook 2026-08-07 11:22:16 -05:00
  • abe839e4e0 docs(README): add Bot Ledger to Related resources Dan 2026-08-07 08:12:59 -06:00
  • 2f5d7ccf39
         Merge pull request #260 from Ilyan321/fix/issue-257-regex-anchoring Glyn Normington 2026-08-05 18:18:57 +01:00
  • 4c9331ebbb Merge branch 'main' of https://github.com/ai-robots-txt/ai.robots.txt into fix/issue-257-regex-anchoring Ilyan321 2026-08-05 21:56:22 +05:00
  • 6f3054bcf0 Merge pull request #261 from fork-graveyard/main ai.robots.txt 2026-08-05 16:07:54 +00:00
  • dbd9aa78d4
         Merge pull request #261 from fork-graveyard/main Glyn Normington 2026-08-05 17:07:44 +01:00
  • 1177eba00a fix lighttpd config for 1.x girst 2026-08-05 14:43:39 +02:00
  • 5658a6d726 test(code): expand false positive integration test coverage Ilyan321 2026-08-05 17:06:50 +05:00
  • 51a49494aa fix(code): use word boundaries \b in list_to_pcre to prevent false positive matches like NotCursor Ilyan321 2026-08-05 17:02:56 +05:00
  • 26510d4cd5 Merge branch 'main' of https://github.com/ai-robots-txt/ai.robots.txt into fix/issue-257-regex-anchoring Ilyan321 2026-08-05 16:58:22 +05:00
  • 7b1026e69c test(code): add integration test verifying regex does not match non-AI user agents Ilyan321 2026-08-05 16:50:35 +05:00
  • 5994ca0b33 Merge pull request #259 from fork-graveyard/main ai.robots.txt 2026-08-05 11:33:22 +00:00
  • d93db7b81d
         Merge pull request #259 from fork-graveyard/main Glyn Normington 2026-08-05 12:33:05 +01:00
  • b06780542b fix(code): unanchor PCRE patterns to allow matching real user-agent headers (#257) Ilyan321 2026-08-04 23:14:02 +05:00
  • 0beb4a2666 update test files to match new output girst 2026-08-04 19:25:12 +02:00
  • 8ab5f96c66
         robots.py: list_to_prcre ]) -> ] guest20 2026-08-04 14:58:04 +02:00
  • d98c220be8
         robots.py: list_to_pcre join a list of patterns isntead guest20 2026-08-04 14:56:33 +02:00
  • eed7c67963
         robots.py: list_to_prcre - joining strings joins charaacters guest20 2026-08-04 14:48:57 +02:00
  • a94b2c93e5 further minify regexps girst 2026-08-04 14:44:41 +02:00
  • 9d3a286796
         robots.py: list_to_pcre don't overwrite patterns guest20 2026-08-04 14:46:01 +02:00
  • eaf22ce5bb
         robots.py: list_to_pcre gets support for sends_full_name guest20 2026-08-04 14:36:25 +02:00
  • 3bd200ba32 make nginx matches case-sensitive girst 2026-08-04 14:14:14 +02:00
  • c4b366740e do not needlessly escape hypens girst 2026-08-04 14:13:52 +02:00
  • 27389c0b3d
         Merge pull request #256 from guest20/patch-1 Glyn Normington 2026-08-04 12:05:31 +01:00
  • 940bf7f7e4
         Merge pull request #255 from INXPRNCD/data/fill-unclear-6-entries Glyn Normington 2026-08-04 11:59:32 +01:00
  • 1e031f41d8
         Merge pull request #254 from INXPRNCD/fix/ci-commit-message-injection Glyn Normington 2026-08-04 11:57:47 +01:00
  • 265ea03066
         robots.py: duplciate parser = argparse.ArgumentParser(...) guest20 2026-08-04 10:24:30 +02:00
  • 2eb9315743 Fill "Unclear at this time." from primary operator docs for 6 entries Özden und Julia 2026-08-04 09:11:42 +02:00
  • 4021a10239 Pass commit messages to bash via env, not template interpolation Özden und Julia 2026-08-04 09:09:10 +02:00
  • 24c010e668 Merge pull request #251 from glyn/delete-IbouBot v1.49 ai.robots.txt 2026-07-31 15:01:58 +00:00
  • 1be8fa6220
         Merge pull request #251 from glyn/delete-IbouBot Glyn Normington 2026-07-31 16:01:41 +01:00
  • 69a87566de Delete IbouBot Glyn Normington 2026-07-31 13:04:31 +01:00
  • 042d00a5ab
         Merge pull request #249 from ai-robots-txt/test-ci Glyn Normington 2026-07-31 03:35:15 +01:00
  • c96a9916cc Empty commit to test CI test-ci Glyn Normington 2026-07-31 03:34:01 +01:00
  • 80c19fcad5 Revert "Fill operator and respect from primary docs for 6 entries" Glyn Normington 2026-07-31 03:30:04 +01:00
  • bbe0579a23
         Merge pull request #248 from INXPRNCD/fill-unclear-operator-and-respect-from-primary-docs Glyn Normington 2026-07-31 03:20:58 +01:00
  • 2155906baa Update from knownagents.com Known Agents 2026-07-31 01:57:31 +00:00
  • e62a95bb8a Fill operator and respect from primary docs for 6 entries Erdinc Özden 2026-07-30 22:05:49 +02:00
  • 3db35be22d
         Merge pull request #241 from iamdhrv/fix/exact-spider-user-agent v1.48 Glyn Normington 2026-07-30 12:15:55 +01:00
  • 49669178ef fix: configure exact user-agent matching Dhruv Maniya 2026-07-29 21:47:42 +05:30
  • e88a2ab426 feat: add nbot AI crawler to blocklist Phạm Tiến Phát 2026-07-22 19:29:10 +07:00
  • af31418609 Anchor the generic Spider user agent Dhruv Maniya 2026-07-15 13:04:57 +05:30
  • a0fed45f0e Update from knownagents.com Known Agents 2026-07-10 02:03:20 +00:00
  • 2a83168bbb Merge pull request #240 from bobmatyas/add/tabstack ai.robots.txt 2026-07-09 15:01:12 +00:00
  • 51f7fd5e3a
         Merge pull request #240 from bobmatyas/add/tabstack Glyn Normington 2026-07-09 16:00:59 +01:00
  • a71b27a232 Update from knownagents.com Known Agents 2026-07-09 02:04:28 +00:00
  • c793fa0a8c Add: Mozilla-Tabstack. Closes #226 bob 2026-07-08 21:16:52 -04:00
  • f030cbe956
         Merge pull request #238 from ghking/rename-dark-visitors-to-known-agents v1.47 Glyn Normington 2026-07-09 01:41:52 +01:00
  • 12a15a3a28
         Update code/robots.py Glyn Normington 2026-07-09 01:40:00 +01:00
  • 5065ca6055
         Update .github/workflows/ai_robots_update.yml Glyn Normington 2026-07-09 01:39:50 +01:00
  • 0b021b9e40
         Update code/robots.py Glyn Normington 2026-07-09 01:39:39 +01:00
  • dad7db5168 Fixed broken URLs Gavin King 2026-07-08 20:38:48 -04:00
  • e1bf273b16 Update robots.py Gavin King 2026-07-08 20:26:48 -04:00
  • a35e6f9a09 Update ai_robots_update.yml Gavin King 2026-07-08 20:23:53 -04:00
  • c43c5afd07 More name changes Gavin King 2026-07-08 15:43:07 -04:00
  • 0c5de5b90b Updating the name Gavin King 2026-07-08 15:30:25 -04:00
  • a394130efd Update from Dark Visitors dark-visitors 2026-06-26 02:32:55 +00:00
  • ee4ae068c2
         Merge pull request #236 from guest20/patch-2 Glyn Normington 2026-06-23 11:15:59 +01:00
  • 82df5f7659
         Apply suggestions from code review Glyn Normington 2026-06-23 11:13:17 +01:00
  • ac207f9947
         tests.py: ) guest20 2026-06-23 04:17:14 +02:00
  • ffad9293c4
         tests.py: consolidate guest20 2026-06-23 04:13:17 +02:00
  • 80357baf9c
         robots.py: pass everything to consolodate guest20 2026-06-21 23:51:32 +02:00
  • 7dbe93295e
         robots.py: pull consolodate out guest20 2026-06-21 23:47:11 +02:00
  • 3f965b292d
         robots.py: keep new, non-default values guest20 2026-06-20 18:29:47 +02:00
  • 5cb3ec765f Update from Dark Visitors dark-visitors 2026-06-17 11:21:12 +00:00
  • 25415e342c
         robots.py: div class=description guest20 2026-06-17 13:20:29 +02:00
  • 118bb772c0 Update from Dark Visitors dark-visitors 2026-06-17 11:01:38 +00:00
  • 8dd8e6c4e4
         robots.py: possibly ruin the whole purpose of consolodate guest20 2026-06-17 13:00:15 +02:00
  • 78fd7b49d8
         \robots.py guest20 2026-06-17 12:23:10 +02:00
  • 101d7ac0ad
         robots.py: punked by auto-) again guest20 2026-06-17 12:10:05 +02:00
  • f2147fd81c
         robots.py: fix darkvistors link guest20 2026-06-17 12:06:50 +02:00
  • bd3ee4702c
         robots.py: agents/agents -> agents guest20 2026-06-17 11:45:55 +02:00
  • 3091ad0a23 Update from Dark Visitors dark-visitors 2026-06-16 02:52:42 +00:00
  • f420408eee Update from Dark Visitors dark-visitors 2026-06-04 02:48:49 +00:00