remote/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2026-07-15 19:04:32 +02:00
Code Issues Projects Releases Packages Wiki Activity

Commit graph

  • 51092665b2
         Merge af31418609 into a0fed45f0e Dhruv 2026-07-15 13:06:30 +05:30
  • af31418609 Anchor the generic Spider user agent Dhruv Maniya 2026-07-15 13:04:57 +05:30
  • a0fed45f0e Update from knownagents.com main Known Agents 2026-07-10 02:03:20 +00:00
  • 2a83168bbb Merge pull request #240 from bobmatyas/add/tabstack ai.robots.txt 2026-07-09 15:01:12 +00:00
  • 51f7fd5e3a
         Merge pull request #240 from bobmatyas/add/tabstack Glyn Normington 2026-07-09 16:00:59 +01:00
  • a71b27a232 Update from knownagents.com Known Agents 2026-07-09 02:04:28 +00:00
  • c793fa0a8c Add: Mozilla-Tabstack. Closes #226 bob 2026-07-08 21:16:52 -04:00
  • f030cbe956
         Merge pull request #238 from ghking/rename-dark-visitors-to-known-agents v1.47 Glyn Normington 2026-07-09 01:41:52 +01:00
  • 12a15a3a28
         Update code/robots.py Glyn Normington 2026-07-09 01:40:00 +01:00
  • 5065ca6055
         Update .github/workflows/ai_robots_update.yml Glyn Normington 2026-07-09 01:39:50 +01:00
  • 0b021b9e40
         Update code/robots.py Glyn Normington 2026-07-09 01:39:39 +01:00
  • dad7db5168 Fixed broken URLs Gavin King 2026-07-08 20:38:48 -04:00
  • e1bf273b16 Update robots.py Gavin King 2026-07-08 20:26:48 -04:00
  • a35e6f9a09 Update ai_robots_update.yml Gavin King 2026-07-08 20:23:53 -04:00
  • c43c5afd07 More name changes Gavin King 2026-07-08 15:43:07 -04:00
  • 0c5de5b90b Updating the name Gavin King 2026-07-08 15:30:25 -04:00
  • a394130efd Update from Dark Visitors dark-visitors 2026-06-26 02:32:55 +00:00
  • ee4ae068c2
         Merge pull request #236 from guest20/patch-2 Glyn Normington 2026-06-23 11:15:59 +01:00
  • 82df5f7659
         Apply suggestions from code review Glyn Normington 2026-06-23 11:13:17 +01:00
  • ac207f9947
         tests.py: ) guest20 2026-06-23 04:17:14 +02:00
  • ffad9293c4
         tests.py: consolidate guest20 2026-06-23 04:13:17 +02:00
  • 80357baf9c
         robots.py: pass everything to consolodate guest20 2026-06-21 23:51:32 +02:00
  • 7dbe93295e
         robots.py: pull consolodate out guest20 2026-06-21 23:47:11 +02:00
  • 3f965b292d
         robots.py: keep new, non-default values guest20 2026-06-20 18:29:47 +02:00
  • 5cb3ec765f Update from Dark Visitors dark-visitors 2026-06-17 11:21:12 +00:00
  • 25415e342c
         robots.py: div class=description guest20 2026-06-17 13:20:29 +02:00
  • 118bb772c0 Update from Dark Visitors dark-visitors 2026-06-17 11:01:38 +00:00
  • 8dd8e6c4e4
         robots.py: possibly ruin the whole purpose of consolodate guest20 2026-06-17 13:00:15 +02:00
  • 78fd7b49d8
         \robots.py guest20 2026-06-17 12:23:10 +02:00
  • 101d7ac0ad
         robots.py: punked by auto-) again guest20 2026-06-17 12:10:05 +02:00
  • f2147fd81c
         robots.py: fix darkvistors link guest20 2026-06-17 12:06:50 +02:00
  • bd3ee4702c
         robots.py: agents/agents -> agents guest20 2026-06-17 11:45:55 +02:00
  • 3091ad0a23 Update from Dark Visitors dark-visitors 2026-06-16 02:52:42 +00:00
  • f420408eee Update from Dark Visitors dark-visitors 2026-06-04 02:48:49 +00:00
  • 000552c2df Merge pull request #232 from justinribeiro/main ai.robots.txt 2026-06-03 08:56:38 +00:00
  • ef50defc1a
         Merge pull request #232 from justinribeiro/main Glyn Normington 2026-06-03 09:56:26 +01:00
  • 7673437733
         add: AgentTimes bot Justin Ribeiro 2026-06-02 11:26:06 -07:00
  • ee3a96ee5c Update from Dark Visitors dark-visitors 2026-05-22 02:33:30 +00:00
  • 1fbf7a06ac
         Merge pull request #231 from cfcryptotrader-rgb/cfcryptotrader-sync-generated-robots v1.46 Glyn Normington 2026-05-13 07:44:48 +01:00
  • 8d5a081a16 Keep generated robot files in dark visitors update cfcryptotrader-rgb 2026-05-13 00:44:06 -03:00
  • aea7db9e34 Update from Dark Visitors dark-visitors 2026-05-13 02:24:20 +00:00
  • 767fcc6f15
         Merge pull request #229 from axeleroy/main Glyn Normington 2026-05-12 04:33:19 +01:00
  • 6adc1f7b53
         Add missing bot categories Axel Leroy 2026-05-11 15:31:22 +02:00
  • 851ce068cf Update from Dark Visitors dark-visitors 2026-05-03 02:01:44 +00:00
  • 8b0bc322f4 Merge pull request #227 from gsauthof/naget-news ai.robots.txt 2026-05-02 15:05:09 +00:00
  • 7244a7e870
         Merge pull request #227 from gsauthof/naget-news Glyn Normington 2026-05-02 16:04:59 +01:00
  • a9f686064f Add NagetBot and newsai user-agents Georg Sauthoff 2026-05-02 13:04:51 +02:00
  • 198653b59a Update from Dark Visitors ai.robots.txt 2026-03-27 01:26:26 +00:00
  • ad47cafc97 Update from Dark Visitors dark-visitors 2026-03-26 01:26:23 +00:00
  • 2d2ec4ce8d Update from Dark Visitors dark-visitors 2026-03-25 01:20:42 +00:00
  • dc0e28989b tests: make nginx bypass assertion explicit shuofengzhang 2026-03-21 19:40:35 +00:00
  • 507d16262a tests: make converter-helper coverage independent and CI-executed shuofengzhang 2026-03-18 19:30:27 +00:00
  • d32f959fea test: reduce converter test circularity and overlap shuofengzhang 2026-03-16 19:24:54 +00:00
  • 0c995a065e Merge remote-tracking branch 'origin/main' into add-tests-converters shuofengzhang 2026-03-15 13:40:47 +00:00
  • c223c8bced chore: suppress expected robot-name normalization noise shuofengzhang 2026-03-15 13:34:40 +00:00
  • e7f8b0ca91 ci: run pytest suite for converter tests shuofengzhang 2026-03-09 13:58:08 +08:00
  • ee50da4b4e test: broaden converter and file update coverage shuofengzhang 2026-03-09 08:51:22 +08:00
  • f8bd13d9a3 test: expand converter coverage for output formats shuofengzhang 2026-03-09 05:57:04 +08:00
  • 86d582b11c Update from Dark Visitors ai.robots.txt 2026-03-07 01:12:48 +00:00
  • 3de8865f00 Update from Dark Visitors dark-visitors 2026-03-06 01:21:01 +00:00
  • e73aacfd22 Add tests for converter helpers shuofengzhang 2026-03-03 20:35:19 +08:00
  • 243ec6b67d
         Merge pull request #216 from flymarq/lighttpd v1.45 Glyn Normington 2026-02-17 17:28:34 +00:00
  • 793b0454f2 Use adverb. Flynn Marquardt 2026-02-15 11:20:34 +01:00
  • 790bff1817 Add reference file for lighttpd test. Flynn Marquardt 2026-02-15 11:18:35 +01:00
  • 742f9b9b5e Add lighttpd test. Flynn Marquardt 2026-02-15 11:17:54 +01:00
  • 48bfefbc5f Update from Dark Visitors ai.robots.txt 2026-02-14 01:16:06 +00:00
  • 5f0dd97ccb Add lighttpd sample file. Flynn Marquardt 2026-02-13 19:23:19 +01:00
  • 55d404c115 Fix typos. Flynn Marquardt 2026-02-13 19:21:31 +01:00
  • deae8eed2d Update with lighttpd instructions. Flynn Marquardt 2026-02-13 19:20:12 +01:00
  • 70e83e6056 Add generation and output of lighttpd configuration fragment. Flynn Marquardt 2026-02-13 19:18:46 +01:00
  • a762015e58 Update from Dark Visitors dark-visitors 2026-02-13 01:23:17 +00:00
  • 03c6890c45
         Rename README.mdf to README.md جواد 2026-01-03 20:08:09 +03:00
  • aa8519ec10 Update from Dark Visitors ai.robots.txt 2025-12-21 01:07:06 +00:00
  • 83485effdb Update from Dark Visitors dark-visitors 2025-12-20 00:58:49 +00:00
  • 8b8bf9da5d Update from Dark Visitors ai.robots.txt 2025-12-06 00:58:16 +00:00
  • f1c752ef12 Update from Dark Visitors dark-visitors 2025-12-05 01:00:44 +00:00
  • 51afa7113a
         Update ai_robots_update.yml with rebase command to fix scheduled run Adam Newbold 2025-12-03 20:15:10 -05:00
  • 7598d77e4a Update from Dark Visitors ai.robots.txt 2025-12-04 01:10:02 +00:00
  • 45b071b29f Update from Dark Visitors dark-visitors 2025-12-04 01:00:27 +00:00
  • f61b3496f7 Update from Dark Visitors dark-visitors 2025-12-03 01:00:32 +00:00
  • 8363d4fdd4 Update from Dark Visitors dark-visitors 2025-12-02 01:25:24 +00:00
  • 2ccd443581
         Update ai_robots_update.yml with workflow_dispatch Adam Newbold 2025-12-01 20:24:41 -05:00
  • 6d75f3c1c9
         Update robots.py to address error on line 57 Adam Newbold 2025-12-01 20:18:29 -05:00
  • 56010ef913
         Merge pull request #205 from fiskhandlarn/fix/editorconfig Glyn Normington 2025-11-29 10:02:08 +00:00
  • 3fadc88a23 Merge pull request #206 from newbold/main ai.robots.txt 2025-11-29 10:00:52 +00:00
  • 47c077a8ef
         Merge pull request #206 from newbold/main Glyn Normington 2025-11-29 10:00:42 +00:00
  • f5d7ccb243
         Fixed invalid JSON Adam Newbold 2025-11-28 14:10:38 -05:00
  • 30d719a09a
         Adding LAIONDownloader Adam Newbold 2025-11-28 14:08:18 -05:00
  • 05bbdebeaa feat: disallow final newline for files generated by python fiskhandlarn 2025-11-28 10:47:22 +01:00
  • c6ce9329a1 fix: ensure whitespace as defined in .editorconfig fiskhandlarn 2025-11-28 10:46:22 +01:00
  • 7d6827bc72 fix: generate requirements.txt for this repo only fiskhandlarn 2025-11-28 10:30:17 +01:00
  • 10d5ae2870 Merge pull request #200 from glyn/deepseek ai.robots.txt 2025-11-27 14:00:50 +00:00
  • 4467002298
         Merge pull request #200 from glyn/deepseek Glyn Normington 2025-11-27 14:00:40 +00:00
  • 4a159a818f
         Merge pull request #199 from glyn/editorconfig Glyn Normington 2025-11-27 13:54:03 +00:00
  • 3d6b33a71a
         Merge pull request #202 from glyn/formatting Glyn Normington 2025-11-27 12:42:31 +00:00
  • c26c8c0911 Tidy README Glyn Normington 2025-11-27 12:41:37 +00:00
  • 91959fe791
         Merge pull request #201 from Anshita-18H/add-requirements-file Glyn Normington 2025-11-27 12:38:28 +00:00
  • b75163e796
         Ensure Python3 is used Glyn Normington 2025-11-27 12:38:12 +00:00
  • f46754d280
         Order deps Glyn Normington 2025-11-27 12:37:35 +00:00
  • c3f2fe758e
         Whitespace Glyn Normington 2025-11-27 12:37:21 +00:00