github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity

Commit graph

  • 261a2b83b9 update README to inclide list of ai bots Cloudflare considers verified always-be-testing 2025-02-14 12:26:19 -05:00
  • bebffccc0c Update from Dark Visitors v1.25 dark-visitors 2025-02-02 00:52:50 +00:00
  • 89d4c6e5ca Merge pull request #73 from nisbet-hubbard/patch-8 ai.robots.txt 2025-02-01 10:51:01 +00:00
  • f9e2c5810b
         Merge pull request #73 from nisbet-hubbard/patch-8 Glyn Normington 2025-02-01 10:50:50 +00:00
  • 05b79b8a58
         Update robots.json nisbet-hubbard 2025-01-27 19:41:03 +08:00
  • 9c060dee1c Update from Dark Visitors v1.24 dark-visitors 2025-01-21 00:49:22 +00:00
  • 6c552a3daa Merge pull request #71 from jsheard/patch-1 ai.robots.txt 2025-01-20 17:45:42 +00:00
  • f621fb4852
         Merge pull request #71 from jsheard/patch-1 Glyn Normington 2025-01-20 17:45:29 +00:00
  • 7427d96bac
         Update robots.json Joshua Sheard 2025-01-20 10:59:02 +00:00
  • 81cc81b35e
         Merge pull request #68 from MassiminoilTrace/main Glyn Normington 2025-01-20 07:33:54 +00:00
  • 4f03818280 Removed if condition and added a little comments Massimo Gismondi 2025-01-20 06:51:06 +01:00
  • a9956f7825 Removed additional sections Massimo Gismondi 2025-01-20 06:50:48 +01:00
  • 33c38ee70b
         Update README.md Massimo Gismondi 2025-01-20 06:28:32 +01:00
  • 52241bdca6
         Update README.md Massimo Gismondi 2025-01-20 06:27:56 +01:00
  • 013b7abfa1
         Update README.md Massimo Gismondi 2025-01-20 06:27:02 +01:00
  • 70fd6c0fb1
         Add mention of htaccess in readme Massimo Gismondi 2025-01-20 06:25:07 +01:00
  • 5aa08bc002
         Add Crawlspace Joshua Sheard 2025-01-19 22:03:50 +00:00
  • d65128d10a
         Removed paragraph in favour of future FAQ.md Massimo Gismondi 2025-01-18 12:41:09 +01:00
  • 1cc4b59dfc
         Shortened htaccess instructions Massimo Gismondi 2025-01-18 12:40:03 +01:00
  • 8aee2f24bb
         Fixed space in comment Massimo Gismondi 2025-01-18 12:39:07 +01:00
  • b455af66e7 Adding clarification about performance and code comment Massimo Gismondi 2025-01-17 21:42:08 +01:00
  • 189e75bbfd Adding usage instructions Massimo Gismondi 2025-01-17 21:25:23 +01:00
  • 933aa6159d Implementing htaccess generation Massimo Gismondi 2025-01-07 11:02:29 +01:00
  • b7f908e305
         Merge pull request #66 from fabianegli/patch-1 v1.23 Glyn Normington 2025-01-07 03:54:40 +00:00
  • ec454b71d3 Merge pull request #67 from Nightfirecat/semrushbot ai.robots.txt 2025-01-06 20:51:56 +00:00
  • 565dca3dc0
         Merge pull request #67 from Nightfirecat/semrushbot Cory Dransfeldt 2025-01-06 12:51:43 -08:00
  • 143f8f2285
         Block SemrushBot Jordan Atwood 2025-01-06 12:34:38 -08:00
  • 8e98cc6049
         Merge pull request #61 from glyn/improve-naming Cory Dransfeldt 2025-01-06 08:10:47 -08:00
  • 30ee957011
         bail when NO changes are staged Fabian Egli 2025-01-06 12:05:42 +01:00
  • 83cd546470
         allow Action to succeed even if no changes were made Fabian Egli 2025-01-06 11:39:41 +01:00
  • 0777befd36
         Create .htaccess Erdem 2025-01-06 02:56:07 +03:00
  • 15fc333a70
         Update robots.txt Erdem 2025-01-06 02:38:12 +03:00
  • ca8620e28b Merge pull request #63 from glyn/push-paths v1.22 ai.robots.txt 2025-01-05 05:05:20 +00:00
  • b9df958b39
         Merge pull request #63 from glyn/push-paths Glyn Normington 2025-01-05 05:05:01 +00:00
  • c01a684036 Convert robots.json more frequently Glyn Normington 2025-01-05 05:03:50 +00:00
  • d2be15447c
         Merge pull request #62 from ai-robots-txt/missing-dependency Glyn Normington 2025-01-05 01:46:27 +00:00
  • 9e372d0696 Ensure dependency installed missing-dependency Glyn Normington 2025-01-05 01:45:33 +00:00
  • 996b9c678c Improve job name Glyn Normington 2025-01-04 05:28:41 +00:00
  • e4c12ee2f8 Rename in test code Glyn Normington 2025-01-04 05:03:48 +00:00
  • 3a43714908 Rename Python code Glyn Normington 2025-01-04 04:55:34 +00:00
  • 2036a68c1f Update from Dark Visitors v1.21 dark-visitors 2024-12-04 00:55:50 +00:00
  • 24666e8b15
         Merge pull request #58 from fabianegli/fabianegli-restore-attribution v1.20 Glyn Normington 2024-11-29 09:05:16 +00:00
  • eb8e1a49b5 Revert "specify file encodings in tests" fabianegli 2024-11-29 09:02:47 +01:00
  • b64284d684 restore correct attribution logic to before PR #55 fabianegli 2024-11-26 09:41:46 +01:00
  • bd38c30194 specify file encodings in tests fabianegli 2024-11-26 09:12:11 +01:00
  • aff37825ce
         Create azure-functions-app-python.yml Tyler Hawthorne 2024-11-25 09:14:53 -05:00
  • db9643b29b
         Create python-publish.yml Tyler Hawthorne 2024-11-25 09:14:18 -05:00
  • 609ddca392 Updated from new robots.json dark-visitors 2024-11-24 00:57:06 +00:00
  • 37065f9118 Update from Dark Visitors dark-visitors 2024-11-24 00:57:05 +00:00
  • 58985737e7 Updated from new robots.json dark-visitors 2024-11-19 16:46:21 +00:00
  • 584e66cb99
         Merge pull request #56 from glyn/40-exclude-facebookexternalhit Cory Dransfeldt 2024-11-19 08:46:05 -08:00
  • 80002f5e17 Allow facebookexternalhit Glyn Normington 2024-11-19 03:33:45 +00:00
  • 71db599b41
         Merge pull request #55 from norwd/feature/add-robots.txt-file-to-release Glyn Normington 2024-11-13 01:39:11 +00:00
  • e8f0784a00
         Explicitly use release tag for checkout Y. Meyer-Norwood 2024-11-13 10:26:37 +13:00
  • 94ceb3cffd
         Add authentication for gh command Y. Meyer-Norwood 2024-11-11 13:04:55 +13:00
  • adfd4af872
         Create upload-robots-txt-file-to-release.yml Y. Meyer-Norwood 2024-11-11 12:58:40 +13:00
  • d50615d394 Improve formatting Glyn Normington 2024-11-10 01:06:13 +00:00
  • 2c88909be3 Fix formatting Glyn Normington 2024-11-10 01:02:18 +00:00
  • 6f58ddc623
         Merge pull request #54 from glyn/rationale Glyn Normington 2024-11-10 00:58:29 +00:00
  • 9295b6a963 Clarify our rationale Glyn Normington 2024-11-09 04:45:47 +00:00
  • 9e06cf3bc9 Updated from new robots.json v1.19 dark-visitors 2024-10-29 00:52:12 +00:00
  • bc0a0ad0e9 Update from Dark Visitors dark-visitors 2024-10-29 00:52:12 +00:00
  • fe5f407673 Update from Dark Visitors dark-visitors 2024-10-27 00:54:47 +00:00
  • a66b16827d
         Merge pull request #51 from fabianegli/php-to-python-plus-tests Adam Newbold 2024-10-22 21:32:58 -04:00
  • 3ab22bc498 make conversions and updates separately triggerable fabianegli 2024-10-19 19:56:41 +02:00
  • 6ab8fb2d37 no more failure when run without network fabianegli 2024-10-19 19:11:01 +02:00
  • 7e2b3ab037 rename action fabianegli 2024-10-19 19:09:34 +02:00
  • 0c05461f84 simplify repo and added some tests fabianegli 2024-10-19 13:06:34 +02:00
  • 6bb598820e ignore venv fabianegli 2024-10-18 23:24:13 +02:00
  • d62cab66c5
         Merge pull request #50 from glyn/fix-typo Glyn Normington 2024-10-19 04:43:09 +01:00
  • 6a359e7fd7 Fix typo and trigger rerun of main job ai.robots.txt 2024-10-19 03:43:00 +00:00
  • 38a388097c Fix typo and trigger rerun of main job Glyn Normington 2024-10-19 04:42:27 +01:00
  • 83c8603071
         Merge pull request #49 from glyn/php-diagnostics Glyn Normington 2024-10-19 04:34:53 +01:00
  • a80bd18fb8 Dump out file contents in PHP script ai.robots.txt 2024-10-19 03:34:29 +00:00
  • bdf30be7dc Dump out file contents in PHP script Glyn Normington 2024-10-19 04:33:46 +01:00
  • 4d47b17c45
         Merge pull request #47 from fabianegli/fabianegli-patch-1 Glyn Normington 2024-10-19 02:58:05 +01:00
  • faf81efb12 Daily update from Dark Visitors dark-visitors 2024-10-19 01:17:15 +00:00
  • 25adc6b802
         log git repository status Fabian Egli 2024-10-19 00:28:41 +02:00
  • b584f613cd
         add some signposts to the log Fabian Egli 2024-10-19 00:13:09 +02:00
  • b3068a8d90
         add some signposts Fabian Egli 2024-10-19 00:12:25 +02:00
  • a46d06d436
         log changes made by the action in main.yml Fabian Egli 2024-10-19 00:04:15 +02:00
  • cfaade6e2f
         log the diff in the update action daily_update.yml Fabian Egli 2024-10-19 00:01:15 +02:00
  • 04f630f7f8
         Merge pull request #45 from glyn/faq-update Cory Dransfeldt 2024-10-18 06:35:47 -07:00
  • 898c8ab82d
         Merge pull request #46 from isagalaev/case-insensitive-sorting Glyn Normington 2024-10-18 07:57:56 +01:00
  • 7bb5efd462
         Sort the content case-insensitively before dumping to JSON Ivan Sagalaev 2024-10-17 21:08:43 -04:00
  • e6bb7cae9e Augment the "why" FAQ Glyn Normington 2024-10-17 12:27:05 +01:00
  • b229f5b936 Re-order the FAQ Glyn Normington 2024-10-17 12:25:54 +01:00
  • b1491d2694 Daily update from Dark Visitors dark-visitors 2024-10-09 01:17:37 +00:00
  • 9be286626d Merge pull request #43 from lxjv/main ai.robots.txt 2024-10-08 02:30:17 +00:00
  • 01993b98c3
         Merge pull request #43 from lxjv/main Glyn Normington 2024-10-08 03:30:07 +01:00
  • dc15afe847
         Update robots.json with Claude respect link Laker Turner 2024-10-07 17:38:01 +01:00
  • 6da804e826 chore: add ISSCyberRiskCrawler v1.18 ai.robots.txt 2024-09-30 23:50:18 +00:00
  • 9c2394f23b
         chore: add ISSCyberRiskCrawler Cory Dransfeldt 2024-09-30 16:25:20 -07:00
  • 6d9ce1d62a chore: add sidetrade bot v1.17 ai.robots.txt 2024-09-28 20:58:18 +00:00
  • 6a988be27f
         chore: add sidetrade bot Cory Dransfeldt 2024-09-28 13:58:00 -07:00
  • 632e9d6510 Daily update from Dark Visitors ai.robots.txt 2024-09-28 01:17:19 +00:00
  • 7851cea4fd Daily update from Dark Visitors dark-visitors 2024-09-27 01:18:04 +00:00
  • 75343c790e
         Merge pull request #38 from urvish-p80/main Glyn Normington 2024-09-27 01:26:04 +01:00
  • 44d975c799 Merge pull request #42 from commoncrawl/main ai.robots.txt 2024-09-27 00:21:49 +00:00
  • 2f67e77ddb
         Merge pull request #42 from commoncrawl/main Glyn Normington 2024-09-27 01:21:37 +01:00