remote/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2026-05-29 21:26:54 +02:00
Code Issues Projects Releases Packages Wiki Activity
630 commits 6 branches 48 tags 900 KiB
Commit graph

630 commits

This branch
This branch
All branches
Author SHA1 Message Date
dark-visitors
ee3a96ee5c Update from Dark Visitors 2026-05-22 02:33:30 +00:00
Glyn Normington
1fbf7a06ac
 		Merge pull request #231 from cfcryptotrader-rgb/cfcryptotrader-sync-generated-robots v1.46 
Keep generated robot files synced after Dark Visitors update
 2026-05-13 07:44:48 +01:00
cfcryptotrader-rgb
8d5a081a16 Keep generated robot files in dark visitors update 2026-05-13 00:44:06 -03:00
dark-visitors
aea7db9e34 Update from Dark Visitors 2026-05-13 02:24:20 +00:00
Glyn Normington
767fcc6f15
 		Merge pull request #229 from axeleroy/main 
Add missing bot categories
 2026-05-12 04:33:19 +01:00
Axel Leroy
6adc1f7b53
 		Add missing bot categories 2026-05-11 15:42:56 +02:00
dark-visitors
851ce068cf Update from Dark Visitors 2026-05-03 02:01:44 +00:00
ai.robots.txt
8b0bc322f4 Merge pull request #227 from gsauthof/naget-news 
Add NagetBot and newsai user-agents
 2026-05-02 15:05:09 +00:00
Glyn Normington
7244a7e870
 		Merge pull request #227 from gsauthof/naget-news 
Add NagetBot and newsai user-agents
 2026-05-02 16:04:59 +01:00
Georg Sauthoff
a9f686064f Add NagetBot and newsai user-agents 
User-agent header values seen in the wild over the last days:

    Mozilla/5.0 (compatible; NagetBot/1.0; +https://naget.ai/bot)
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 newsai/1.0 Safari/537.36
 2026-05-02 13:04:51 +02:00
ai.robots.txt
198653b59a Update from Dark Visitors 2026-03-27 01:26:26 +00:00
dark-visitors
ad47cafc97 Update from Dark Visitors 2026-03-26 01:26:23 +00:00
dark-visitors
2d2ec4ce8d Update from Dark Visitors 2026-03-25 01:20:42 +00:00
ai.robots.txt
86d582b11c Update from Dark Visitors 2026-03-07 01:12:48 +00:00
dark-visitors
3de8865f00 Update from Dark Visitors 2026-03-06 01:21:01 +00:00
Glyn Normington
243ec6b67d
 		Merge pull request #216 from flymarq/lighttpd v1.45 
Lighttpd
 2026-02-17 17:28:34 +00:00
Flynn Marquardt
793b0454f2 Use adverb. 2026-02-15 11:20:34 +01:00
Flynn Marquardt
790bff1817 Add reference file for lighttpd test. 2026-02-15 11:18:35 +01:00
Flynn Marquardt
742f9b9b5e Add lighttpd test. 2026-02-15 11:17:54 +01:00
ai.robots.txt
48bfefbc5f Update from Dark Visitors 2026-02-14 01:16:06 +00:00
Flynn Marquardt
5f0dd97ccb Add lighttpd sample file. 2026-02-13 19:23:19 +01:00
Flynn Marquardt
55d404c115 Fix typos. 2026-02-13 19:21:31 +01:00
Flynn Marquardt
deae8eed2d Update with lighttpd instructions. 2026-02-13 19:20:12 +01:00
Flynn Marquardt
70e83e6056 Add generation and output of lighttpd configuration fragment. 2026-02-13 19:18:46 +01:00
dark-visitors
a762015e58 Update from Dark Visitors 2026-02-13 01:23:17 +00:00
ai.robots.txt
aa8519ec10 Update from Dark Visitors 2025-12-21 01:07:06 +00:00
dark-visitors
83485effdb Update from Dark Visitors 2025-12-20 00:58:49 +00:00
ai.robots.txt
8b8bf9da5d Update from Dark Visitors 2025-12-06 00:58:16 +00:00
dark-visitors
f1c752ef12 Update from Dark Visitors 2025-12-05 01:00:44 +00:00
Adam Newbold
51afa7113a
 		Update ai_robots_update.yml with rebase command to fix scheduled run 2025-12-03 20:15:10 -05:00
ai.robots.txt
7598d77e4a Update from Dark Visitors 2025-12-04 01:10:02 +00:00
dark-visitors
45b071b29f Update from Dark Visitors 2025-12-04 01:00:27 +00:00
dark-visitors
f61b3496f7 Update from Dark Visitors 2025-12-03 01:00:32 +00:00
dark-visitors
8363d4fdd4 Update from Dark Visitors 2025-12-02 01:25:24 +00:00
Adam Newbold
2ccd443581
 		Update ai_robots_update.yml with workflow_dispatch 
Adding workflow_dispatch to enable manual triggers of this schedule job (for testing)
 2025-12-01 20:24:41 -05:00
Adam Newbold
6d75f3c1c9
 		Update robots.py to address error on line 57 
Attempting to work around an error that prevents parsing the Dark Visitors site
 2025-12-01 20:18:29 -05:00
Glyn Normington
56010ef913
 		Merge pull request #205 from fiskhandlarn/fix/editorconfig 
Fix/editorconfig
 2025-11-29 10:02:08 +00:00
ai.robots.txt
3fadc88a23 Merge pull request #206 from newbold/main 
Adding LAIONDownloader
 2025-11-29 10:00:52 +00:00
Glyn Normington
47c077a8ef
 		Merge pull request #206 from newbold/main 
Adding LAIONDownloader
 2025-11-29 10:00:42 +00:00
Adam Newbold
f5d7ccb243
 		Fixed invalid JSON 2025-11-28 14:10:38 -05:00
Adam Newbold
30d719a09a
 		Adding LAIONDownloader 2025-11-28 14:08:18 -05:00
fiskhandlarn
05bbdebeaa feat: disallow final newline for files generated by python 
if any of these files have ending newlines the tests will fail
 2025-11-28 10:47:22 +01:00
fiskhandlarn
c6ce9329a1 fix: ensure whitespace as defined in .editorconfig 2025-11-28 10:46:22 +01:00
ai.robots.txt
10d5ae2870 Merge pull request #200 from glyn/deepseek 
Clarify that DeepSeekBot does not respect robots.txt
 2025-11-27 14:00:50 +00:00
Glyn Normington
4467002298
 		Merge pull request #200 from glyn/deepseek 
Clarify that DeepSeekBot does not respect robots.txt
 2025-11-27 14:00:40 +00:00
Glyn Normington
4a159a818f
 		Merge pull request #199 from glyn/editorconfig 
Standardise editor options
 2025-11-27 13:54:03 +00:00
Glyn Normington
3d6b33a71a
 		Merge pull request #202 from glyn/formatting 
Tidy README
 2025-11-27 12:42:31 +00:00
Glyn Normington
c26c8c0911 Tidy README 2025-11-27 12:41:37 +00:00
Glyn Normington
91959fe791
 		Merge pull request #201 from Anshita-18H/add-requirements-file 
Add requirements.txt with project dependencies
 2025-11-27 12:38:28 +00:00
Glyn Normington
b75163e796
 		Ensure Python3 is used 2025-11-27 12:38:12 +00:00