remote/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2026-07-09 08:04:34 +02:00
Code Issues Projects Releases Packages Wiki Activity
635 commits 6 branches 49 tags 1.3 MiB
Commit graph

635 commits

This branch
This branch
All branches
Author SHA1 Message Date
dark-visitors
3091ad0a23 Update from Dark Visitors 2026-06-16 02:52:42 +00:00
dark-visitors
f420408eee Update from Dark Visitors 2026-06-04 02:48:49 +00:00
ai.robots.txt
000552c2df Merge pull request #232 from justinribeiro/main 
add: AgentTimes bot
 2026-06-03 08:56:38 +00:00
Glyn Normington
ef50defc1a
 		Merge pull request #232 from justinribeiro/main 
add: AgentTimes bot
 2026-06-03 09:56:26 +01:00
Justin Ribeiro
7673437733
 		add: AgentTimes bot 2026-06-02 11:26:06 -07:00
dark-visitors
ee3a96ee5c Update from Dark Visitors 2026-05-22 02:33:30 +00:00
Glyn Normington
1fbf7a06ac
 		Merge pull request #231 from cfcryptotrader-rgb/cfcryptotrader-sync-generated-robots v1.46 
Keep generated robot files synced after Dark Visitors update
 2026-05-13 07:44:48 +01:00
cfcryptotrader-rgb
8d5a081a16 Keep generated robot files in dark visitors update 2026-05-13 00:44:06 -03:00
dark-visitors
aea7db9e34 Update from Dark Visitors 2026-05-13 02:24:20 +00:00
Glyn Normington
767fcc6f15
 		Merge pull request #229 from axeleroy/main 
Add missing bot categories
 2026-05-12 04:33:19 +01:00
Axel Leroy
6adc1f7b53
 		Add missing bot categories 2026-05-11 15:42:56 +02:00
dark-visitors
851ce068cf Update from Dark Visitors 2026-05-03 02:01:44 +00:00
ai.robots.txt
8b0bc322f4 Merge pull request #227 from gsauthof/naget-news 
Add NagetBot and newsai user-agents
 2026-05-02 15:05:09 +00:00
Glyn Normington
7244a7e870
 		Merge pull request #227 from gsauthof/naget-news 
Add NagetBot and newsai user-agents
 2026-05-02 16:04:59 +01:00
Georg Sauthoff
a9f686064f Add NagetBot and newsai user-agents 
User-agent header values seen in the wild over the last days:

    Mozilla/5.0 (compatible; NagetBot/1.0; +https://naget.ai/bot)
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 newsai/1.0 Safari/537.36
 2026-05-02 13:04:51 +02:00
ai.robots.txt
198653b59a Update from Dark Visitors 2026-03-27 01:26:26 +00:00
dark-visitors
ad47cafc97 Update from Dark Visitors 2026-03-26 01:26:23 +00:00
dark-visitors
2d2ec4ce8d Update from Dark Visitors 2026-03-25 01:20:42 +00:00
ai.robots.txt
86d582b11c Update from Dark Visitors 2026-03-07 01:12:48 +00:00
dark-visitors
3de8865f00 Update from Dark Visitors 2026-03-06 01:21:01 +00:00
Glyn Normington
243ec6b67d
 		Merge pull request #216 from flymarq/lighttpd v1.45 
Lighttpd
 2026-02-17 17:28:34 +00:00
Flynn Marquardt
793b0454f2 Use adverb. 2026-02-15 11:20:34 +01:00
Flynn Marquardt
790bff1817 Add reference file for lighttpd test. 2026-02-15 11:18:35 +01:00
Flynn Marquardt
742f9b9b5e Add lighttpd test. 2026-02-15 11:17:54 +01:00
ai.robots.txt
48bfefbc5f Update from Dark Visitors 2026-02-14 01:16:06 +00:00
Flynn Marquardt
5f0dd97ccb Add lighttpd sample file. 2026-02-13 19:23:19 +01:00
Flynn Marquardt
55d404c115 Fix typos. 2026-02-13 19:21:31 +01:00
Flynn Marquardt
deae8eed2d Update with lighttpd instructions. 2026-02-13 19:20:12 +01:00
Flynn Marquardt
70e83e6056 Add generation and output of lighttpd configuration fragment. 2026-02-13 19:18:46 +01:00
dark-visitors
a762015e58 Update from Dark Visitors 2026-02-13 01:23:17 +00:00
ai.robots.txt
aa8519ec10 Update from Dark Visitors 2025-12-21 01:07:06 +00:00
dark-visitors
83485effdb Update from Dark Visitors 2025-12-20 00:58:49 +00:00
ai.robots.txt
8b8bf9da5d Update from Dark Visitors 2025-12-06 00:58:16 +00:00
dark-visitors
f1c752ef12 Update from Dark Visitors 2025-12-05 01:00:44 +00:00
Adam Newbold
51afa7113a
 		Update ai_robots_update.yml with rebase command to fix scheduled run 2025-12-03 20:15:10 -05:00
ai.robots.txt
7598d77e4a Update from Dark Visitors 2025-12-04 01:10:02 +00:00
dark-visitors
45b071b29f Update from Dark Visitors 2025-12-04 01:00:27 +00:00
dark-visitors
f61b3496f7 Update from Dark Visitors 2025-12-03 01:00:32 +00:00
dark-visitors
8363d4fdd4 Update from Dark Visitors 2025-12-02 01:25:24 +00:00
Adam Newbold
2ccd443581
 		Update ai_robots_update.yml with workflow_dispatch 
Adding workflow_dispatch to enable manual triggers of this schedule job (for testing)
 2025-12-01 20:24:41 -05:00
Adam Newbold
6d75f3c1c9
 		Update robots.py to address error on line 57 
Attempting to work around an error that prevents parsing the Dark Visitors site
 2025-12-01 20:18:29 -05:00
Glyn Normington
56010ef913
 		Merge pull request #205 from fiskhandlarn/fix/editorconfig 
Fix/editorconfig
 2025-11-29 10:02:08 +00:00
ai.robots.txt
3fadc88a23 Merge pull request #206 from newbold/main 
Adding LAIONDownloader
 2025-11-29 10:00:52 +00:00
Glyn Normington
47c077a8ef
 		Merge pull request #206 from newbold/main 
Adding LAIONDownloader
 2025-11-29 10:00:42 +00:00
Adam Newbold
f5d7ccb243
 		Fixed invalid JSON 2025-11-28 14:10:38 -05:00
Adam Newbold
30d719a09a
 		Adding LAIONDownloader 2025-11-28 14:08:18 -05:00
fiskhandlarn
05bbdebeaa feat: disallow final newline for files generated by python 
if any of these files have ending newlines the tests will fail
 2025-11-28 10:47:22 +01:00
fiskhandlarn
c6ce9329a1 fix: ensure whitespace as defined in .editorconfig 2025-11-28 10:46:22 +01:00
ai.robots.txt
10d5ae2870 Merge pull request #200 from glyn/deepseek 
Clarify that DeepSeekBot does not respect robots.txt
 2025-11-27 14:00:50 +00:00
Glyn Normington
4467002298
 		Merge pull request #200 from glyn/deepseek 
Clarify that DeepSeekBot does not respect robots.txt
 2025-11-27 14:00:40 +00:00