github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity
567 commits 6 branches 46 tags 797 KiB
Commit graph

567 commits

This branch
This branch
All branches
Author SHA1 Message Date
Glyn Normington
4e7e28335f
 		Merge pull request #195 from fiskhandlarn/patch-2 
feat: allow robots access to `/robots.txt` in nginx
 2025-11-26 11:50:15 +00:00
fiskhandlarn
a6cf6b204b test: update test nginx conf 2025-11-25 16:47:04 +01:00
fiskhandlarn
2679fcad34 feat: update nginx generator 2025-11-25 16:46:31 +01:00
fiskhandlarn
ef8eda4fe6 chore: normalize quote style 2025-11-25 16:39:35 +01:00
Adam Newbold
a29102f0fc
 		Merge pull request #196 from glyn/releasing v1.44 
Document how to ship a new release
 2025-11-25 10:34:25 -05:00
fiskhandlarn
0b3266b35f feat: allow robots access to /robots.txt in nginx 2025-11-25 16:22:14 +01:00
Glyn Normington
6225e3e98e Document how to ship a new release 2025-11-23 04:01:43 +00:00
ai.robots.txt
be4d74412c Merge pull request #194 from glyn/193-kendra 
delete extraneous hyphen
 2025-11-21 16:32:11 +00:00
Cory Dransfeldt
729be4693a
 		Merge pull request #194 from glyn/193-kendra 
delete extraneous hyphen
 2025-11-21 08:32:01 -08:00
Glyn Normington
efb4d260da delete extraneous hyphen 
Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/193
 2025-11-20 10:23:48 +00:00
ai.robots.txt
e2726ac160 Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo v1.43 
chore: adds NotebookLM and KlaviyoAIBot agents
 2025-11-14 21:01:25 +00:00
Glyn Normington
663d030f96
 		Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo 
chore: adds NotebookLM and KlaviyoAIBot agents
 2025-11-14 21:01:13 +00:00
Cory Dransfeldt
28b45ea08d
 		chore: adds NotebookLM and KlaviyoAIBot agents 2025-11-14 11:03:03 -08:00
ai.robots.txt
443dd27527 Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots 
chore(robots.json): add AmazonBuyForMe and atlassian-bot
 2025-11-05 23:50:59 +00:00
Glyn Normington
60b6a0829d
 		Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots 
chore(robots.json): add AmazonBuyForMe and atlassian-bot
 2025-11-05 23:50:48 +00:00
Cory Dransfeldt
00bf2b0e13
 		chore(robots.json): add AmazonBuyForMe and atlassian-bot 2025-11-05 14:02:37 -08:00
ai.robots.txt
e87eb706e3 Merge pull request #188 from ai-robots-txt/cdransf/buddybot 
chore: adds BuddyBot
 2025-11-04 05:16:28 +00:00
Glyn Normington
3d41350256
 		Merge pull request #188 from ai-robots-txt/cdransf/buddybot 
chore: adds BuddyBot
 2025-11-04 05:16:15 +00:00
Cory Dransfeldt
808451055c
 		chore: adds BuddyBot 2025-11-03 13:39:07 -08:00
ai.robots.txt
5cad0ee389 Merge pull request #187 from ai-robots-txt/cdransf/add-Linguee-Bot 
chore: adds Linguee Bot
 2025-10-24 18:05:01 +00:00
Glyn Normington
d55c9980cd
 		Merge pull request #187 from ai-robots-txt/cdransf/add-Linguee-Bot 
chore: adds Linguee Bot
 2025-10-24 19:04:42 +01:00
Cory Dransfeldt
192b0a2eef
 		chore: adds Linguee Bot 2025-10-24 10:35:04 -07:00
dark-visitors
97e19445ce Update from Dark Visitors 2025-10-24 00:53:30 +00:00
ai.robots.txt
0bc2361be8 Merge pull request #186 from ai-robots-txt/third-party 
Use third party in text, rather than first
 2025-10-21 15:23:46 +00:00
Cory Dransfeldt
511d8c955d
 		Merge pull request #186 from ai-robots-txt/third-party 
Use third party in text, rather than first
 2025-10-21 08:23:28 -07:00
Glyn Normington
b89a9eae6a Use third party in text, rather than first 2025-10-21 01:11:32 +01:00
ai.robots.txt
646ab08e15 Merge pull request #185 from ai-robots-txt/cdransf/add-IbouBot 
chore: adds IbouBot
 2025-10-21 00:09:24 +00:00
Glyn Normington
5c3da1c1af
 		Merge pull request #185 from ai-robots-txt/cdransf/add-IbouBot 
chore: adds IbouBot
 2025-10-21 01:09:06 +01:00
Glyn Normington
d22b8dfd7b
 		Description is third party, not first 2025-10-21 01:08:42 +01:00
Cory Dransfeldt
19c1d346c3
 		chore: adds IbouBot 2025-10-20 09:28:31 -07:00
ai.robots.txt
2fa0e9119c Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare v1.42 
chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot
 2025-10-20 14:34:43 +00:00
Glyn Normington
0874a92503
 		Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare 
chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot
 2025-10-20 15:34:30 +01:00
Cory Dransfeldt
28d2d09633
 		chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot 2025-10-19 15:57:28 -07:00
ai.robots.txt
260f5029fe Update from Dark Visitors 2025-10-16 00:56:33 +00:00
dark-visitors
91bf905fa9 Update from Dark Visitors 2025-10-15 00:56:30 +00:00
dark-visitors
56d03d46fb Update from Dark Visitors v1.41 2025-09-26 00:54:30 +00:00
ai.robots.txt
38d60b928c Merge pull request #179 from ai-robots-txt/deepseekbot 
chore(robots.json): add DeepSeekBot
 2025-09-25 10:17:38 +00:00
Glyn Normington
e2266bbc1d
 		Merge pull request #179 from ai-robots-txt/deepseekbot 
chore(robots.json): add DeepSeekBot
 2025-09-25 11:17:23 +01:00
Cory Dransfeldt
bf347bdf91
 		chore(robots.json): add DeepSeekBot 2025-09-24 09:42:01 -07:00
nisbet-hubbard
c6e7d69dd5
 		Update README.md 2025-09-24 09:39:21 -07:00
ai.robots.txt
8906f6b447 Update from Dark Visitors 2025-09-12 00:53:06 +00:00
dark-visitors
2fd93029ca Update from Dark Visitors 2025-09-11 00:55:06 +00:00
ai.robots.txt
b6338ddc73 Merge pull request #176 from emersion/TerraCotta 
Add TerraCotta
 2025-09-10 14:11:16 +00:00
Cory Dransfeldt
7ffbf33baf
 		Merge pull request #176 from emersion/TerraCotta 
Add TerraCotta
 2025-09-10 07:11:04 -07:00
Simon Ser
50870ba911 Add TerraCotta 
Example server log:

    X.X.X.X - - [10/Sep/2025:00:06:17 +0000] "GET /archives/dri-devel/2023-November/430875.html HTTP/1.1" 200 7388 "-" "TerraCotta https://github.com/CeramicTeam/CeramicTerracotta"
 2025-09-10 12:04:08 +02:00
dark-visitors
dd391bf960 Update from Dark Visitors 2025-09-10 00:53:56 +00:00
ai.robots.txt
229d1b4dbc Merge pull request #174 from karolyi/master 
Update Brightbot operator and details; add meta-webindexer entry
 2025-09-09 02:45:32 +00:00
Glyn Normington
4d506ca322
 		Merge pull request #174 from karolyi/master 
Update Brightbot operator and details; add meta-webindexer entry
 2025-09-09 03:45:23 +01:00
László Károlyi
ec508ab434
 		Update Brightbot operator and details; add meta-webindexer entry 
- Update Brightbot operator to https://brightdata.com/brightbot.
- Change Brightbot frequency to "At least one per minute."
- Expand Brightbot description with disguise tactics link.
- Add new entry for meta-webindexer under Meta operator.
- Set meta-webindexer respect to "Unclear at this time."
- Define meta-webindexer function as "AI Assistants."
- Set meta-webindexer frequency to "Unhinged, more than 1 per second."
- Include meta-webindexer description on improving Meta AI search.
 2025-09-08 21:42:06 +02:00
dark-visitors
0ed29412c9 Update from Dark Visitors v1.40 2025-08-28 00:55:47 +00:00