ai.robots.txt

github/ai.robots.txt

Fork 0

mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00

Commit graph

8e6bf36738

Merge 7d6827bc72 into aa8519ec10 fiskhandlarn 2025-12-21 12:46:59 +00:00
aa8519ec10 Update from Dark Visitors main ai.robots.txt 2025-12-21 01:07:06 +00:00
83485effdb Update from Dark Visitors dark-visitors 2025-12-20 00:58:49 +00:00
8b8bf9da5d Update from Dark Visitors ai.robots.txt 2025-12-06 00:58:16 +00:00
f1c752ef12 Update from Dark Visitors dark-visitors 2025-12-05 01:00:44 +00:00
51afa7113a

Update ai_robots_update.yml with rebase command to fix scheduled run Adam Newbold 2025-12-03 20:15:10 -05:00
7598d77e4a Update from Dark Visitors ai.robots.txt 2025-12-04 01:10:02 +00:00
45b071b29f Update from Dark Visitors dark-visitors 2025-12-04 01:00:27 +00:00
f61b3496f7 Update from Dark Visitors dark-visitors 2025-12-03 01:00:32 +00:00
8363d4fdd4 Update from Dark Visitors dark-visitors 2025-12-02 01:25:24 +00:00
2ccd443581

Update ai_robots_update.yml with workflow_dispatch Adam Newbold 2025-12-01 20:24:41 -05:00
6d75f3c1c9

Update robots.py to address error on line 57 Adam Newbold 2025-12-01 20:18:29 -05:00
56010ef913

Merge pull request #205 from fiskhandlarn/fix/editorconfig Glyn Normington 2025-11-29 10:02:08 +00:00
3fadc88a23 Merge pull request #206 from newbold/main ai.robots.txt 2025-11-29 10:00:52 +00:00
47c077a8ef

Merge pull request #206 from newbold/main Glyn Normington 2025-11-29 10:00:42 +00:00
f5d7ccb243

Fixed invalid JSON Adam Newbold 2025-11-28 14:10:38 -05:00
30d719a09a

Adding LAIONDownloader Adam Newbold 2025-11-28 14:08:18 -05:00
05bbdebeaa feat: disallow final newline for files generated by python fiskhandlarn 2025-11-28 10:47:22 +01:00
c6ce9329a1 fix: ensure whitespace as defined in .editorconfig fiskhandlarn 2025-11-28 10:46:22 +01:00
7d6827bc72 fix: generate requirements.txt for this repo only fiskhandlarn 2025-11-28 10:30:17 +01:00
10d5ae2870 Merge pull request #200 from glyn/deepseek ai.robots.txt 2025-11-27 14:00:50 +00:00
4467002298

Merge pull request #200 from glyn/deepseek Glyn Normington 2025-11-27 14:00:40 +00:00
4a159a818f

Merge pull request #199 from glyn/editorconfig Glyn Normington 2025-11-27 13:54:03 +00:00
3d6b33a71a

Merge pull request #202 from glyn/formatting Glyn Normington 2025-11-27 12:42:31 +00:00
c26c8c0911 Tidy README Glyn Normington 2025-11-27 12:41:37 +00:00
91959fe791

Merge pull request #201 from Anshita-18H/add-requirements-file Glyn Normington 2025-11-27 12:38:28 +00:00
b75163e796

Ensure Python3 is used Glyn Normington 2025-11-27 12:38:12 +00:00
f46754d280

Order deps Glyn Normington 2025-11-27 12:37:35 +00:00
c3f2fe758e

Whitespace Glyn Normington 2025-11-27 12:37:21 +00:00
7521c3af50

Fix link Glyn Normington 2025-11-27 12:37:07 +00:00
4302fd1aca Improve contributing section and fix formatting in README Anshita-18H 2025-11-27 17:56:47 +05:30
9ca5033927 Deduplicate requirements.txt and add installation instructions to README Anshita-18H 2025-11-27 17:44:47 +05:30
9cc3dbc05f Add requirements.txt with project dependencies Anshita-18H 2025-11-27 17:06:59 +05:30
c322d7852d

Merge pull request #130 from maxheadroom/main Glyn Normington 2025-11-26 12:08:11 +00:00
35588b2ddb

Typo. Glyn Normington 2025-11-26 12:07:39 +00:00
ab24e41106 Clarify that DeepSeekBot does not respect robots.txt Glyn Normington 2025-11-26 12:02:42 +00:00
b681c0c0d8 Standardise editor options Glyn Normington 2025-11-26 11:54:10 +00:00
4e7e28335f

Merge pull request #195 from fiskhandlarn/patch-2 Glyn Normington 2025-11-26 11:50:15 +00:00
a6cf6b204b test: update test nginx conf fiskhandlarn 2025-11-25 16:47:04 +01:00
2679fcad34 feat: update nginx generator fiskhandlarn 2025-11-25 16:46:31 +01:00
ef8eda4fe6 chore: normalize quote style fiskhandlarn 2025-11-25 16:39:35 +01:00
a29102f0fc

Merge pull request #196 from glyn/releasing v1.44 Adam Newbold 2025-11-25 10:34:25 -05:00
0b3266b35f feat: allow robots access to /robots.txt in nginx fiskhandlarn 2025-11-20 11:50:37 +01:00
6225e3e98e Document how to ship a new release Glyn Normington 2025-11-23 04:01:43 +00:00
be4d74412c Merge pull request #194 from glyn/193-kendra ai.robots.txt 2025-11-21 16:32:11 +00:00
729be4693a

Merge pull request #194 from glyn/193-kendra Cory Dransfeldt 2025-11-21 08:32:01 -08:00
efb4d260da delete extraneous hyphen Glyn Normington 2025-11-20 10:23:09 +00:00
e2726ac160 Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo v1.43 ai.robots.txt 2025-11-14 21:01:25 +00:00
663d030f96

Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo Glyn Normington 2025-11-14 21:01:13 +00:00
28b45ea08d

chore: adds NotebookLM and KlaviyoAIBot agents Cory Dransfeldt 2025-11-14 11:03:03 -08:00
443dd27527 Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots ai.robots.txt 2025-11-05 23:50:59 +00:00
60b6a0829d

Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots Glyn Normington 2025-11-05 23:50:48 +00:00
00bf2b0e13

chore(robots.json): add AmazonBuyForMe and atlassian-bot Cory Dransfeldt 2025-11-05 14:02:37 -08:00
e87eb706e3 Merge pull request #188 from ai-robots-txt/cdransf/buddybot ai.robots.txt 2025-11-04 05:16:28 +00:00
3d41350256

Merge pull request #188 from ai-robots-txt/cdransf/buddybot Glyn Normington 2025-11-04 05:16:15 +00:00
808451055c

chore: adds BuddyBot Cory Dransfeldt 2025-11-03 13:39:07 -08:00
5cad0ee389 Merge pull request #187 from ai-robots-txt/cdransf/add-Linguee-Bot ai.robots.txt 2025-10-24 18:05:01 +00:00
d55c9980cd

Merge pull request #187 from ai-robots-txt/cdransf/add-Linguee-Bot Glyn Normington 2025-10-24 19:04:42 +01:00
192b0a2eef

chore: adds Linguee Bot Cory Dransfeldt 2025-10-24 10:35:04 -07:00
97e19445ce Update from Dark Visitors dark-visitors 2025-10-24 00:53:30 +00:00
0bc2361be8 Merge pull request #186 from ai-robots-txt/third-party ai.robots.txt 2025-10-21 15:23:46 +00:00
511d8c955d

Merge pull request #186 from ai-robots-txt/third-party Cory Dransfeldt 2025-10-21 08:23:28 -07:00
b89a9eae6a Use third party in text, rather than first third-party Glyn Normington 2025-10-21 01:11:32 +01:00
646ab08e15 Merge pull request #185 from ai-robots-txt/cdransf/add-IbouBot ai.robots.txt 2025-10-21 00:09:24 +00:00
5c3da1c1af

Merge pull request #185 from ai-robots-txt/cdransf/add-IbouBot Glyn Normington 2025-10-21 01:09:06 +01:00
d22b8dfd7b

Description is third party, not first Glyn Normington 2025-10-21 01:08:42 +01:00
19c1d346c3

chore: adds IbouBot Cory Dransfeldt 2025-10-20 09:28:31 -07:00
2fa0e9119c Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare v1.42 ai.robots.txt 2025-10-20 14:34:43 +00:00
0874a92503

Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare Glyn Normington 2025-10-20 15:34:30 +01:00
28d2d09633

chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot cdransf/bot-additions-cloudflare Cory Dransfeldt 2025-10-19 14:26:05 -07:00
25ba28c219

chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot cdransf/bot-additions Cory Dransfeldt 2025-10-19 14:26:05 -07:00
260f5029fe Update from Dark Visitors ai.robots.txt 2025-10-16 00:56:33 +00:00
91bf905fa9 Update from Dark Visitors dark-visitors 2025-10-15 00:56:30 +00:00
56d03d46fb Update from Dark Visitors v1.41 dark-visitors 2025-09-26 00:54:30 +00:00
38d60b928c Merge pull request #179 from ai-robots-txt/deepseekbot ai.robots.txt 2025-09-25 10:17:38 +00:00
e2266bbc1d

Merge pull request #179 from ai-robots-txt/deepseekbot Glyn Normington 2025-09-25 11:17:23 +01:00
bf347bdf91

chore(robots.json): add DeepSeekBot Cory Dransfeldt 2025-09-24 09:42:01 -07:00
c6e7d69dd5

Update README.md nisbet-hubbard 2025-09-14 19:41:36 +08:00
9e6a479bd3

Update README.md nisbet-hubbard 2025-09-14 19:41:36 +08:00
8906f6b447 Update from Dark Visitors ai.robots.txt 2025-09-12 00:53:06 +00:00
2fd93029ca Update from Dark Visitors dark-visitors 2025-09-11 00:55:06 +00:00
b6338ddc73 Merge pull request #176 from emersion/TerraCotta ai.robots.txt 2025-09-10 14:11:16 +00:00
7ffbf33baf

Merge pull request #176 from emersion/TerraCotta Cory Dransfeldt 2025-09-10 07:11:04 -07:00
50870ba911 Add TerraCotta Simon Ser 2025-09-10 12:04:08 +02:00
dd391bf960 Update from Dark Visitors dark-visitors 2025-09-10 00:53:56 +00:00
229d1b4dbc Merge pull request #174 from karolyi/master ai.robots.txt 2025-09-09 02:45:32 +00:00
4d506ca322

Merge pull request #174 from karolyi/master Glyn Normington 2025-09-09 03:45:23 +01:00
ec508ab434

Update Brightbot operator and details; add meta-webindexer entry László Károlyi 2025-09-08 21:42:06 +02:00
d1e0a9a757

Add meta-webindexer bot and update Brightbot operator info László Károlyi 2025-09-07 12:12:35 +02:00
0ed29412c9 Update from Dark Visitors v1.40 dark-visitors 2025-08-28 00:55:47 +00:00
2ad1c3e831 Merge pull request #168 from ai-robots-txt/google-firebase-shapbot 1.40 ai.robots.txt 2025-08-27 10:13:17 +00:00
1677278c5a

Merge pull request #168 from ai-robots-txt/google-firebase-shapbot Glyn Normington 2025-08-27 11:13:04 +01:00
1a8edfa84a

chore: add Google-Firebase and ShapBot Cory Dransfeldt 2025-08-26 19:39:42 -07:00
cf073d49f2 Update from Dark Visitors dark-visitors 2025-08-15 01:01:42 +00:00
6d3f3e1712 Merge pull request #167 from ai-robots-txt/OpenAI-bot v1.39 ai.robots.txt 2025-08-14 01:45:05 +00:00
784b8440a5

Merge pull request #167 from ai-robots-txt/OpenAI-bot Glyn Normington 2025-08-14 02:44:53 +01:00
0e687a5b58

chore: adds OpenAI but user agent Cory Dransfeldt 2025-08-13 09:01:55 -07:00
ff9fc26404 Update from Dark Visitors ai.robots.txt 2025-08-01 01:12:53 +00:00
146a229662 Update from Dark Visitors dark-visitors 2025-07-31 01:05:21 +00:00
64f9d6ce9c Update from Dark Visitors ai.robots.txt 2025-07-30 01:05:21 +00:00