ai.robots.txt
|
aa8519ec10
|
Update from Dark Visitors
|
2025-12-21 01:07:06 +00:00 |
|
ai.robots.txt
|
8b8bf9da5d
|
Update from Dark Visitors
|
2025-12-06 00:58:16 +00:00 |
|
ai.robots.txt
|
7598d77e4a
|
Update from Dark Visitors
|
2025-12-04 01:10:02 +00:00 |
|
ai.robots.txt
|
3fadc88a23
|
Merge pull request #206 from newbold/main
Adding LAIONDownloader
|
2025-11-29 10:00:52 +00:00 |
|
ai.robots.txt
|
be4d74412c
|
Merge pull request #194 from glyn/193-kendra
delete extraneous hyphen
|
2025-11-21 16:32:11 +00:00 |
|
ai.robots.txt
|
e2726ac160
|
Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo
chore: adds NotebookLM and KlaviyoAIBot agents
|
2025-11-14 21:01:25 +00:00 |
|
ai.robots.txt
|
443dd27527
|
Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots
chore(robots.json): add AmazonBuyForMe and atlassian-bot
|
2025-11-05 23:50:59 +00:00 |
|
ai.robots.txt
|
e87eb706e3
|
Merge pull request #188 from ai-robots-txt/cdransf/buddybot
chore: adds BuddyBot
|
2025-11-04 05:16:28 +00:00 |
|
ai.robots.txt
|
5cad0ee389
|
Merge pull request #187 from ai-robots-txt/cdransf/add-Linguee-Bot
chore: adds Linguee Bot
|
2025-10-24 18:05:01 +00:00 |
|
ai.robots.txt
|
646ab08e15
|
Merge pull request #185 from ai-robots-txt/cdransf/add-IbouBot
chore: adds IbouBot
|
2025-10-21 00:09:24 +00:00 |
|
ai.robots.txt
|
2fa0e9119c
|
Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare
chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot
|
2025-10-20 14:34:43 +00:00 |
|
ai.robots.txt
|
260f5029fe
|
Update from Dark Visitors
|
2025-10-16 00:56:33 +00:00 |
|
ai.robots.txt
|
38d60b928c
|
Merge pull request #179 from ai-robots-txt/deepseekbot
chore(robots.json): add DeepSeekBot
|
2025-09-25 10:17:38 +00:00 |
|
ai.robots.txt
|
8906f6b447
|
Update from Dark Visitors
|
2025-09-12 00:53:06 +00:00 |
|
ai.robots.txt
|
b6338ddc73
|
Merge pull request #176 from emersion/TerraCotta
Add TerraCotta
|
2025-09-10 14:11:16 +00:00 |
|
ai.robots.txt
|
229d1b4dbc
|
Merge pull request #174 from karolyi/master
Update Brightbot operator and details; add meta-webindexer entry
|
2025-09-09 02:45:32 +00:00 |
|
ai.robots.txt
|
2ad1c3e831
|
Merge pull request #168 from ai-robots-txt/google-firebase-shapbot
chore: add Google-Firebase and ShapBot
|
2025-08-27 10:13:17 +00:00 |
|
ai.robots.txt
|
6d3f3e1712
|
Merge pull request #167 from ai-robots-txt/OpenAI-bot
chore: adds OpenAI but user agent
|
2025-08-14 01:45:05 +00:00 |
|
ai.robots.txt
|
ff9fc26404
|
Update from Dark Visitors
|
2025-08-01 01:12:53 +00:00 |
|
ai.robots.txt
|
64f9d6ce9c
|
Update from Dark Visitors
|
2025-07-30 01:05:21 +00:00 |
|
ai.robots.txt
|
9565c11d4c
|
Merge pull request #166 from ai-robots-txt/summaly-bot
fix: remove summaly
|
2025-07-28 19:19:15 +00:00 |
|
ai.robots.txt
|
27420d6fed
|
Merge pull request #164 from GlitzSmarter/main
Adding YaK
|
2025-07-28 18:29:57 +00:00 |
|
ai.robots.txt
|
47769c8429
|
Update from Dark Visitors
|
2025-07-17 01:04:20 +00:00 |
|
ai.robots.txt
|
567e94a6cf
|
Merge pull request #160 from sdavids/users/sdavids/feat/htaccess
Simplify htaccess rewrite rule
|
2025-07-08 16:00:16 +00:00 |
|
Sebastian Davids
|
c7407e721d
|
Simplify htaccess rewrite rule
https://httpd.apache.org/docs/2.4/rewrite/flags.html#flag_f
fix #159
Signed-off-by: Sebastian Davids <sdavids@gmx.de>
|
2025-07-08 12:33:24 +02:00 |
|
ai.robots.txt
|
63f1f56307
|
Merge pull request #158 from ai-robots-txt/thinkbot-summaly
chore(robots.json): add SummalyBot + Thinkbot
|
2025-07-07 16:55:31 +00:00 |
|
ai.robots.txt
|
20a242d390
|
Merge pull request #156 from nisbet-hubbard/patch-9
Update funtion and description for SemrushBot
|
2025-07-02 15:58:19 +00:00 |
|
ai.robots.txt
|
2bfcee794d
|
Update from Dark Visitors
|
2025-06-29 01:07:47 +00:00 |
|
ai.robots.txt
|
5326c202b5
|
Merge pull request #154 from paulrudy/main
re-add facebookexternalhit
|
2025-06-16 15:12:42 +00:00 |
|
ai.robots.txt
|
eb05f2f527
|
Merge pull request #153 from sergiospagnuolo/Poseidon
Update robots.json with new crawler
|
2025-06-14 14:04:03 +00:00 |
|
ai.robots.txt
|
e53d81c66d
|
Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot
chore(robots.json): adds MyCentralAIScraperBot
|
2025-06-13 09:28:41 +00:00 |
|
ai.robots.txt
|
842e2256e8
|
Merge pull request #150 from ai-robots-txt/semrush-bots
chore(robots.json): adds additional SemrushBot user agents
|
2025-06-12 07:12:00 +00:00 |
|
ai.robots.txt
|
3759a6bf14
|
chore(robots.json): adds EchoboxBot (#148)
|
2025-06-09 15:44:36 +00:00 |
|
ai.robots.txt
|
ac7ed17e71
|
Merge pull request #145 from ai-robots-txt/aws-bedrockbot
chore(robots.json): adds bedrockbot
|
2025-06-05 16:51:17 +00:00 |
|
ai.robots.txt
|
75ea75a95b
|
Merge pull request #143 from ai-robots-txt/panscient
chore(robots.json): adds Panscient
|
2025-06-04 18:04:06 +00:00 |
|
ai.robots.txt
|
9c28c63a0c
|
Merge pull request #142 from ai-robots-txt/quillbot
chore(robots.json): adds Quillbot
|
2025-06-04 17:54:57 +00:00 |
|
ai.robots.txt
|
3efabc603d
|
Merge pull request #141 from Ivan-Chupin/patch-1
Add SBIntuitionsBot
|
2025-06-03 23:28:48 +00:00 |
|
ai.robots.txt
|
080946c360
|
Merge pull request #140 from ai-robots-txt/yandex-bots
chore(robots.json): adds YandexAdditional crawlers
|
2025-06-03 19:51:25 +00:00 |
|
ai.robots.txt
|
d239e7e5ad
|
Merge pull request #139 from ai-robots-txt/workflow-fix
chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard
|
2025-06-03 01:52:35 +00:00 |
|
ai.robots.txt
|
093ab81d78
|
Update from Dark Visitors
|
2025-05-23 00:58:57 +00:00 |
|
ai.robots.txt
|
fedb658cc0
|
Merge pull request #133 from ai-robots-txt/wpbot
chore(robots.json): adds wpbot
|
2025-05-21 21:06:05 +00:00 |
|
ai.robots.txt
|
7c5389f4a0
|
Merge pull request #98 from kylebuckingham/main
Updating Claude Bots
|
2025-05-21 19:00:23 +00:00 |
|
ai.robots.txt
|
1c2acd75b7
|
Merge pull request #126 from ai-robots-txt/mistral-bot
chore(robots.json): adds MistralAI-User/1.0 crawler
|
2025-05-21 15:27:26 +00:00 |
|
ai.robots.txt
|
dd1ed174b7
|
Merge pull request #129 from ai-robots-txt/google-cloudvertexbot
chore(robots.json): adds Google-CloudVertexBot
|
2025-05-16 11:35:15 +00:00 |
|
ai.robots.txt
|
1c470babbe
|
Merge pull request #123 from joehoyle/patch-1
Fix JSON syntax error
|
2025-05-15 16:12:30 +00:00 |
|
ai.robots.txt
|
c6c7f1748f
|
Update from Dark Visitors
|
2025-04-26 00:55:12 +00:00 |
|
ai.robots.txt
|
4654e14e9c
|
Merge pull request #112 from maiavixen/main
Fixed meta-external* being titlecase, and removed period for consistency
|
2025-04-24 07:00:34 +00:00 |
|
ai.robots.txt
|
bbec639c14
|
Merge pull request #109 from dennislee1/patch-1
AI bots to consider adding
|
2025-04-22 14:50:26 +00:00 |
|
ai.robots.txt
|
e0cdb278fb
|
Update from Dark Visitors
|
2025-04-16 00:57:11 +00:00 |
|
ai.robots.txt
|
4a764bba18
|
Merge pull request #102 from ai-robots-txt/imgproxy-bot
chore(robots.json): adds imgproxy crawler
|
2025-04-10 19:22:34 +00:00 |
|