ai.robots.txt
260f5029fe
Update from Dark Visitors
2025-10-16 00:56:33 +00:00
ai.robots.txt
38d60b928c
Merge pull request #179 from ai-robots-txt/deepseekbot
...
chore(robots.json): add DeepSeekBot
2025-09-25 10:17:38 +00:00
ai.robots.txt
8906f6b447
Update from Dark Visitors
2025-09-12 00:53:06 +00:00
ai.robots.txt
b6338ddc73
Merge pull request #176 from emersion/TerraCotta
...
Add TerraCotta
2025-09-10 14:11:16 +00:00
ai.robots.txt
229d1b4dbc
Merge pull request #174 from karolyi/master
...
Update Brightbot operator and details; add meta-webindexer entry
2025-09-09 02:45:32 +00:00
ai.robots.txt
2ad1c3e831
Merge pull request #168 from ai-robots-txt/google-firebase-shapbot
...
chore: add Google-Firebase and ShapBot
2025-08-27 10:13:17 +00:00
ai.robots.txt
6d3f3e1712
Merge pull request #167 from ai-robots-txt/OpenAI-bot
...
chore: adds OpenAI but user agent
2025-08-14 01:45:05 +00:00
ai.robots.txt
ff9fc26404
Update from Dark Visitors
2025-08-01 01:12:53 +00:00
ai.robots.txt
64f9d6ce9c
Update from Dark Visitors
2025-07-30 01:05:21 +00:00
ai.robots.txt
9565c11d4c
Merge pull request #166 from ai-robots-txt/summaly-bot
...
fix: remove summaly
2025-07-28 19:19:15 +00:00
ai.robots.txt
27420d6fed
Merge pull request #164 from GlitzSmarter/main
...
Adding YaK
2025-07-28 18:29:57 +00:00
ai.robots.txt
47769c8429
Update from Dark Visitors
2025-07-17 01:04:20 +00:00
ai.robots.txt
567e94a6cf
Merge pull request #160 from sdavids/users/sdavids/feat/htaccess
...
Simplify htaccess rewrite rule
2025-07-08 16:00:16 +00:00
ai.robots.txt
63f1f56307
Merge pull request #158 from ai-robots-txt/thinkbot-summaly
...
chore(robots.json): add SummalyBot + Thinkbot
2025-07-07 16:55:31 +00:00
ai.robots.txt
20a242d390
Merge pull request #156 from nisbet-hubbard/patch-9
...
Update funtion and description for SemrushBot
2025-07-02 15:58:19 +00:00
ai.robots.txt
2bfcee794d
Update from Dark Visitors
2025-06-29 01:07:47 +00:00
ai.robots.txt
5326c202b5
Merge pull request #154 from paulrudy/main
...
re-add facebookexternalhit
2025-06-16 15:12:42 +00:00
ai.robots.txt
eb05f2f527
Merge pull request #153 from sergiospagnuolo/Poseidon
...
Update robots.json with new crawler
2025-06-14 14:04:03 +00:00
ai.robots.txt
e53d81c66d
Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot
...
chore(robots.json): adds MyCentralAIScraperBot
2025-06-13 09:28:41 +00:00
ai.robots.txt
842e2256e8
Merge pull request #150 from ai-robots-txt/semrush-bots
...
chore(robots.json): adds additional SemrushBot user agents
2025-06-12 07:12:00 +00:00
ai.robots.txt
3759a6bf14
chore(robots.json): adds EchoboxBot ( #148 )
2025-06-09 15:44:36 +00:00
ai.robots.txt
ac7ed17e71
Merge pull request #145 from ai-robots-txt/aws-bedrockbot
...
chore(robots.json): adds bedrockbot
2025-06-05 16:51:17 +00:00
ai.robots.txt
75ea75a95b
Merge pull request #143 from ai-robots-txt/panscient
...
chore(robots.json): adds Panscient
2025-06-04 18:04:06 +00:00
ai.robots.txt
9c28c63a0c
Merge pull request #142 from ai-robots-txt/quillbot
...
chore(robots.json): adds Quillbot
2025-06-04 17:54:57 +00:00
ai.robots.txt
3efabc603d
Merge pull request #141 from Ivan-Chupin/patch-1
...
Add SBIntuitionsBot
2025-06-03 23:28:48 +00:00
ai.robots.txt
080946c360
Merge pull request #140 from ai-robots-txt/yandex-bots
...
chore(robots.json): adds YandexAdditional crawlers
2025-06-03 19:51:25 +00:00
ai.robots.txt
d239e7e5ad
Merge pull request #139 from ai-robots-txt/workflow-fix
...
chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard
2025-06-03 01:52:35 +00:00
ai.robots.txt
093ab81d78
Update from Dark Visitors
2025-05-23 00:58:57 +00:00
ai.robots.txt
fedb658cc0
Merge pull request #133 from ai-robots-txt/wpbot
...
chore(robots.json): adds wpbot
2025-05-21 21:06:05 +00:00
ai.robots.txt
7c5389f4a0
Merge pull request #98 from kylebuckingham/main
...
Updating Claude Bots
2025-05-21 19:00:23 +00:00
ai.robots.txt
1c2acd75b7
Merge pull request #126 from ai-robots-txt/mistral-bot
...
chore(robots.json): adds MistralAI-User/1.0 crawler
2025-05-21 15:27:26 +00:00
ai.robots.txt
dd1ed174b7
Merge pull request #129 from ai-robots-txt/google-cloudvertexbot
...
chore(robots.json): adds Google-CloudVertexBot
2025-05-16 11:35:15 +00:00
ai.robots.txt
1c470babbe
Merge pull request #123 from joehoyle/patch-1
...
Fix JSON syntax error
2025-05-15 16:12:30 +00:00
ai.robots.txt
c6c7f1748f
Update from Dark Visitors
2025-04-26 00:55:12 +00:00
ai.robots.txt
4654e14e9c
Merge pull request #112 from maiavixen/main
...
Fixed meta-external* being titlecase, and removed period for consistency
2025-04-24 07:00:34 +00:00
ai.robots.txt
bbec639c14
Merge pull request #109 from dennislee1/patch-1
...
AI bots to consider adding
2025-04-22 14:50:26 +00:00
ai.robots.txt
e0cdb278fb
Update from Dark Visitors
2025-04-16 00:57:11 +00:00
ai.robots.txt
4a764bba18
Merge pull request #102 from ai-robots-txt/imgproxy-bot
...
chore(robots.json): adds imgproxy crawler
2025-04-10 19:22:34 +00:00
Frederic Barthelemy
5f5a89c38c
Fix html-mangled hyphen in Perplexity-Users
...
Fixes : #99
2025-04-04 17:34:14 -07:00
ai.robots.txt
5b8650b99b
Update from Dark Visitors
2025-03-29 00:54:10 +00:00
Cory Dransfeldt
ec18af7624
Revert "Merge pull request #91 from deyigifts/perplexity-user"
...
This reverts commit 68d1d93714
2025-03-27 12:51:22 -07:00
ai.robots.txt
68d1d93714
Merge pull request #91 from deyigifts/perplexity-user
...
Update perplexity bots
2025-03-27 19:29:30 +00:00
Dennis Camera
0bd3fa63b8
table-of-bot-metrics.md: Escape robot names for Markdown table
...
Some characters which could occur in a crawler's name have a special meaning in
Markdown. They are escaped to prevent them from having unintended side effects.
The escaping is only applied to the first (Name) column of the table. The rest
of the columns is expected to already be Markdown encoded in robots.json.
2025-02-18 12:53:27 +01:00
ai.robots.txt
693289bb29
chore: add Brightbot 1.0
2025-02-16 21:37:52 +00:00
ai.robots.txt
89d4c6e5ca
Merge pull request #73 from nisbet-hubbard/patch-8
...
Actually block Semrush’s AI tools
2025-02-01 10:51:01 +00:00
ai.robots.txt
6c552a3daa
Merge pull request #71 from jsheard/patch-1
...
Add Crawlspace
2025-01-20 17:45:42 +00:00
ai.robots.txt
ec454b71d3
Merge pull request #67 from Nightfirecat/semrushbot
...
Block SemrushBot
2025-01-06 20:51:56 +00:00
ai.robots.txt
ca8620e28b
Merge pull request #63 from glyn/push-paths
...
Convert robots.json more frequently
2025-01-05 05:05:20 +00:00
dark-visitors
609ddca392
Updated from new robots.json
2024-11-24 00:57:06 +00:00
dark-visitors
58985737e7
Updated from new robots.json
2024-11-19 16:46:21 +00:00
