ai.robots.txt
|
646ab08e15
|
Merge pull request #185 from ai-robots-txt/cdransf/add-IbouBot
chore: adds IbouBot
|
2025-10-21 00:09:24 +00:00 |
|
ai.robots.txt
|
2fa0e9119c
|
Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare
chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot
|
2025-10-20 14:34:43 +00:00 |
|
ai.robots.txt
|
260f5029fe
|
Update from Dark Visitors
|
2025-10-16 00:56:33 +00:00 |
|
ai.robots.txt
|
38d60b928c
|
Merge pull request #179 from ai-robots-txt/deepseekbot
chore(robots.json): add DeepSeekBot
|
2025-09-25 10:17:38 +00:00 |
|
ai.robots.txt
|
8906f6b447
|
Update from Dark Visitors
|
2025-09-12 00:53:06 +00:00 |
|
ai.robots.txt
|
b6338ddc73
|
Merge pull request #176 from emersion/TerraCotta
Add TerraCotta
|
2025-09-10 14:11:16 +00:00 |
|
ai.robots.txt
|
229d1b4dbc
|
Merge pull request #174 from karolyi/master
Update Brightbot operator and details; add meta-webindexer entry
|
2025-09-09 02:45:32 +00:00 |
|
ai.robots.txt
|
2ad1c3e831
|
Merge pull request #168 from ai-robots-txt/google-firebase-shapbot
chore: add Google-Firebase and ShapBot
|
2025-08-27 10:13:17 +00:00 |
|
ai.robots.txt
|
6d3f3e1712
|
Merge pull request #167 from ai-robots-txt/OpenAI-bot
chore: adds OpenAI but user agent
|
2025-08-14 01:45:05 +00:00 |
|
ai.robots.txt
|
ff9fc26404
|
Update from Dark Visitors
|
2025-08-01 01:12:53 +00:00 |
|
ai.robots.txt
|
64f9d6ce9c
|
Update from Dark Visitors
|
2025-07-30 01:05:21 +00:00 |
|
ai.robots.txt
|
9565c11d4c
|
Merge pull request #166 from ai-robots-txt/summaly-bot
fix: remove summaly
|
2025-07-28 19:19:15 +00:00 |
|
ai.robots.txt
|
27420d6fed
|
Merge pull request #164 from GlitzSmarter/main
Adding YaK
|
2025-07-28 18:29:57 +00:00 |
|
ai.robots.txt
|
47769c8429
|
Update from Dark Visitors
|
2025-07-17 01:04:20 +00:00 |
|
ai.robots.txt
|
567e94a6cf
|
Merge pull request #160 from sdavids/users/sdavids/feat/htaccess
Simplify htaccess rewrite rule
|
2025-07-08 16:00:16 +00:00 |
|
ai.robots.txt
|
63f1f56307
|
Merge pull request #158 from ai-robots-txt/thinkbot-summaly
chore(robots.json): add SummalyBot + Thinkbot
|
2025-07-07 16:55:31 +00:00 |
|
ai.robots.txt
|
20a242d390
|
Merge pull request #156 from nisbet-hubbard/patch-9
Update funtion and description for SemrushBot
|
2025-07-02 15:58:19 +00:00 |
|
ai.robots.txt
|
2bfcee794d
|
Update from Dark Visitors
|
2025-06-29 01:07:47 +00:00 |
|
ai.robots.txt
|
5326c202b5
|
Merge pull request #154 from paulrudy/main
re-add facebookexternalhit
|
2025-06-16 15:12:42 +00:00 |
|
ai.robots.txt
|
eb05f2f527
|
Merge pull request #153 from sergiospagnuolo/Poseidon
Update robots.json with new crawler
|
2025-06-14 14:04:03 +00:00 |
|
ai.robots.txt
|
e53d81c66d
|
Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot
chore(robots.json): adds MyCentralAIScraperBot
|
2025-06-13 09:28:41 +00:00 |
|
ai.robots.txt
|
842e2256e8
|
Merge pull request #150 from ai-robots-txt/semrush-bots
chore(robots.json): adds additional SemrushBot user agents
|
2025-06-12 07:12:00 +00:00 |
|
ai.robots.txt
|
3759a6bf14
|
chore(robots.json): adds EchoboxBot (#148)
|
2025-06-09 15:44:36 +00:00 |
|
ai.robots.txt
|
ac7ed17e71
|
Merge pull request #145 from ai-robots-txt/aws-bedrockbot
chore(robots.json): adds bedrockbot
|
2025-06-05 16:51:17 +00:00 |
|
ai.robots.txt
|
75ea75a95b
|
Merge pull request #143 from ai-robots-txt/panscient
chore(robots.json): adds Panscient
|
2025-06-04 18:04:06 +00:00 |
|
ai.robots.txt
|
9c28c63a0c
|
Merge pull request #142 from ai-robots-txt/quillbot
chore(robots.json): adds Quillbot
|
2025-06-04 17:54:57 +00:00 |
|
ai.robots.txt
|
3efabc603d
|
Merge pull request #141 from Ivan-Chupin/patch-1
Add SBIntuitionsBot
|
2025-06-03 23:28:48 +00:00 |
|
ai.robots.txt
|
080946c360
|
Merge pull request #140 from ai-robots-txt/yandex-bots
chore(robots.json): adds YandexAdditional crawlers
|
2025-06-03 19:51:25 +00:00 |
|
ai.robots.txt
|
d239e7e5ad
|
Merge pull request #139 from ai-robots-txt/workflow-fix
chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard
|
2025-06-03 01:52:35 +00:00 |
|
ai.robots.txt
|
093ab81d78
|
Update from Dark Visitors
|
2025-05-23 00:58:57 +00:00 |
|
ai.robots.txt
|
fedb658cc0
|
Merge pull request #133 from ai-robots-txt/wpbot
chore(robots.json): adds wpbot
|
2025-05-21 21:06:05 +00:00 |
|
Cory Dransfeldt
|
af597586b6
|
Merge pull request #98 from kylebuckingham/main
Updating Claude Bots
|
2025-05-21 12:00:11 -07:00 |
|
ai.robots.txt
|
1c2acd75b7
|
Merge pull request #126 from ai-robots-txt/mistral-bot
chore(robots.json): adds MistralAI-User/1.0 crawler
|
2025-05-21 15:27:26 +00:00 |
|
ai.robots.txt
|
dd1ed174b7
|
Merge pull request #129 from ai-robots-txt/google-cloudvertexbot
chore(robots.json): adds Google-CloudVertexBot
|
2025-05-16 11:35:15 +00:00 |
|
ai.robots.txt
|
1c470babbe
|
Merge pull request #123 from joehoyle/patch-1
Fix JSON syntax error
|
2025-05-15 16:12:30 +00:00 |
|
ai.robots.txt
|
c6c7f1748f
|
Update from Dark Visitors
|
2025-04-26 00:55:12 +00:00 |
|
ai.robots.txt
|
4654e14e9c
|
Merge pull request #112 from maiavixen/main
Fixed meta-external* being titlecase, and removed period for consistency
|
2025-04-24 07:00:34 +00:00 |
|
ai.robots.txt
|
bbec639c14
|
Merge pull request #109 from dennislee1/patch-1
AI bots to consider adding
|
2025-04-22 14:50:26 +00:00 |
|
ai.robots.txt
|
e0cdb278fb
|
Update from Dark Visitors
|
2025-04-16 00:57:11 +00:00 |
|
ai.robots.txt
|
4a764bba18
|
Merge pull request #102 from ai-robots-txt/imgproxy-bot
chore(robots.json): adds imgproxy crawler
|
2025-04-10 19:22:34 +00:00 |
|
Frederic Barthelemy
|
5f5a89c38c
|
Fix html-mangled hyphen in Perplexity-Users
Fixes: #99
|
2025-04-04 17:34:14 -07:00 |
|
Kyle Buckingham
|
8dc36aa2e2
|
Update robots.txt
|
2025-04-01 15:23:28 -07:00 |
|
ai.robots.txt
|
5b8650b99b
|
Update from Dark Visitors
|
2025-03-29 00:54:10 +00:00 |
|
Cory Dransfeldt
|
ec18af7624
|
Revert "Merge pull request #91 from deyigifts/perplexity-user"
This reverts commit 68d1d93714.
|
2025-03-27 12:51:22 -07:00 |
|
ai.robots.txt
|
68d1d93714
|
Merge pull request #91 from deyigifts/perplexity-user
Update perplexity bots
|
2025-03-27 19:29:30 +00:00 |
|
ai.robots.txt
|
693289bb29
|
chore: add Brightbot 1.0
|
2025-02-16 21:37:52 +00:00 |
|
ai.robots.txt
|
89d4c6e5ca
|
Merge pull request #73 from nisbet-hubbard/patch-8
Actually block Semrush’s AI tools
|
2025-02-01 10:51:01 +00:00 |
|
ai.robots.txt
|
6c552a3daa
|
Merge pull request #71 from jsheard/patch-1
Add Crawlspace
|
2025-01-20 17:45:42 +00:00 |
|
ai.robots.txt
|
ec454b71d3
|
Merge pull request #67 from Nightfirecat/semrushbot
Block SemrushBot
|
2025-01-06 20:51:56 +00:00 |
|
ai.robots.txt
|
ca8620e28b
|
Merge pull request #63 from glyn/push-paths
Convert robots.json more frequently
|
2025-01-05 05:05:20 +00:00 |
|