László Károlyi
d1e0a9a757
Add meta-webindexer bot and update Brightbot operator info
...
- Added 'meta-webindexer' to HAProxy, Nginx, robots.txt blocklists
- Updated Brightbot operator to brightdata.com in robots.json and metrics
- Added Brightbot frequency and disguise tactics documentation link
- Added meta-webindexer entry in robots.json with Meta's official description
- Added meta-webindexer row in table-of-bot-metrics.md with details
2025-09-07 12:12:35 +02:00
ai.robots.txt
2ad1c3e831
Merge pull request #168 from ai-robots-txt/google-firebase-shapbot
...
chore: add Google-Firebase and ShapBot
2025-08-27 10:13:17 +00:00
ai.robots.txt
6d3f3e1712
Merge pull request #167 from ai-robots-txt/OpenAI-bot
...
chore: adds OpenAI but user agent
2025-08-14 01:45:05 +00:00
ai.robots.txt
ff9fc26404
Update from Dark Visitors
2025-08-01 01:12:53 +00:00
ai.robots.txt
64f9d6ce9c
Update from Dark Visitors
2025-07-30 01:05:21 +00:00
ai.robots.txt
9565c11d4c
Merge pull request #166 from ai-robots-txt/summaly-bot
...
fix: remove summaly
2025-07-28 19:19:15 +00:00
ai.robots.txt
27420d6fed
Merge pull request #164 from GlitzSmarter/main
...
Adding YaK
2025-07-28 18:29:57 +00:00
ai.robots.txt
47769c8429
Update from Dark Visitors
2025-07-17 01:04:20 +00:00
ai.robots.txt
567e94a6cf
Merge pull request #160 from sdavids/users/sdavids/feat/htaccess
...
Simplify htaccess rewrite rule
2025-07-08 16:00:16 +00:00
ai.robots.txt
63f1f56307
Merge pull request #158 from ai-robots-txt/thinkbot-summaly
...
chore(robots.json): add SummalyBot + Thinkbot
2025-07-07 16:55:31 +00:00
ai.robots.txt
20a242d390
Merge pull request #156 from nisbet-hubbard/patch-9
...
Update funtion and description for SemrushBot
2025-07-02 15:58:19 +00:00
ai.robots.txt
2bfcee794d
Update from Dark Visitors
2025-06-29 01:07:47 +00:00
ai.robots.txt
5326c202b5
Merge pull request #154 from paulrudy/main
...
re-add facebookexternalhit
2025-06-16 15:12:42 +00:00
ai.robots.txt
eb05f2f527
Merge pull request #153 from sergiospagnuolo/Poseidon
...
Update robots.json with new crawler
2025-06-14 14:04:03 +00:00
ai.robots.txt
e53d81c66d
Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot
...
chore(robots.json): adds MyCentralAIScraperBot
2025-06-13 09:28:41 +00:00
ai.robots.txt
842e2256e8
Merge pull request #150 from ai-robots-txt/semrush-bots
...
chore(robots.json): adds additional SemrushBot user agents
2025-06-12 07:12:00 +00:00
ai.robots.txt
3759a6bf14
chore(robots.json): adds EchoboxBot ( #148 )
2025-06-09 15:44:36 +00:00
ai.robots.txt
ac7ed17e71
Merge pull request #145 from ai-robots-txt/aws-bedrockbot
...
chore(robots.json): adds bedrockbot
2025-06-05 16:51:17 +00:00
ai.robots.txt
75ea75a95b
Merge pull request #143 from ai-robots-txt/panscient
...
chore(robots.json): adds Panscient
2025-06-04 18:04:06 +00:00
ai.robots.txt
9c28c63a0c
Merge pull request #142 from ai-robots-txt/quillbot
...
chore(robots.json): adds Quillbot
2025-06-04 17:54:57 +00:00
ai.robots.txt
3efabc603d
Merge pull request #141 from Ivan-Chupin/patch-1
...
Add SBIntuitionsBot
2025-06-03 23:28:48 +00:00
ai.robots.txt
080946c360
Merge pull request #140 from ai-robots-txt/yandex-bots
...
chore(robots.json): adds YandexAdditional crawlers
2025-06-03 19:51:25 +00:00
ai.robots.txt
d239e7e5ad
Merge pull request #139 from ai-robots-txt/workflow-fix
...
chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard
2025-06-03 01:52:35 +00:00
ai.robots.txt
093ab81d78
Update from Dark Visitors
2025-05-23 00:58:57 +00:00
ai.robots.txt
fedb658cc0
Merge pull request #133 from ai-robots-txt/wpbot
...
chore(robots.json): adds wpbot
2025-05-21 21:06:05 +00:00
ai.robots.txt
7c5389f4a0
Merge pull request #98 from kylebuckingham/main
...
Updating Claude Bots
2025-05-21 19:00:23 +00:00
ai.robots.txt
1c2acd75b7
Merge pull request #126 from ai-robots-txt/mistral-bot
...
chore(robots.json): adds MistralAI-User/1.0 crawler
2025-05-21 15:27:26 +00:00
ai.robots.txt
dd1ed174b7
Merge pull request #129 from ai-robots-txt/google-cloudvertexbot
...
chore(robots.json): adds Google-CloudVertexBot
2025-05-16 11:35:15 +00:00
ai.robots.txt
1c470babbe
Merge pull request #123 from joehoyle/patch-1
...
Fix JSON syntax error
2025-05-15 16:12:30 +00:00
Rik Wijnen
50e739dd73
HAProxy converter added.
2025-04-28 08:51:02 +02:00
