github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity

Add meta-webindexer bot and update Brightbot operator info

Browse source 
- Added 'meta-webindexer' to HAProxy, Nginx, robots.txt blocklists
- Updated Brightbot operator to brightdata.com in robots.json and metrics
- Added Brightbot frequency and disguise tactics documentation link
- Added meta-webindexer entry in robots.json with Meta's official description
- Added meta-webindexer row in table-of-bot-metrics.md with details
This commit is contained in:
László Károlyi 2025-09-07 12:12:35 +02:00
commit d1e0a9a757
No known key found for this signature in database
GPG key ID: 8EAA28E6FCF2F46C
5 changed files with 18 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

3
haproxy-block-ai-bots.txt
View file

@ -55,6 +55,7 @@ meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
meta-webindexer
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
@ -92,4 +93,4 @@ wpbot
YaK
YandexAdditional
YandexAdditionalBot
YouBot
YouBot