github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity

Add meta-webindexer bot and update Brightbot operator info

Browse source 
- Added 'meta-webindexer' to HAProxy, Nginx, robots.txt blocklists
- Updated Brightbot operator to brightdata.com in robots.json and metrics
- Added Brightbot frequency and disguise tactics documentation link
- Added meta-webindexer entry in robots.json with Meta's official description
- Added meta-webindexer row in table-of-bot-metrics.md with details
This commit is contained in:
László Károlyi 2025-09-07 12:12:35 +02:00
commit d1e0a9a757
No known key found for this signature in database
GPG key ID: 8EAA28E6FCF2F46C
5 changed files with 18 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

1
robots.txt
View file

@ -55,6 +55,7 @@ User-agent: meta-externalagent
User-agent: Meta-ExternalAgent
User-agent: meta-externalfetcher
User-agent: Meta-ExternalFetcher
User-agent: meta-webindexer
User-agent: MistralAI-User
User-agent: MistralAI-User/1.0
User-agent: MyCentralAIScraperBot