remote/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2026-10-26 10:17:01 +01:00
Code Issues Projects Releases Packages Wiki Activity

Merge 4b1b128249 into 1c4694a3ac

Browse source
This commit is contained in:
jamiezseo 2026-10-24 00:26:41 +07:00 • committed by GitHub
commit ff1c94c914
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 2 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -62,6 +62,8 @@ file on-the-fly.
- [AI Discovery Radar](https://github.com/flober81/ai-discovery-radar): monthly measurement of how many websites actually publish the files that tell AI systems what they may read or use (`robots.txt`, `llms.txt`, `ai.txt`, `tdmrep.json` and similar), and whether those files can be fetched at all. Open data, CC BY 4.0.
- [State of AI Crawler Access](https://queryrev.com/research/ai-crawler-access): open dataset measuring how the top 9,973 Tranco sites treat 11 AI crawlers in robots.txt and whether they publish a well-formed llms.txt, with per-domain results published for each run. Every figure is reported against three denominators (all sites, sites with a readable robots.txt, sites with AI-specific rules). Open data, CC BY 4.0.
## Contributing
Please note that AI-generated contributions are not permitted.