remote/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2026-10-25 17:56:58 +01:00
Code Issues Projects Releases Packages Wiki Activity

Enhance README with file links and notes

Browse source 
Updated the usage section to include links for files and added a note about test files.
This commit is contained in:
Glyn Normington 2026-10-09 12:21:53 +01:00 • committed by GitHub
commit ee24bcfeb7
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 10 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

16
README.md
View file

@ -10,14 +10,16 @@ If you'd like to add an AI-related crawler to the list, please see "Contributing
## Usage
This repository provides the following files:
This repository provides the following files in the top-level directory[^root]:
- `robots.txt`
- `.htaccess`
- `nginx-block-ai-bots.conf`
- `Caddyfile`
- `haproxy-block-ai-bots.txt`
- `lighttpd-block-ai-bots.conf`
- [`robots.txt`](./robots.txt)
- [`.htaccess`](./.htaccess)
- [`nginx-block-ai-bots.conf`](./nginx-block-ai-bots.conf)
- [`Caddyfile`](./Caddyfile)
- [`haproxy-block-ai-bots.txt`](./haproxy-block-ai-bots.txt)
- [`lighttpd-block-ai-bots.conf`](./lighttpd-block-ai-bots.conf)
[^root]: Other files with similar names appear in the directory `code/test_files`. These files are for test purposes only and you must not use them on your own site.
`robots.txt` implements the Robots Exclusion Protocol ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)).