From ee24bcfeb7f7d5a715f7232c8376edc73a295b17 Mon Sep 17 00:00:00 2001 From: Glyn Normington Date: Fri, 9 Oct 2026 12:21:53 +0100 Subject: [PATCH] Enhance README with file links and notes Updated the usage section to include links for files and added a note about test files. --- README.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index a8108a9..93843d9 100644 --- a/README.md +++ b/README.md @@ -10,14 +10,16 @@ If you'd like to add an AI-related crawler to the list, please see "Contributing ## Usage -This repository provides the following files: +This repository provides the following files in the top-level directory[^root]: -- `robots.txt` -- `.htaccess` -- `nginx-block-ai-bots.conf` -- `Caddyfile` -- `haproxy-block-ai-bots.txt` -- `lighttpd-block-ai-bots.conf` +- [`robots.txt`](./robots.txt) +- [`.htaccess`](./.htaccess) +- [`nginx-block-ai-bots.conf`](./nginx-block-ai-bots.conf) +- [`Caddyfile`](./Caddyfile) +- [`haproxy-block-ai-bots.txt`](./haproxy-block-ai-bots.txt) +- [`lighttpd-block-ai-bots.conf`](./lighttpd-block-ai-bots.conf) + +[^root]: Other files with similar names appear in the directory `code/test_files`. These files are for test purposes only and you must not use them on your own site. `robots.txt` implements the Robots Exclusion Protocol ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)).