github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #205 from fiskhandlarn/fix/editorconfig

Browse source 
Fix/editorconfig
This commit is contained in:
Glyn Normington 2025-11-29 10:02:08 +00:00 • committed by GitHub
commit 56010ef913
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 8 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

3
.editorconfig
View file

@ -4,3 +4,6 @@ root = true
end_of_line = lf end_of_line = lf
insert_final_newline = true insert_final_newline = true
trim_trailing_whitespace = true trim_trailing_whitespace = true
[{Caddyfile,haproxy-block-ai-bots.txt,nginx-block-ai-bots.conf}]
insert_final_newline = false

2
README.md
View file

@ -103,5 +103,3 @@ But even if you don't use Cloudflare's hard block, their list of [verified bots]
- [Blockin' bots on Netlify](https://www.jeremiak.com/blog/block-bots-netlify-edge-functions/) by Jeremia Kimelman - [Blockin' bots on Netlify](https://www.jeremiak.com/blog/block-bots-netlify-edge-functions/) by Jeremia Kimelman
- [Blocking AI web crawlers](https://underlap.org/blocking-ai-web-crawlers) by Glyn Normington - [Blocking AI web crawlers](https://underlap.org/blocking-ai-web-crawlers) by Glyn Normington
- [Block AI Bots from Crawling Websites Using Robots.txt](https://originality.ai/ai-bot-blocking) by Jonathan Gillham, Originality.AI - [Block AI Bots from Crawling Websites Using Robots.txt](https://originality.ai/ai-bot-blocking) by Jonathan Gillham, Originality.AI

1
docs/traefik-manual-setup.md
View file

@ -31,7 +31,6 @@ networks:
external: true external: true
``` ```
The Traefik HTTP Routers rule explicitly does not contain a Hostname. Traefik will print a warning about this for the TLS setup but it will work. The high priority of 3000 should ensure this rule is evaluated first for incoming requests. The Traefik HTTP Routers rule explicitly does not contain a Hostname. Traefik will print a warning about this for the TLS setup but it will work. The high priority of 3000 should ensure this rule is evaluated first for incoming requests.
Place your robots.txt in the local `./static/` directory and NGINX will serve it for all services behind your Traefik proxy. Place your robots.txt in the local `./static/` directory and NGINX will serve it for all services behind your Traefik proxy.