mirror of
https://github.com/ai-robots-txt/ai.robots.txt.git
synced 2025-12-29 12:18:33 +01:00
Merge pull request #205 from fiskhandlarn/fix/editorconfig
Fix/editorconfig
This commit is contained in:
commit
56010ef913
3 changed files with 8 additions and 8 deletions
|
|
@ -4,3 +4,6 @@ root = true
|
|||
end_of_line = lf
|
||||
insert_final_newline = true
|
||||
trim_trailing_whitespace = true
|
||||
|
||||
[{Caddyfile,haproxy-block-ai-bots.txt,nginx-block-ai-bots.conf}]
|
||||
insert_final_newline = false
|
||||
|
|
|
|||
|
|
@ -103,5 +103,3 @@ But even if you don't use Cloudflare's hard block, their list of [verified bots]
|
|||
- [Blockin' bots on Netlify](https://www.jeremiak.com/blog/block-bots-netlify-edge-functions/) by Jeremia Kimelman
|
||||
- [Blocking AI web crawlers](https://underlap.org/blocking-ai-web-crawlers) by Glyn Normington
|
||||
- [Block AI Bots from Crawling Websites Using Robots.txt](https://originality.ai/ai-bot-blocking) by Jonathan Gillham, Originality.AI
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -31,7 +31,6 @@ networks:
|
|||
external: true
|
||||
```
|
||||
|
||||
|
||||
The Traefik HTTP Routers rule explicitly does not contain a Hostname. Traefik will print a warning about this for the TLS setup but it will work. The high priority of 3000 should ensure this rule is evaluated first for incoming requests.
|
||||
|
||||
Place your robots.txt in the local `./static/` directory and NGINX will serve it for all services behind your Traefik proxy.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue