github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity

Improve contributing section and fix formatting in README

Browse source
This commit is contained in:
Anshita-18H 2025-11-27 17:56:47 +05:30
commit 4302fd1aca
1 changed files with 12 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

14
README.md
View file

@ -49,9 +49,17 @@ file on-the-fly.
A note about contributing: updates should be added/made to `robots.json`. A GitHub action will then generate the updated `robots.txt`, `table-of-bot-metrics.md`, `.htaccess` and `nginx-block-ai-bots.conf`.
You can run the tests by [installing](https://www.python.org/about/gettingstarted/) Python 3 and issuing:
You can run the tests by [installing Python 3 and issuing:
```console
code/tests.py
python code/tests.py
### Installing Dependencies
Before running the tests, install all required Python packages:
pip install -r requirements.txt
```
## Releasing
@ -97,3 +105,5 @@ But even if you don't use Cloudflare's hard block, their list of [verified bots]
- [Blockin' bots on Netlify](https://www.jeremiak.com/blog/block-bots-netlify-edge-functions/) by Jeremia Kimelman
- [Blocking AI web crawlers](https://underlap.org/blocking-ai-web-crawlers) by Glyn Normington
- [Block AI Bots from Crawling Websites Using Robots.txt](https://originality.ai/ai-bot-blocking) by Jonathan Gillham, Originality.AI