github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity
366 commits 6 branches 46 tags 797 KiB
Commit graph

366 commits

This branch
This branch
All branches
Author SHA1 Message Date
dark-visitors
c249de99a3 Update from Dark Visitors 2025-03-28 00:54:28 +00:00
Cory Dransfeldt
ec18af7624
 		Revert "Merge pull request #91 from deyigifts/perplexity-user" 
This reverts commit 68d1d93714.
 2025-03-27 12:51:22 -07:00
ai.robots.txt
6851413c52 Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export 
Implement Nginx configuration snippet export
 2025-03-27 19:49:15 +00:00
Glyn Normington
dba03d809c
 		Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export 
Implement Nginx configuration snippet export
 2025-03-27 19:49:05 +00:00
ai.robots.txt
68d1d93714 Merge pull request #91 from deyigifts/perplexity-user 
Update perplexity bots
 2025-03-27 19:29:30 +00:00
Cory Dransfeldt
1183187be9
 		Merge pull request #91 from deyigifts/perplexity-user 
Update perplexity bots
 2025-03-27 12:29:21 -07:00
Thomas Leister
7c3b5a2cb2
 		Add tests for Nginx config generator 2025-03-27 18:28:21 +01:00
Thomas Leister
4f3f4cd0dd
 		Add assembled version of nginx-block-ai-bots.conf file 2025-03-27 12:43:36 +01:00
Thomas Leister
5a312c5f4d
 		Mention Nginx config feature in README 2025-03-27 12:43:29 +01:00
Thomas Leister
da85207314
 		Implement new function "json_to_nginx" which outputs an Nginx 
configuration snippet
 2025-03-27 12:27:09 +01:00
deyigifts
6ecfcdfcbf
 		Update perplexity bot 
Update based on perplexity bot docs
 2025-03-24 14:16:57 +08:00
Cory Dransfeldt
5e7c3c432f
 		Merge pull request #83 from glyn/81-doc-testing 
Document testing in README
 2025-02-19 09:19:44 -08:00
Glyn Normington
9f41d4c11c
 		Merge pull request #84 from sideeffect42/tests-workflow 
Add run-tests workflow
 2025-02-18 19:42:55 +00:00
Dennis Camera
8a74896333 Add workflow to run tests on pull request or push to main 2025-02-18 20:30:27 +01:00
Glyn Normington
1d55a205e4 Document testing in README 
Fixes: https://github.com/ai-robots-txt/ai.robots.txt/issues/81
 2025-02-18 16:49:08 +00:00
Glyn Normington
8494a7fcaa
 		Merge pull request #80 from sideeffect42/htaccess-allow-robots_txt 
.htaccess: Allow robots access to `/robots.txt`
 2025-02-18 16:42:36 +00:00
Dennis Camera
c7c1e7b96f robots.py: Make executable 2025-02-18 12:55:17 +01:00
Dennis Camera
17b826a6d3 Update tests and convert to stock unittest 
For these simple tests Python's built-in unittest framework is more than enough.
No additional dependencies are required.

Added some more test cases with "special" characters to test the escaping code
better.
 2025-02-18 12:55:15 +01:00
Dennis Camera
0bd3fa63b8 table-of-bot-metrics.md: Escape robot names for Markdown table 
Some characters which could occur in a crawler's name have a special meaning in
Markdown. They are escaped to prevent them from having unintended side effects.

The escaping is only applied to the first (Name) column of the table. The rest
of the columns is expected to already be Markdown encoded in robots.json.
 2025-02-18 12:53:27 +01:00
Dennis Camera
a884a2afb9 .htaccess: Make regex in RewriteCond safe 
Improve the regular expression by removing unneeded anchors and
escaping special characters (not just space) to prevent false positives
or a misbehaving rewrite rule.
 2025-02-18 12:53:22 +01:00
Dennis Camera
c0d418cd87 .htaccess: Allow robots access to /robots.txt 2025-02-18 12:49:29 +01:00
dark-visitors
abfd6dfcd1 Update from Dark Visitors 2025-02-17 00:53:32 +00:00
ai.robots.txt
693289bb29 chore: add Brightbot 1.0 v1.26 2025-02-16 21:37:52 +00:00
Cory Dransfeldt
a9ec4ffa6f
 		chore: add Brightbot 1.0 2025-02-16 13:36:39 -08:00
Glyn Normington
03aa829913
 		Merge pull request #79 from always-be-testing/main 
List of AI bots Cloudflare considers "Verified"
 2025-02-16 04:33:40 +00:00
always-be-testing
5b13c2e504
 		add more concise message about verified bots 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-02-15 11:22:10 -05:00
always-be-testing
af87b85d7f include return after heading 2025-02-14 12:39:08 -05:00
always-be-testing
f99339922f grammar update and include syntax for verified bot condition 2025-02-14 12:36:33 -05:00
always-be-testing
e396a2ec78 forgot to include heading 2025-02-14 12:31:20 -05:00
always-be-testing
261a2b83b9 update README to inclide list of ai bots Cloudflare considers verified 2025-02-14 12:26:19 -05:00
dark-visitors
bebffccc0c Update from Dark Visitors v1.25 2025-02-02 00:52:50 +00:00
ai.robots.txt
89d4c6e5ca Merge pull request #73 from nisbet-hubbard/patch-8 
Actually block Semrush’s AI tools
 2025-02-01 10:51:01 +00:00
Glyn Normington
f9e2c5810b
 		Merge pull request #73 from nisbet-hubbard/patch-8 
Actually block Semrush’s AI tools
 2025-02-01 10:50:50 +00:00
nisbet-hubbard
05b79b8a58
 		Update robots.json 2025-01-27 19:41:03 +08:00
dark-visitors
9c060dee1c Update from Dark Visitors v1.24 2025-01-21 00:49:22 +00:00
ai.robots.txt
6c552a3daa Merge pull request #71 from jsheard/patch-1 
Add Crawlspace
 2025-01-20 17:45:42 +00:00
Glyn Normington
f621fb4852
 		Merge pull request #71 from jsheard/patch-1 
Add Crawlspace
 2025-01-20 17:45:29 +00:00
Joshua Sheard
7427d96bac
 		Update robots.json 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-01-20 10:59:02 +00:00
Glyn Normington
81cc81b35e
 		Merge pull request #68 from MassiminoilTrace/main 
Implementing automatic htaccess generation
 2025-01-20 07:33:54 +00:00
Massimo Gismondi
4f03818280 Removed if condition and added a little comments 2025-01-20 06:51:06 +01:00
Massimo Gismondi
a9956f7825 Removed additional sections 2025-01-20 06:50:48 +01:00
Massimo Gismondi
33c38ee70b
 		Update README.md 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-01-20 06:28:32 +01:00
Massimo Gismondi
52241bdca6
 		Update README.md 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-01-20 06:27:56 +01:00
Massimo Gismondi
013b7abfa1
 		Update README.md 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-01-20 06:27:02 +01:00
Massimo Gismondi
70fd6c0fb1
 		Add mention of htaccess in readme 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-01-20 06:25:07 +01:00
Joshua Sheard
5aa08bc002
 		Add Crawlspace 2025-01-19 22:03:50 +00:00
Massimo Gismondi
d65128d10a
 		Removed paragraph in favour of future FAQ.md 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-01-18 12:41:09 +01:00
Massimo Gismondi
1cc4b59dfc
 		Shortened htaccess instructions 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-01-18 12:40:03 +01:00
Massimo Gismondi
8aee2f24bb
 		Fixed space in comment 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-01-18 12:39:07 +01:00
Massimo Gismondi
b455af66e7 Adding clarification about performance and code comment 2025-01-17 21:42:08 +01:00