github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity
542 commits 6 branches 46 tags 797 KiB
Commit graph

542 commits

This branch
This branch
All branches
Author SHA1 Message Date
Glyn Normington
9bf31fbca8
 		Merge pull request #112 from maiavixen/main 
Fixed meta-external* being titlecase, and removed period for consistency
 2025-04-24 08:00:24 +01:00
maia
9d846ced45
 		Update robots.json 
Lowercase meta-external* as that was not technically the UA for the bots, also removed a period in the "respect" for consistency
 2025-04-24 04:08:20 +02:00
dark-visitors
8d25a424d9 Update from Dark Visitors 2025-04-23 00:56:52 +00:00
ai.robots.txt
bbec639c14 Merge pull request #109 from dennislee1/patch-1 v1.29 
AI bots to consider adding
 2025-04-22 14:50:26 +00:00
Cory Dransfeldt
422cf9e29b
 		Merge pull request #109 from dennislee1/patch-1 
AI bots to consider adding
 2025-04-22 07:50:14 -07:00
Dennis Lee
33c5ce1326
 		Update robots.json 
Updated robots list with five new proposed AI bots:

aiHitBot
Cotoyogi
Factset_spyderbot
FirecrawlAgent
TikTokSpider
 2025-04-21 18:55:11 +01:00
Cory Dransfeldt
774b1ddf52
 		Merge pull request #107 from glyn/sponsorship 
Clarify our position on sponsorship
 2025-04-18 11:40:06 -07:00
Glyn Normington
b1856e6988 Donations 2025-04-18 18:40:44 +01:00
Glyn Normington
d05ede8fe1 Clarify our position on sponsorship 
Some firms, including those with .ai domains, have
offered to sponsor this project. So make our position
clear.
 2025-04-18 17:46:56 +01:00
Kyle Buckingham
fd41de8522
 		Update robots.json 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-04-16 16:43:03 -07:00
Kyle Buckingham
4a6f37d727
 		Update robots.json 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-04-16 16:42:58 -07:00
ai.robots.txt
e0cdb278fb Update from Dark Visitors v1.28 2025-04-16 00:57:11 +00:00
dark-visitors
a96e330989 Update from Dark Visitors 2025-04-15 00:57:01 +00:00
Cory Dransfeldt
156e6baa09
 		Merge pull request #105 from jsheard/patch-1 
Include "AI Agents" from Dark Visitors
 2025-04-14 10:08:38 -07:00
Joshua Sheard
d9f882a9b2
 		Include "AI Agents" from Dark Visitors 2025-04-14 15:46:01 +01:00
dark-visitors
305188b2e7 Update from Dark Visitors 2025-04-11 00:55:52 +00:00
ai.robots.txt
4a764bba18 Merge pull request #102 from ai-robots-txt/imgproxy-bot v1.27 
chore(robots.json): adds imgproxy crawler
 2025-04-10 19:22:34 +00:00
Cory Dransfeldt
a891ad7213
 		Merge pull request #102 from ai-robots-txt/imgproxy-bot 
chore(robots.json): adds imgproxy crawler
 2025-04-10 12:22:23 -07:00
Cory Dransfeldt
b65f45e408
 		chore(robots.json): adds imgproxy crawler 2025-04-10 10:12:51 -07:00
Glyn Normington
49e58b1573
 		Merge pull request #100 from fbartho/fb/fix-perplexity-users 
Fix html-mangled hyphen in 'Perplexity-Users' bot name
 2025-04-05 17:32:19 +01:00
Frederic Barthelemy
c6f308cbd0
 		PR Feedback: log special-case, comment consistency 2025-04-05 09:01:52 -07:00
Frederic Barthelemy
5f5a89c38c
 		Fix html-mangled hyphen in Perplexity-Users 
Fixes: #99
 2025-04-04 17:34:14 -07:00
Frederic Barthelemy
6b0349f37d
 		fix python complaining about f-string syntax 
```
python code/tests.py
Traceback (most recent call last):
  File "/Users/fbarthelemy/Code/ai.robots.txt/code/tests.py", line 7, in <module>
    from robots import json_to_txt, json_to_table, json_to_htaccess, json_to_nginx
  File "/Users/fbarthelemy/Code/ai.robots.txt/code/robots.py", line 144
    return f"({"|".join(map(re.escape, lst))})"
                ^
SyntaxError: f-string: expecting '}'
```
 2025-04-04 15:20:30 -07:00
Kyle Buckingham
8dc36aa2e2
 		Update robots.txt 2025-04-01 15:23:28 -07:00
Kyle Buckingham
ae8f74c10c
 		Update robots.json 2025-04-01 15:22:04 -07:00
ai.robots.txt
5b8650b99b Update from Dark Visitors 2025-03-29 00:54:10 +00:00
dark-visitors
c249de99a3 Update from Dark Visitors 2025-03-28 00:54:28 +00:00
Cory Dransfeldt
ec18af7624
 		Revert "Merge pull request #91 from deyigifts/perplexity-user" 
This reverts commit 68d1d93714.
 2025-03-27 12:51:22 -07:00
ai.robots.txt
6851413c52 Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export 
Implement Nginx configuration snippet export
 2025-03-27 19:49:15 +00:00
Glyn Normington
dba03d809c
 		Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export 
Implement Nginx configuration snippet export
 2025-03-27 19:49:05 +00:00
ai.robots.txt
68d1d93714 Merge pull request #91 from deyigifts/perplexity-user 
Update perplexity bots
 2025-03-27 19:29:30 +00:00
Cory Dransfeldt
1183187be9
 		Merge pull request #91 from deyigifts/perplexity-user 
Update perplexity bots
 2025-03-27 12:29:21 -07:00
Thomas Leister
7c3b5a2cb2
 		Add tests for Nginx config generator 2025-03-27 18:28:21 +01:00
Thomas Leister
4f3f4cd0dd
 		Add assembled version of nginx-block-ai-bots.conf file 2025-03-27 12:43:36 +01:00
Thomas Leister
5a312c5f4d
 		Mention Nginx config feature in README 2025-03-27 12:43:29 +01:00
Thomas Leister
da85207314
 		Implement new function "json_to_nginx" which outputs an Nginx 
configuration snippet
 2025-03-27 12:27:09 +01:00
deyigifts
6ecfcdfcbf
 		Update perplexity bot 
Update based on perplexity bot docs
 2025-03-24 14:16:57 +08:00
Cory Dransfeldt
5e7c3c432f
 		Merge pull request #83 from glyn/81-doc-testing 
Document testing in README
 2025-02-19 09:19:44 -08:00
Glyn Normington
9f41d4c11c
 		Merge pull request #84 from sideeffect42/tests-workflow 
Add run-tests workflow
 2025-02-18 19:42:55 +00:00
Dennis Camera
8a74896333 Add workflow to run tests on pull request or push to main 2025-02-18 20:30:27 +01:00
Glyn Normington
1d55a205e4 Document testing in README 
Fixes: https://github.com/ai-robots-txt/ai.robots.txt/issues/81
 2025-02-18 16:49:08 +00:00
Glyn Normington
8494a7fcaa
 		Merge pull request #80 from sideeffect42/htaccess-allow-robots_txt 
.htaccess: Allow robots access to `/robots.txt`
 2025-02-18 16:42:36 +00:00
Dennis Camera
c7c1e7b96f robots.py: Make executable 2025-02-18 12:55:17 +01:00
Dennis Camera
17b826a6d3 Update tests and convert to stock unittest 
For these simple tests Python's built-in unittest framework is more than enough.
No additional dependencies are required.

Added some more test cases with "special" characters to test the escaping code
better.
 2025-02-18 12:55:15 +01:00
Dennis Camera
0bd3fa63b8 table-of-bot-metrics.md: Escape robot names for Markdown table 
Some characters which could occur in a crawler's name have a special meaning in
Markdown. They are escaped to prevent them from having unintended side effects.

The escaping is only applied to the first (Name) column of the table. The rest
of the columns is expected to already be Markdown encoded in robots.json.
 2025-02-18 12:53:27 +01:00
Dennis Camera
a884a2afb9 .htaccess: Make regex in RewriteCond safe 
Improve the regular expression by removing unneeded anchors and
escaping special characters (not just space) to prevent false positives
or a misbehaving rewrite rule.
 2025-02-18 12:53:22 +01:00
Dennis Camera
c0d418cd87 .htaccess: Allow robots access to /robots.txt 2025-02-18 12:49:29 +01:00
dark-visitors
abfd6dfcd1 Update from Dark Visitors 2025-02-17 00:53:32 +00:00
ai.robots.txt
693289bb29 chore: add Brightbot 1.0 v1.26 2025-02-16 21:37:52 +00:00
Cory Dransfeldt
a9ec4ffa6f
 		chore: add Brightbot 1.0 2025-02-16 13:36:39 -08:00