github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity
411 commits 6 branches 46 tags 797 KiB
Commit graph

411 commits

This branch
This branch
All branches
Author SHA1 Message Date
Glyn Normington
73f6f67adf
 		Merge pull request #125 from holysoles/lint_robots_json v1.30 
lint robots.json during pull requests
 2025-05-15 17:26:15 +01:00
Patrick Evans
498aa50760 lint robots.json during pull requests 2025-05-15 11:15:25 -05:00
ai.robots.txt
1c470babbe Merge pull request #123 from joehoyle/patch-1 
Fix JSON syntax error
 2025-05-15 16:12:30 +00:00
Adam Newbold
84d63916d2
 		Merge pull request #123 from joehoyle/patch-1 
Fix JSON syntax error
 2025-05-15 12:12:21 -04:00
Joe Hoyle
0c56b96fd9
 		Fix JSON syntax error 2025-05-15 11:26:47 -04:00
Cory Dransfeldt
28e69e631b
 		Merge pull request #122 from ai-robots-txt/qualified-bot 
chore(robots.json): adds QualifiedBot crawler
 2025-05-15 07:17:51 -07:00
Cory Dransfeldt
9539256cb3
 		chore(robots.json): adds QualifiedBot crawler 2025-05-15 07:16:07 -07:00
Cory Dransfeldt
9659c88b0c
 		Merge pull request #121 from solution-libre/add-traefik-plugin 
Add Traefik plugin to the README.md file
 2025-05-14 16:45:34 -07:00
Florent Poinsaut
c66d180295
 		Merge branch 'main' into add-traefik-plugin 2025-05-14 22:06:56 +02:00
Glyn Normington
9a9b1b41c0
 		Merge pull request #119 from ai-robots-txt/bing-ai-opt-out-instructions 
Bing AI opt-out instructions
 2025-05-14 19:18:20 +01:00
Florent Poinsaut
b4610a725c Add Traefik plugin 2025-05-14 14:11:56 +02:00
Cory Dransfeldt
36a52a88d8
 		Bing AI opt-out instructions 2025-05-12 20:20:18 -07:00
ai.robots.txt
678380727e Merge pull request #115 from glyn/syntax 
Fix Python syntax error
 2025-05-01 10:29:06 +00:00
Glyn Normington
fb8188c49d
 		Merge pull request #115 from glyn/syntax 
Fix Python syntax error
 2025-05-01 11:28:54 +01:00
Glyn Normington
ec995cd686 Fix Python syntax error 2025-05-01 11:27:40 +01:00
Crazyroostereye
1310dbae46
 		Added a Caddyfile converter (#110) 
Co-authored-by: Julian Beittel <julian@beittel.net>
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-05-01 11:21:32 +01:00
Glyn Normington
91a88e2fa8
 		Merge pull request #113 from rwijnen-um/feature/haproxy 
HAProxy converter added.
 2025-04-28 09:00:16 +01:00
Rik Wijnen
a4a9f2ac2b Tests for HAProxy file added. 2025-04-28 09:30:26 +02:00
Rik Wijnen
66da70905f Fixed incorrect English sentence. 2025-04-28 09:09:40 +02:00
Rik Wijnen
50e739dd73 HAProxy converter added. 2025-04-28 08:51:02 +02:00
ai.robots.txt
c6c7f1748f Update from Dark Visitors 2025-04-26 00:55:12 +00:00
dark-visitors
934ac7b318 Update from Dark Visitors 2025-04-25 00:56:57 +00:00
ai.robots.txt
4654e14e9c Merge pull request #112 from maiavixen/main 
Fixed meta-external* being titlecase, and removed period for consistency
 2025-04-24 07:00:34 +00:00
Glyn Normington
9bf31fbca8
 		Merge pull request #112 from maiavixen/main 
Fixed meta-external* being titlecase, and removed period for consistency
 2025-04-24 08:00:24 +01:00
maia
9d846ced45
 		Update robots.json 
Lowercase meta-external* as that was not technically the UA for the bots, also removed a period in the "respect" for consistency
 2025-04-24 04:08:20 +02:00
dark-visitors
8d25a424d9 Update from Dark Visitors 2025-04-23 00:56:52 +00:00
ai.robots.txt
bbec639c14 Merge pull request #109 from dennislee1/patch-1 v1.29 
AI bots to consider adding
 2025-04-22 14:50:26 +00:00
Cory Dransfeldt
422cf9e29b
 		Merge pull request #109 from dennislee1/patch-1 
AI bots to consider adding
 2025-04-22 07:50:14 -07:00
Dennis Lee
33c5ce1326
 		Update robots.json 
Updated robots list with five new proposed AI bots:

aiHitBot
Cotoyogi
Factset_spyderbot
FirecrawlAgent
TikTokSpider
 2025-04-21 18:55:11 +01:00
Cory Dransfeldt
774b1ddf52
 		Merge pull request #107 from glyn/sponsorship 
Clarify our position on sponsorship
 2025-04-18 11:40:06 -07:00
Glyn Normington
b1856e6988 Donations 2025-04-18 18:40:44 +01:00
Glyn Normington
d05ede8fe1 Clarify our position on sponsorship 
Some firms, including those with .ai domains, have
offered to sponsor this project. So make our position
clear.
 2025-04-18 17:46:56 +01:00
ai.robots.txt
e0cdb278fb Update from Dark Visitors v1.28 2025-04-16 00:57:11 +00:00
dark-visitors
a96e330989 Update from Dark Visitors 2025-04-15 00:57:01 +00:00
Cory Dransfeldt
156e6baa09
 		Merge pull request #105 from jsheard/patch-1 
Include "AI Agents" from Dark Visitors
 2025-04-14 10:08:38 -07:00
Joshua Sheard
d9f882a9b2
 		Include "AI Agents" from Dark Visitors 2025-04-14 15:46:01 +01:00
dark-visitors
305188b2e7 Update from Dark Visitors 2025-04-11 00:55:52 +00:00
ai.robots.txt
4a764bba18 Merge pull request #102 from ai-robots-txt/imgproxy-bot v1.27 
chore(robots.json): adds imgproxy crawler
 2025-04-10 19:22:34 +00:00
Cory Dransfeldt
a891ad7213
 		Merge pull request #102 from ai-robots-txt/imgproxy-bot 
chore(robots.json): adds imgproxy crawler
 2025-04-10 12:22:23 -07:00
Cory Dransfeldt
b65f45e408
 		chore(robots.json): adds imgproxy crawler 2025-04-10 10:12:51 -07:00
Glyn Normington
49e58b1573
 		Merge pull request #100 from fbartho/fb/fix-perplexity-users 
Fix html-mangled hyphen in 'Perplexity-Users' bot name
 2025-04-05 17:32:19 +01:00
Frederic Barthelemy
c6f308cbd0
 		PR Feedback: log special-case, comment consistency 2025-04-05 09:01:52 -07:00
Frederic Barthelemy
5f5a89c38c
 		Fix html-mangled hyphen in Perplexity-Users 
Fixes: #99
 2025-04-04 17:34:14 -07:00
Frederic Barthelemy
6b0349f37d
 		fix python complaining about f-string syntax 
```
python code/tests.py
Traceback (most recent call last):
  File "/Users/fbarthelemy/Code/ai.robots.txt/code/tests.py", line 7, in <module>
    from robots import json_to_txt, json_to_table, json_to_htaccess, json_to_nginx
  File "/Users/fbarthelemy/Code/ai.robots.txt/code/robots.py", line 144
    return f"({"|".join(map(re.escape, lst))})"
                ^
SyntaxError: f-string: expecting '}'
```
 2025-04-04 15:20:30 -07:00
ai.robots.txt
5b8650b99b Update from Dark Visitors 2025-03-29 00:54:10 +00:00
dark-visitors
c249de99a3 Update from Dark Visitors 2025-03-28 00:54:28 +00:00
Cory Dransfeldt
ec18af7624
 		Revert "Merge pull request #91 from deyigifts/perplexity-user" 
This reverts commit 68d1d93714.
 2025-03-27 12:51:22 -07:00
ai.robots.txt
6851413c52 Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export 
Implement Nginx configuration snippet export
 2025-03-27 19:49:15 +00:00
Glyn Normington
dba03d809c
 		Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export 
Implement Nginx configuration snippet export
 2025-03-27 19:49:05 +00:00
ai.robots.txt
68d1d93714 Merge pull request #91 from deyigifts/perplexity-user 
Update perplexity bots
 2025-03-27 19:29:30 +00:00