420 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Cory Dransfeldt	8b151b2cdc	Update README.md ... Co-authored-by: Glyn Normington <glyn.normington@gmail.com>	2025-05-21 06:52:36 -07:00
Cory Dransfeldt	8a8001cbec	chore(README): updates the opening line of our README to clarify the types of agents we block	2025-05-20 13:55:25 -07:00
Glyn Normington	fe1267e290	Merge pull request #131 from Mihitoko/mention-x-robots-tag-for-bing ... Mention X-Robots-Tag header as alternative for bing	2025-05-20 07:52:32 +01:00
Mihitoko	9297c7dfa3	Mention X-Robots-Tag header as alternative for bing	2025-05-20 00:10:05 +02:00
dark-visitors	7a2e6cba52	Update from Dark Visitors	2025-05-17 00:57:28 +00:00
ai.robots.txt	dd1ed174b7	Merge pull request #129 from ai-robots-txt/google-cloudvertexbot ... chore(robots.json): adds Google-CloudVertexBot	2025-05-16 11:35:15 +00:00
Glyn Normington	89c0fbaf86	Merge pull request #129 from ai-robots-txt/google-cloudvertexbot ... chore(robots.json): adds Google-CloudVertexBot	2025-05-16 12:35:04 +01:00
Cory Dransfeldt	ca918a963f	chore(robots.json): adds Google-CloudVertexBot	2025-05-15 21:16:49 -07:00
dark-visitors	16d1de7094	Update from Dark Visitors	2025-05-16 00:59:08 +00:00
Glyn Normington	73f6f67adf	Merge pull request #125 from holysoles/lint_robots_json ... v1.30 lint robots.json during pull requests	2025-05-15 17:26:15 +01:00
Patrick Evans	498aa50760	lint robots.json during pull requests	2025-05-15 11:15:25 -05:00
ai.robots.txt	1c470babbe	Merge pull request #123 from joehoyle/patch-1 ... Fix JSON syntax error	2025-05-15 16:12:30 +00:00
Adam Newbold	84d63916d2	Merge pull request #123 from joehoyle/patch-1 ... Fix JSON syntax error	2025-05-15 12:12:21 -04:00
Joe Hoyle	0c56b96fd9	Fix JSON syntax error	2025-05-15 11:26:47 -04:00
Cory Dransfeldt	28e69e631b	Merge pull request #122 from ai-robots-txt/qualified-bot ... chore(robots.json): adds QualifiedBot crawler	2025-05-15 07:17:51 -07:00
Cory Dransfeldt	9539256cb3	chore(robots.json): adds QualifiedBot crawler	2025-05-15 07:16:07 -07:00
Cory Dransfeldt	9659c88b0c	Merge pull request #121 from solution-libre/add-traefik-plugin ... Add Traefik plugin to the README.md file	2025-05-14 16:45:34 -07:00
Florent Poinsaut	c66d180295	Merge branch 'main' into add-traefik-plugin	2025-05-14 22:06:56 +02:00
Glyn Normington	9a9b1b41c0	Merge pull request #119 from ai-robots-txt/bing-ai-opt-out-instructions ... Bing AI opt-out instructions	2025-05-14 19:18:20 +01:00
Florent Poinsaut	b4610a725c	Add Traefik plugin	2025-05-14 14:11:56 +02:00
Cory Dransfeldt	36a52a88d8	Bing AI opt-out instructions	2025-05-12 20:20:18 -07:00
ai.robots.txt	678380727e	Merge pull request #115 from glyn/syntax ... Fix Python syntax error	2025-05-01 10:29:06 +00:00
Glyn Normington	fb8188c49d	Merge pull request #115 from glyn/syntax ... Fix Python syntax error	2025-05-01 11:28:54 +01:00
Glyn Normington	ec995cd686	Fix Python syntax error	2025-05-01 11:27:40 +01:00
Crazyroostereye	1310dbae46	Added a Caddyfile converter (#110) ... Co-authored-by: Julian Beittel <julian@beittel.net> Co-authored-by: Glyn Normington <work@underlap.org>	2025-05-01 11:21:32 +01:00
Glyn Normington	91a88e2fa8	Merge pull request #113 from rwijnen-um/feature/haproxy ... HAProxy converter added.	2025-04-28 09:00:16 +01:00
Rik Wijnen	a4a9f2ac2b	Tests for HAProxy file added.	2025-04-28 09:30:26 +02:00
Rik Wijnen	66da70905f	Fixed incorrect English sentence.	2025-04-28 09:09:40 +02:00
Rik Wijnen	50e739dd73	HAProxy converter added.	2025-04-28 08:51:02 +02:00
ai.robots.txt	c6c7f1748f	Update from Dark Visitors	2025-04-26 00:55:12 +00:00
dark-visitors	934ac7b318	Update from Dark Visitors	2025-04-25 00:56:57 +00:00
ai.robots.txt	4654e14e9c	Merge pull request #112 from maiavixen/main ... Fixed meta-external* being titlecase, and removed period for consistency	2025-04-24 07:00:34 +00:00
Glyn Normington	9bf31fbca8	Merge pull request #112 from maiavixen/main ... Fixed meta-external* being titlecase, and removed period for consistency	2025-04-24 08:00:24 +01:00
maia	9d846ced45	Update robots.json ... Lowercase meta-external* as that was not technically the UA for the bots, also removed a period in the "respect" for consistency	2025-04-24 04:08:20 +02:00
dark-visitors	8d25a424d9	Update from Dark Visitors	2025-04-23 00:56:52 +00:00
ai.robots.txt	bbec639c14	Merge pull request #109 from dennislee1/patch-1 ... v1.29 AI bots to consider adding	2025-04-22 14:50:26 +00:00
Cory Dransfeldt	422cf9e29b	Merge pull request #109 from dennislee1/patch-1 ... AI bots to consider adding	2025-04-22 07:50:14 -07:00
Dennis Lee	33c5ce1326	Update robots.json ... Updated robots list with five new proposed AI bots: aiHitBot Cotoyogi Factset_spyderbot FirecrawlAgent TikTokSpider	2025-04-21 18:55:11 +01:00
Cory Dransfeldt	774b1ddf52	Merge pull request #107 from glyn/sponsorship ... Clarify our position on sponsorship	2025-04-18 11:40:06 -07:00
Glyn Normington	b1856e6988	Donations	2025-04-18 18:40:44 +01:00
Glyn Normington	d05ede8fe1	Clarify our position on sponsorship ... Some firms, including those with .ai domains, have offered to sponsor this project. So make our position clear.	2025-04-18 17:46:56 +01:00
ai.robots.txt	e0cdb278fb	Update from Dark Visitors v1.28	2025-04-16 00:57:11 +00:00
dark-visitors	a96e330989	Update from Dark Visitors	2025-04-15 00:57:01 +00:00
Cory Dransfeldt	156e6baa09	Merge pull request #105 from jsheard/patch-1 ... Include "AI Agents" from Dark Visitors	2025-04-14 10:08:38 -07:00
Joshua Sheard	d9f882a9b2	Include "AI Agents" from Dark Visitors	2025-04-14 15:46:01 +01:00
dark-visitors	305188b2e7	Update from Dark Visitors	2025-04-11 00:55:52 +00:00
ai.robots.txt	4a764bba18	Merge pull request #102 from ai-robots-txt/imgproxy-bot ... v1.27 chore(robots.json): adds imgproxy crawler	2025-04-10 19:22:34 +00:00
Cory Dransfeldt	a891ad7213	Merge pull request #102 from ai-robots-txt/imgproxy-bot ... chore(robots.json): adds imgproxy crawler	2025-04-10 12:22:23 -07:00
Cory Dransfeldt	b65f45e408	chore(robots.json): adds imgproxy crawler	2025-04-10 10:12:51 -07:00
Glyn Normington	49e58b1573	Merge pull request #100 from fbartho/fb/fix-perplexity-users ... Fix html-mangled hyphen in 'Perplexity-Users' bot name	2025-04-05 17:32:19 +01:00