356 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
deyigifts	6ecfcdfcbf	Update perplexity bot ... Update based on perplexity bot docs	2025-03-24 14:16:57 +08:00
Cory Dransfeldt	5e7c3c432f	Merge pull request #83 from glyn/81-doc-testing ... Document testing in README	2025-02-19 09:19:44 -08:00
Glyn Normington	9f41d4c11c	Merge pull request #84 from sideeffect42/tests-workflow ... Add run-tests workflow	2025-02-18 19:42:55 +00:00
Dennis Camera	8a74896333	Add workflow to run tests on pull request or push to main	2025-02-18 20:30:27 +01:00
Glyn Normington	1d55a205e4	Document testing in README ... Fixes: https://github.com/ai-robots-txt/ai.robots.txt/issues/81	2025-02-18 16:49:08 +00:00
Glyn Normington	8494a7fcaa	Merge pull request #80 from sideeffect42/htaccess-allow-robots_txt ... .htaccess: Allow robots access to `/robots.txt`	2025-02-18 16:42:36 +00:00
Dennis Camera	c7c1e7b96f	robots.py: Make executable	2025-02-18 12:55:17 +01:00
Dennis Camera	17b826a6d3	Update tests and convert to stock unittest ... For these simple tests Python's built-in unittest framework is more than enough. No additional dependencies are required. Added some more test cases with "special" characters to test the escaping code better.	2025-02-18 12:55:15 +01:00
Dennis Camera	0bd3fa63b8	table-of-bot-metrics.md: Escape robot names for Markdown table ... Some characters which could occur in a crawler's name have a special meaning in Markdown. They are escaped to prevent them from having unintended side effects. The escaping is only applied to the first (Name) column of the table. The rest of the columns is expected to already be Markdown encoded in robots.json.	2025-02-18 12:53:27 +01:00
Dennis Camera	a884a2afb9	.htaccess: Make regex in RewriteCond safe ... Improve the regular expression by removing unneeded anchors and escaping special characters (not just space) to prevent false positives or a misbehaving rewrite rule.	2025-02-18 12:53:22 +01:00
Dennis Camera	c0d418cd87	.htaccess: Allow robots access to /robots.txt	2025-02-18 12:49:29 +01:00
dark-visitors	abfd6dfcd1	Update from Dark Visitors	2025-02-17 00:53:32 +00:00
ai.robots.txt	693289bb29	chore: add Brightbot 1.0 v1.26	2025-02-16 21:37:52 +00:00
Cory Dransfeldt	a9ec4ffa6f	chore: add Brightbot 1.0	2025-02-16 13:36:39 -08:00
Glyn Normington	03aa829913	Merge pull request #79 from always-be-testing/main ... List of AI bots Cloudflare considers "Verified"	2025-02-16 04:33:40 +00:00
always-be-testing	5b13c2e504	add more concise message about verified bots ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-02-15 11:22:10 -05:00
always-be-testing	af87b85d7f	include return after heading	2025-02-14 12:39:08 -05:00
always-be-testing	f99339922f	grammar update and include syntax for verified bot condition	2025-02-14 12:36:33 -05:00
always-be-testing	e396a2ec78	forgot to include heading	2025-02-14 12:31:20 -05:00
always-be-testing	261a2b83b9	update README to inclide list of ai bots Cloudflare considers verified	2025-02-14 12:26:19 -05:00
dark-visitors	bebffccc0c	Update from Dark Visitors v1.25	2025-02-02 00:52:50 +00:00
ai.robots.txt	89d4c6e5ca	Merge pull request #73 from nisbet-hubbard/patch-8 ... Actually block Semrush’s AI tools	2025-02-01 10:51:01 +00:00
Glyn Normington	f9e2c5810b	Merge pull request #73 from nisbet-hubbard/patch-8 ... Actually block Semrush’s AI tools	2025-02-01 10:50:50 +00:00
nisbet-hubbard	05b79b8a58	Update robots.json	2025-01-27 19:41:03 +08:00
dark-visitors	9c060dee1c	Update from Dark Visitors v1.24	2025-01-21 00:49:22 +00:00
ai.robots.txt	6c552a3daa	Merge pull request #71 from jsheard/patch-1 ... Add Crawlspace	2025-01-20 17:45:42 +00:00
Glyn Normington	f621fb4852	Merge pull request #71 from jsheard/patch-1 ... Add Crawlspace	2025-01-20 17:45:29 +00:00
Joshua Sheard	7427d96bac	Update robots.json ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-01-20 10:59:02 +00:00
Glyn Normington	81cc81b35e	Merge pull request #68 from MassiminoilTrace/main ... Implementing automatic htaccess generation	2025-01-20 07:33:54 +00:00
Massimo Gismondi	4f03818280	Removed if condition and added a little comments	2025-01-20 06:51:06 +01:00
Massimo Gismondi	a9956f7825	Removed additional sections	2025-01-20 06:50:48 +01:00
Massimo Gismondi	33c38ee70b	Update README.md ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-01-20 06:28:32 +01:00
Massimo Gismondi	52241bdca6	Update README.md ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-01-20 06:27:56 +01:00
Massimo Gismondi	013b7abfa1	Update README.md ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-01-20 06:27:02 +01:00
Massimo Gismondi	70fd6c0fb1	Add mention of htaccess in readme ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-01-20 06:25:07 +01:00
Joshua Sheard	5aa08bc002	Add Crawlspace	2025-01-19 22:03:50 +00:00
Massimo Gismondi	d65128d10a	Removed paragraph in favour of future FAQ.md ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-01-18 12:41:09 +01:00
Massimo Gismondi	1cc4b59dfc	Shortened htaccess instructions ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-01-18 12:40:03 +01:00
Massimo Gismondi	8aee2f24bb	Fixed space in comment ... Co-authored-by: Glyn Normington <work@underlap.org>	2025-01-18 12:39:07 +01:00
Massimo Gismondi	b455af66e7	Adding clarification about performance and code comment	2025-01-17 21:42:08 +01:00
Massimo Gismondi	189e75bbfd	Adding usage instructions	2025-01-17 21:25:23 +01:00
Massimo Gismondi	933aa6159d	Implementing htaccess generation	2025-01-07 11:02:29 +01:00
Glyn Normington	b7f908e305	Merge pull request #66 from fabianegli/patch-1 ... v1.23 Allow Action to succeed even if no changes were made	2025-01-07 03:54:40 +00:00
ai.robots.txt	ec454b71d3	Merge pull request #67 from Nightfirecat/semrushbot ... Block SemrushBot	2025-01-06 20:51:56 +00:00
Cory Dransfeldt	565dca3dc0	Merge pull request #67 from Nightfirecat/semrushbot ... Block SemrushBot	2025-01-06 12:51:43 -08:00
Jordan Atwood	143f8f2285	Block SemrushBot	2025-01-06 12:34:38 -08:00
Cory Dransfeldt	8e98cc6049	Merge pull request #61 from glyn/improve-naming ... Rename Python code	2025-01-06 08:10:47 -08:00
Fabian Egli	30ee957011	bail when NO changes are staged	2025-01-06 12:05:42 +01:00
Fabian Egli	83cd546470	allow Action to succeed even if no changes were made ... Before, the Action would fail in case there were no changes made to any files by the converter.	2025-01-06 11:39:41 +01:00
ai.robots.txt	ca8620e28b	Merge pull request #63 from glyn/push-paths ... v1.22 Convert robots.json more frequently	2025-01-05 05:05:20 +00:00