595 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Adam Newbold	6d75f3c1c9	Update robots.py to address error on line 57 ... Attempting to work around an error that prevents parsing the Dark Visitors site	2025-12-01 20:18:29 -05:00
Glyn Normington	56010ef913	Merge pull request #205 from fiskhandlarn/fix/editorconfig ... Fix/editorconfig	2025-11-29 10:02:08 +00:00
ai.robots.txt	3fadc88a23	Merge pull request #206 from newbold/main ... Adding LAIONDownloader	2025-11-29 10:00:52 +00:00
Glyn Normington	47c077a8ef	Merge pull request #206 from newbold/main ... Adding LAIONDownloader	2025-11-29 10:00:42 +00:00
Adam Newbold	f5d7ccb243	Fixed invalid JSON	2025-11-28 14:10:38 -05:00
Adam Newbold	30d719a09a	Adding LAIONDownloader	2025-11-28 14:08:18 -05:00
fiskhandlarn	05bbdebeaa	feat: disallow final newline for files generated by python ... if any of these files have ending newlines the tests will fail	2025-11-28 10:47:22 +01:00
fiskhandlarn	c6ce9329a1	fix: ensure whitespace as defined in .editorconfig	2025-11-28 10:46:22 +01:00
ai.robots.txt	10d5ae2870	Merge pull request #200 from glyn/deepseek ... Clarify that DeepSeekBot does not respect robots.txt	2025-11-27 14:00:50 +00:00
Glyn Normington	4467002298	Merge pull request #200 from glyn/deepseek ... Clarify that DeepSeekBot does not respect robots.txt	2025-11-27 14:00:40 +00:00
Glyn Normington	4a159a818f	Merge pull request #199 from glyn/editorconfig ... Standardise editor options	2025-11-27 13:54:03 +00:00
Glyn Normington	3d6b33a71a	Merge pull request #202 from glyn/formatting ... Tidy README	2025-11-27 12:42:31 +00:00
Glyn Normington	c26c8c0911	Tidy README	2025-11-27 12:41:37 +00:00
Glyn Normington	91959fe791	Merge pull request #201 from Anshita-18H/add-requirements-file ... Add requirements.txt with project dependencies	2025-11-27 12:38:28 +00:00
Glyn Normington	b75163e796	Ensure Python3 is used	2025-11-27 12:38:12 +00:00
Glyn Normington	f46754d280	Order deps	2025-11-27 12:37:35 +00:00
Glyn Normington	c3f2fe758e	Whitespace	2025-11-27 12:37:21 +00:00
Glyn Normington	7521c3af50	Fix link	2025-11-27 12:37:07 +00:00
Anshita-18H	4302fd1aca	Improve contributing section and fix formatting in README	2025-11-27 17:56:47 +05:30
Anshita-18H	9ca5033927	Deduplicate requirements.txt and add installation instructions to README	2025-11-27 17:44:47 +05:30
Anshita-18H	9cc3dbc05f	Add requirements.txt with project dependencies	2025-11-27 17:06:59 +05:30
Glyn Normington	c322d7852d	Merge pull request #130 from maxheadroom/main ... Create traefik-manual-setup.md	2025-11-26 12:08:11 +00:00
Glyn Normington	35588b2ddb	Typo.	2025-11-26 12:07:39 +00:00
Glyn Normington	ab24e41106	Clarify that DeepSeekBot does not respect robots.txt ... Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/198	2025-11-26 12:02:42 +00:00
Glyn Normington	b681c0c0d8	Standardise editor options ... This was motivated by @fiskhandlarn's comment: https://github.com/ai-robots-txt/ai.robots.txt/pull/195#issuecomment-3576331322	2025-11-26 11:54:10 +00:00
Glyn Normington	4e7e28335f	Merge pull request #195 from fiskhandlarn/patch-2 ... feat: allow robots access to `/robots.txt` in nginx	2025-11-26 11:50:15 +00:00
fiskhandlarn	a6cf6b204b	test: update test nginx conf	2025-11-25 16:47:04 +01:00
fiskhandlarn	2679fcad34	feat: update nginx generator	2025-11-25 16:46:31 +01:00
fiskhandlarn	ef8eda4fe6	chore: normalize quote style	2025-11-25 16:39:35 +01:00
Adam Newbold	a29102f0fc	Merge pull request #196 from glyn/releasing ... v1.44 Document how to ship a new release	2025-11-25 10:34:25 -05:00
fiskhandlarn	0b3266b35f	feat: allow robots access to /robots.txt in nginx	2025-11-25 16:22:14 +01:00
Glyn Normington	6225e3e98e	Document how to ship a new release	2025-11-23 04:01:43 +00:00
ai.robots.txt	be4d74412c	Merge pull request #194 from glyn/193-kendra ... delete extraneous hyphen	2025-11-21 16:32:11 +00:00
Cory Dransfeldt	729be4693a	Merge pull request #194 from glyn/193-kendra ... delete extraneous hyphen	2025-11-21 08:32:01 -08:00
Glyn Normington	efb4d260da	delete extraneous hyphen ... Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/193	2025-11-20 10:23:48 +00:00
ai.robots.txt	e2726ac160	Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo ... v1.43 chore: adds NotebookLM and KlaviyoAIBot agents	2025-11-14 21:01:25 +00:00
Glyn Normington	663d030f96	Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo ... chore: adds NotebookLM and KlaviyoAIBot agents	2025-11-14 21:01:13 +00:00
Cory Dransfeldt	28b45ea08d	chore: adds NotebookLM and KlaviyoAIBot agents	2025-11-14 11:03:03 -08:00
ai.robots.txt	443dd27527	Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots ... chore(robots.json): add AmazonBuyForMe and atlassian-bot	2025-11-05 23:50:59 +00:00
Glyn Normington	60b6a0829d	Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots ... chore(robots.json): add AmazonBuyForMe and atlassian-bot	2025-11-05 23:50:48 +00:00
Cory Dransfeldt	00bf2b0e13	chore(robots.json): add AmazonBuyForMe and atlassian-bot	2025-11-05 14:02:37 -08:00
ai.robots.txt	e87eb706e3	Merge pull request #188 from ai-robots-txt/cdransf/buddybot ... chore: adds BuddyBot	2025-11-04 05:16:28 +00:00
Glyn Normington	3d41350256	Merge pull request #188 from ai-robots-txt/cdransf/buddybot ... chore: adds BuddyBot	2025-11-04 05:16:15 +00:00
Cory Dransfeldt	808451055c	chore: adds BuddyBot	2025-11-03 13:39:07 -08:00
ai.robots.txt	5cad0ee389	Merge pull request #187 from ai-robots-txt/cdransf/add-Linguee-Bot ... chore: adds Linguee Bot	2025-10-24 18:05:01 +00:00
Glyn Normington	d55c9980cd	Merge pull request #187 from ai-robots-txt/cdransf/add-Linguee-Bot ... chore: adds Linguee Bot	2025-10-24 19:04:42 +01:00
Cory Dransfeldt	192b0a2eef	chore: adds Linguee Bot	2025-10-24 10:35:04 -07:00
dark-visitors	97e19445ce	Update from Dark Visitors	2025-10-24 00:53:30 +00:00
ai.robots.txt	0bc2361be8	Merge pull request #186 from ai-robots-txt/third-party ... Use third party in text, rather than first	2025-10-21 15:23:46 +00:00
Cory Dransfeldt	511d8c955d	Merge pull request #186 from ai-robots-txt/third-party ... Use third party in text, rather than first	2025-10-21 08:23:28 -07:00