603 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
ai.robots.txt	8b8bf9da5d	Update from Dark Visitors	2025-12-06 00:58:16 +00:00
dark-visitors	f1c752ef12	Update from Dark Visitors	2025-12-05 01:00:44 +00:00
Adam Newbold	51afa7113a	Update ai_robots_update.yml with rebase command to fix scheduled run	2025-12-03 20:15:10 -05:00
ai.robots.txt	7598d77e4a	Update from Dark Visitors	2025-12-04 01:10:02 +00:00
dark-visitors	45b071b29f	Update from Dark Visitors	2025-12-04 01:00:27 +00:00
dark-visitors	f61b3496f7	Update from Dark Visitors	2025-12-03 01:00:32 +00:00
dark-visitors	8363d4fdd4	Update from Dark Visitors	2025-12-02 01:25:24 +00:00
Adam Newbold	2ccd443581	Update ai_robots_update.yml with workflow_dispatch ... Adding workflow_dispatch to enable manual triggers of this schedule job (for testing)	2025-12-01 20:24:41 -05:00
Adam Newbold	6d75f3c1c9	Update robots.py to address error on line 57 ... Attempting to work around an error that prevents parsing the Dark Visitors site	2025-12-01 20:18:29 -05:00
Glyn Normington	56010ef913	Merge pull request #205 from fiskhandlarn/fix/editorconfig ... Fix/editorconfig	2025-11-29 10:02:08 +00:00
ai.robots.txt	3fadc88a23	Merge pull request #206 from newbold/main ... Adding LAIONDownloader	2025-11-29 10:00:52 +00:00
Glyn Normington	47c077a8ef	Merge pull request #206 from newbold/main ... Adding LAIONDownloader	2025-11-29 10:00:42 +00:00
Adam Newbold	f5d7ccb243	Fixed invalid JSON	2025-11-28 14:10:38 -05:00
Adam Newbold	30d719a09a	Adding LAIONDownloader	2025-11-28 14:08:18 -05:00
fiskhandlarn	05bbdebeaa	feat: disallow final newline for files generated by python ... if any of these files have ending newlines the tests will fail	2025-11-28 10:47:22 +01:00
fiskhandlarn	c6ce9329a1	fix: ensure whitespace as defined in .editorconfig	2025-11-28 10:46:22 +01:00
ai.robots.txt	10d5ae2870	Merge pull request #200 from glyn/deepseek ... Clarify that DeepSeekBot does not respect robots.txt	2025-11-27 14:00:50 +00:00
Glyn Normington	4467002298	Merge pull request #200 from glyn/deepseek ... Clarify that DeepSeekBot does not respect robots.txt	2025-11-27 14:00:40 +00:00
Glyn Normington	4a159a818f	Merge pull request #199 from glyn/editorconfig ... Standardise editor options	2025-11-27 13:54:03 +00:00
Glyn Normington	3d6b33a71a	Merge pull request #202 from glyn/formatting ... Tidy README	2025-11-27 12:42:31 +00:00
Glyn Normington	c26c8c0911	Tidy README	2025-11-27 12:41:37 +00:00
Glyn Normington	91959fe791	Merge pull request #201 from Anshita-18H/add-requirements-file ... Add requirements.txt with project dependencies	2025-11-27 12:38:28 +00:00
Glyn Normington	b75163e796	Ensure Python3 is used	2025-11-27 12:38:12 +00:00
Glyn Normington	f46754d280	Order deps	2025-11-27 12:37:35 +00:00
Glyn Normington	c3f2fe758e	Whitespace	2025-11-27 12:37:21 +00:00
Glyn Normington	7521c3af50	Fix link	2025-11-27 12:37:07 +00:00
Anshita-18H	4302fd1aca	Improve contributing section and fix formatting in README	2025-11-27 17:56:47 +05:30
Anshita-18H	9ca5033927	Deduplicate requirements.txt and add installation instructions to README	2025-11-27 17:44:47 +05:30
Anshita-18H	9cc3dbc05f	Add requirements.txt with project dependencies	2025-11-27 17:06:59 +05:30
Glyn Normington	c322d7852d	Merge pull request #130 from maxheadroom/main ... Create traefik-manual-setup.md	2025-11-26 12:08:11 +00:00
Glyn Normington	35588b2ddb	Typo.	2025-11-26 12:07:39 +00:00
Glyn Normington	ab24e41106	Clarify that DeepSeekBot does not respect robots.txt ... Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/198	2025-11-26 12:02:42 +00:00
Glyn Normington	b681c0c0d8	Standardise editor options ... This was motivated by @fiskhandlarn's comment: https://github.com/ai-robots-txt/ai.robots.txt/pull/195#issuecomment-3576331322	2025-11-26 11:54:10 +00:00
Glyn Normington	4e7e28335f	Merge pull request #195 from fiskhandlarn/patch-2 ... feat: allow robots access to `/robots.txt` in nginx	2025-11-26 11:50:15 +00:00
fiskhandlarn	a6cf6b204b	test: update test nginx conf	2025-11-25 16:47:04 +01:00
fiskhandlarn	2679fcad34	feat: update nginx generator	2025-11-25 16:46:31 +01:00
fiskhandlarn	ef8eda4fe6	chore: normalize quote style	2025-11-25 16:39:35 +01:00
Adam Newbold	a29102f0fc	Merge pull request #196 from glyn/releasing ... v1.44 Document how to ship a new release	2025-11-25 10:34:25 -05:00
fiskhandlarn	0b3266b35f	feat: allow robots access to /robots.txt in nginx	2025-11-25 16:22:14 +01:00
Glyn Normington	6225e3e98e	Document how to ship a new release	2025-11-23 04:01:43 +00:00
ai.robots.txt	be4d74412c	Merge pull request #194 from glyn/193-kendra ... delete extraneous hyphen	2025-11-21 16:32:11 +00:00
Cory Dransfeldt	729be4693a	Merge pull request #194 from glyn/193-kendra ... delete extraneous hyphen	2025-11-21 08:32:01 -08:00
Glyn Normington	efb4d260da	delete extraneous hyphen ... Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/193	2025-11-20 10:23:48 +00:00
ai.robots.txt	e2726ac160	Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo ... v1.43 chore: adds NotebookLM and KlaviyoAIBot agents	2025-11-14 21:01:25 +00:00
Glyn Normington	663d030f96	Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo ... chore: adds NotebookLM and KlaviyoAIBot agents	2025-11-14 21:01:13 +00:00
Cory Dransfeldt	28b45ea08d	chore: adds NotebookLM and KlaviyoAIBot agents	2025-11-14 11:03:03 -08:00
ai.robots.txt	443dd27527	Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots ... chore(robots.json): add AmazonBuyForMe and atlassian-bot	2025-11-05 23:50:59 +00:00
Glyn Normington	60b6a0829d	Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots ... chore(robots.json): add AmazonBuyForMe and atlassian-bot	2025-11-05 23:50:48 +00:00
Cory Dransfeldt	00bf2b0e13	chore(robots.json): add AmazonBuyForMe and atlassian-bot	2025-11-05 14:02:37 -08:00
ai.robots.txt	e87eb706e3	Merge pull request #188 from ai-robots-txt/cdransf/buddybot ... chore: adds BuddyBot	2025-11-04 05:16:28 +00:00