487 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Cory Dransfeldt	91ad302d68	Merge pull request #156 from nisbet-hubbard/patch-9 ... Update funtion and description for SemrushBot	2025-07-02 08:58:08 -07:00
dark-visitors	79bc22523f	Update from Dark Visitors	2025-07-02 01:01:05 +00:00
ai.robots.txt	2bfcee794d	Update from Dark Visitors	2025-06-29 01:07:47 +00:00
dark-visitors	7eb2099d3f	Update from Dark Visitors	2025-06-28 00:58:57 +00:00
nisbet-hubbard	accc48f327	Update funtion and description for SemrushBot	2025-06-21 08:48:58 +08:00
dark-visitors	4ed17b8e4a	Update from Dark Visitors	2025-06-17 01:00:21 +00:00
ai.robots.txt	5326c202b5	Merge pull request #154 from paulrudy/main ... re-add facebookexternalhit	2025-06-16 15:12:42 +00:00
Cory Dransfeldt	a31ae1e6d0	Merge pull request #154 from paulrudy/main ... re-add facebookexternalhit	2025-06-16 08:12:31 -07:00
paulrudy	7535893aec	re-add facebookexternalhit	2025-06-15 16:49:07 -07:00
ai.robots.txt	eb05f2f527	Merge pull request #153 from sergiospagnuolo/Poseidon ... v1.37 Update robots.json with new crawler	2025-06-14 14:04:03 +00:00
Cory Dransfeldt	26a46c409d	Merge pull request #153 from sergiospagnuolo/Poseidon ... Update robots.json with new crawler	2025-06-14 07:03:52 -07:00
dark-visitors	2b68568ac2	Update from Dark Visitors	2025-06-14 00:58:11 +00:00
Sérgio Spagnuolo	b05f2fee00	Update robots.json with new crawler ... Update with Poseidon Research Crawler as found in nytimes.com/robots.txt	2025-06-13 17:15:13 -03:00
ai.robots.txt	e53d81c66d	Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ... chore(robots.json): adds MyCentralAIScraperBot	2025-06-13 09:28:41 +00:00
Glyn Normington	20e327e74e	Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ... chore(robots.json): adds MyCentralAIScraperBot	2025-06-13 10:28:32 +01:00
Glyn Normington	8f17718e76	Fix typo	2025-06-13 10:28:12 +01:00
Cory Dransfeldt	d760f9216f	chore(robots.json): adds MyCentralAIScraperBot	2025-06-12 13:08:29 -07:00
ai.robots.txt	842e2256e8	Merge pull request #150 from ai-robots-txt/semrush-bots ... chore(robots.json): adds additional SemrushBot user agents	2025-06-12 07:12:00 +00:00
Glyn Normington	229ea20426	Merge pull request #150 from ai-robots-txt/semrush-bots ... chore(robots.json): adds additional SemrushBot user agents	2025-06-12 08:11:51 +01:00
Cory Dransfeldt	14d68f05ba	chore(robots.json): adds additional SemrushBot user agents	2025-06-11 13:50:53 -07:00
dark-visitors	cf598b6b71	Update from Dark Visitors v1.36	2025-06-10 01:00:37 +00:00
ai.robots.txt	3759a6bf14	chore(robots.json): adds EchoboxBot (#148)	2025-06-09 15:44:36 +00:00
Cory Dransfeldt	7867c3e26c	chore(robots.json): adds EchoboxBot (#148)	2025-06-09 16:44:25 +01:00
dark-visitors	e21f6ae1b6	Update from Dark Visitors v1.35	2025-06-06 00:59:25 +00:00
ai.robots.txt	ac7ed17e71	Merge pull request #145 from ai-robots-txt/aws-bedrockbot ... chore(robots.json): adds bedrockbot	2025-06-05 16:51:17 +00:00
Glyn Normington	81747e6772	Merge pull request #145 from ai-robots-txt/aws-bedrockbot ... chore(robots.json): adds bedrockbot	2025-06-05 17:51:03 +01:00
Cory Dransfeldt	528d77bf07	chore(robots.json): adds bedrockbot	2025-06-05 09:14:23 -07:00
dark-visitors	77393df5aa	Update from Dark Visitors	2025-06-05 00:59:28 +00:00
ai.robots.txt	75ea75a95b	Merge pull request #143 from ai-robots-txt/panscient ... v1.34 chore(robots.json): adds Panscient	2025-06-04 18:04:06 +00:00
Glyn Normington	2fca1ddcf1	Merge pull request #143 from ai-robots-txt/panscient ... chore(robots.json): adds Panscient	2025-06-04 19:03:53 +01:00
ai.robots.txt	9c28c63a0c	Merge pull request #142 from ai-robots-txt/quillbot ... chore(robots.json): adds Quillbot	2025-06-04 17:54:57 +00:00
Cory Dransfeldt	395c013eea	Merge pull request #142 from ai-robots-txt/quillbot ... chore(robots.json): adds Quillbot	2025-06-04 10:54:46 -07:00
Cory Dransfeldt	4568d69b0e	chore(robots.json): adds Panscient	2025-06-04 10:54:14 -07:00
Cory Dransfeldt	03831a7eb5	chore(robots.json): adds Quillbot	2025-06-04 10:46:58 -07:00
dark-visitors	2b5a59a303	Update from Dark Visitors	2025-06-04 01:00:07 +00:00
ai.robots.txt	3efabc603d	Merge pull request #141 from Ivan-Chupin/patch-1 ... v1.33 Add SBIntuitionsBot	2025-06-03 23:28:48 +00:00
Cory Dransfeldt	b35f9a31d7	Merge pull request #141 from Ivan-Chupin/patch-1 ... Add SBIntuitionsBot	2025-06-03 16:28:36 -07:00
Ivan Chupin	8f75f4a2f5	Add SBIntuitionsBot	2025-06-04 03:48:42 +05:00
ai.robots.txt	080946c360	Merge pull request #140 from ai-robots-txt/yandex-bots ... v1.32 chore(robots.json): adds YandexAdditional crawlers	2025-06-03 19:51:25 +00:00
Glyn Normington	7eec033cad	Merge pull request #140 from ai-robots-txt/yandex-bots ... chore(robots.json): adds YandexAdditional crawlers	2025-06-03 20:51:14 +01:00
Cory Dransfeldt	3187fd8a32	chore(robots.json): adds YandexAdditional crawlers	2025-06-03 12:41:57 -07:00
ai.robots.txt	d239e7e5ad	Merge pull request #139 from ai-robots-txt/workflow-fix ... chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard	2025-06-03 01:52:35 +00:00
Glyn Normington	9dbf34010a	Merge pull request #139 from ai-robots-txt/workflow-fix ... chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard	2025-06-03 02:52:23 +01:00
dark-visitors	87016d1504	Update from Dark Visitors	2025-06-03 01:00:29 +00:00
Cory Dransfeldt	899ce01c55	chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard	2025-06-02 14:56:09 -07:00
Glyn Normington	4af776f0a0	Merge pull request #136 from ai-robots-txt/imgproxy-revert ... chore(robots.json): revert "adds imgproxy crawler"	2025-06-02 20:21:10 +01:00
Cory Dransfeldt	1dd66b6969	Revert "chore(robots.json): adds imgproxy crawler" ... This reverts commit b65f45e408.	2025-06-02 11:53:06 -07:00
Cory Dransfeldt	814df6b9a0	Merge pull request #134 from not-not-the-imp/patch-1 ... Add AndiBot and PhindBot	2025-05-31 16:03:16 -07:00
Cory Dransfeldt	268922f8f2	Update robots.json	2025-05-31 16:02:05 -07:00
Cory Dransfeldt	4259b25ccc	Update robots.json	2025-05-31 16:01:09 -07:00