500 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Glitz Smarter	c15065544a	Rewriting function in YaK ... Rewrote the function to make it clear the quote is according to the page on the Clearwater website.	2025-07-28 12:54:25 -04:00
Gregory	8b188d0612	Added YaK user-agent to list of robots ... Added Meltwater's AI to list of bots that use AI and scrape websites.	2025-07-26 23:50:11 -04:00
ai.robots.txt	47769c8429	Update from Dark Visitors v1.38	2025-07-17 01:04:20 +00:00
dark-visitors	9843c31303	Update from Dark Visitors	2025-07-16 01:03:43 +00:00
ai.robots.txt	567e94a6cf	Merge pull request #160 from sdavids/users/sdavids/feat/htaccess ... Simplify htaccess rewrite rule	2025-07-08 16:00:16 +00:00
Glyn Normington	30c4c037d9	Merge pull request #160 from sdavids/users/sdavids/feat/htaccess ... Simplify htaccess rewrite rule	2025-07-08 16:59:52 +01:00
Sebastian Davids	c7407e721d	Simplify htaccess rewrite rule ... https://httpd.apache.org/docs/2.4/rewrite/flags.html#flag_f fix #159 Signed-off-by: Sebastian Davids <sdavids@gmx.de>	2025-07-08 12:33:24 +02:00
dark-visitors	f6f504012f	Update from Dark Visitors	2025-07-08 01:01:25 +00:00
ai.robots.txt	63f1f56307	Merge pull request #158 from ai-robots-txt/thinkbot-summaly ... chore(robots.json): add SummalyBot + Thinkbot	2025-07-07 16:55:31 +00:00
Glyn Normington	91a5b3c995	Merge pull request #158 from ai-robots-txt/thinkbot-summaly ... chore(robots.json): add SummalyBot + Thinkbot	2025-07-07 17:55:22 +01:00
Cory Dransfeldt	c5dd4e98b6	chore(robots.json): add SummalyBot + Thinkbot	2025-07-07 09:49:26 -07:00
dark-visitors	70ff0c9fbd	Update from Dark Visitors	2025-07-03 01:00:49 +00:00
ai.robots.txt	20a242d390	Merge pull request #156 from nisbet-hubbard/patch-9 ... Update funtion and description for SemrushBot	2025-07-02 15:58:19 +00:00
Cory Dransfeldt	91ad302d68	Merge pull request #156 from nisbet-hubbard/patch-9 ... Update funtion and description for SemrushBot	2025-07-02 08:58:08 -07:00
dark-visitors	79bc22523f	Update from Dark Visitors	2025-07-02 01:01:05 +00:00
ai.robots.txt	2bfcee794d	Update from Dark Visitors	2025-06-29 01:07:47 +00:00
dark-visitors	7eb2099d3f	Update from Dark Visitors	2025-06-28 00:58:57 +00:00
nisbet-hubbard	accc48f327	Update funtion and description for SemrushBot	2025-06-21 08:48:58 +08:00
dark-visitors	4ed17b8e4a	Update from Dark Visitors	2025-06-17 01:00:21 +00:00
ai.robots.txt	5326c202b5	Merge pull request #154 from paulrudy/main ... re-add facebookexternalhit	2025-06-16 15:12:42 +00:00
Cory Dransfeldt	a31ae1e6d0	Merge pull request #154 from paulrudy/main ... re-add facebookexternalhit	2025-06-16 08:12:31 -07:00
paulrudy	7535893aec	re-add facebookexternalhit	2025-06-15 16:49:07 -07:00
ai.robots.txt	eb05f2f527	Merge pull request #153 from sergiospagnuolo/Poseidon ... v1.37 Update robots.json with new crawler	2025-06-14 14:04:03 +00:00
Cory Dransfeldt	26a46c409d	Merge pull request #153 from sergiospagnuolo/Poseidon ... Update robots.json with new crawler	2025-06-14 07:03:52 -07:00
dark-visitors	2b68568ac2	Update from Dark Visitors	2025-06-14 00:58:11 +00:00
Sérgio Spagnuolo	b05f2fee00	Update robots.json with new crawler ... Update with Poseidon Research Crawler as found in nytimes.com/robots.txt	2025-06-13 17:15:13 -03:00
ai.robots.txt	e53d81c66d	Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ... chore(robots.json): adds MyCentralAIScraperBot	2025-06-13 09:28:41 +00:00
Glyn Normington	20e327e74e	Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ... chore(robots.json): adds MyCentralAIScraperBot	2025-06-13 10:28:32 +01:00
Glyn Normington	8f17718e76	Fix typo	2025-06-13 10:28:12 +01:00
Cory Dransfeldt	d760f9216f	chore(robots.json): adds MyCentralAIScraperBot	2025-06-12 13:08:29 -07:00
ai.robots.txt	842e2256e8	Merge pull request #150 from ai-robots-txt/semrush-bots ... chore(robots.json): adds additional SemrushBot user agents	2025-06-12 07:12:00 +00:00
Glyn Normington	229ea20426	Merge pull request #150 from ai-robots-txt/semrush-bots ... chore(robots.json): adds additional SemrushBot user agents	2025-06-12 08:11:51 +01:00
Cory Dransfeldt	14d68f05ba	chore(robots.json): adds additional SemrushBot user agents	2025-06-11 13:50:53 -07:00
dark-visitors	cf598b6b71	Update from Dark Visitors v1.36	2025-06-10 01:00:37 +00:00
ai.robots.txt	3759a6bf14	chore(robots.json): adds EchoboxBot (#148)	2025-06-09 15:44:36 +00:00
Cory Dransfeldt	7867c3e26c	chore(robots.json): adds EchoboxBot (#148)	2025-06-09 16:44:25 +01:00
dark-visitors	e21f6ae1b6	Update from Dark Visitors v1.35	2025-06-06 00:59:25 +00:00
ai.robots.txt	ac7ed17e71	Merge pull request #145 from ai-robots-txt/aws-bedrockbot ... chore(robots.json): adds bedrockbot	2025-06-05 16:51:17 +00:00
Glyn Normington	81747e6772	Merge pull request #145 from ai-robots-txt/aws-bedrockbot ... chore(robots.json): adds bedrockbot	2025-06-05 17:51:03 +01:00
Cory Dransfeldt	528d77bf07	chore(robots.json): adds bedrockbot	2025-06-05 09:14:23 -07:00
dark-visitors	77393df5aa	Update from Dark Visitors	2025-06-05 00:59:28 +00:00
ai.robots.txt	75ea75a95b	Merge pull request #143 from ai-robots-txt/panscient ... v1.34 chore(robots.json): adds Panscient	2025-06-04 18:04:06 +00:00
Glyn Normington	2fca1ddcf1	Merge pull request #143 from ai-robots-txt/panscient ... chore(robots.json): adds Panscient	2025-06-04 19:03:53 +01:00
ai.robots.txt	9c28c63a0c	Merge pull request #142 from ai-robots-txt/quillbot ... chore(robots.json): adds Quillbot	2025-06-04 17:54:57 +00:00
Cory Dransfeldt	395c013eea	Merge pull request #142 from ai-robots-txt/quillbot ... chore(robots.json): adds Quillbot	2025-06-04 10:54:46 -07:00
Cory Dransfeldt	4568d69b0e	chore(robots.json): adds Panscient	2025-06-04 10:54:14 -07:00
Cory Dransfeldt	03831a7eb5	chore(robots.json): adds Quillbot	2025-06-04 10:46:58 -07:00
dark-visitors	2b5a59a303	Update from Dark Visitors	2025-06-04 01:00:07 +00:00
ai.robots.txt	3efabc603d	Merge pull request #141 from Ivan-Chupin/patch-1 ... v1.33 Add SBIntuitionsBot	2025-06-03 23:28:48 +00:00
Cory Dransfeldt	b35f9a31d7	Merge pull request #141 from Ivan-Chupin/patch-1 ... Add SBIntuitionsBot	2025-06-03 16:28:36 -07:00