516 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Glyn Normington	1677278c5a	Merge pull request #168 from ai-robots-txt/google-firebase-shapbot ... chore: add Google-Firebase and ShapBot	2025-08-27 11:13:04 +01:00
Cory Dransfeldt	1a8edfa84a	chore: add Google-Firebase and ShapBot	2025-08-26 19:39:42 -07:00
dark-visitors	cf073d49f2	Update from Dark Visitors	2025-08-15 01:01:42 +00:00
ai.robots.txt	6d3f3e1712	Merge pull request #167 from ai-robots-txt/OpenAI-bot ... v1.39 chore: adds OpenAI but user agent	2025-08-14 01:45:05 +00:00
Glyn Normington	784b8440a5	Merge pull request #167 from ai-robots-txt/OpenAI-bot ... chore: adds OpenAI but user agent	2025-08-14 02:44:53 +01:00
Cory Dransfeldt	0e687a5b58	chore: adds OpenAI but user agent	2025-08-13 09:01:55 -07:00
ai.robots.txt	ff9fc26404	Update from Dark Visitors	2025-08-01 01:12:53 +00:00
dark-visitors	146a229662	Update from Dark Visitors	2025-07-31 01:05:21 +00:00
ai.robots.txt	64f9d6ce9c	Update from Dark Visitors	2025-07-30 01:05:21 +00:00
dark-visitors	085dd1071e	Update from Dark Visitors	2025-07-29 01:11:26 +00:00
ai.robots.txt	9565c11d4c	Merge pull request #166 from ai-robots-txt/summaly-bot ... fix: remove summaly	2025-07-28 19:19:15 +00:00
Glyn Normington	8869442615	Merge pull request #166 from ai-robots-txt/summaly-bot ... fix: remove summaly	2025-07-28 20:19:06 +01:00
ai.robots.txt	27420d6fed	Merge pull request #164 from GlitzSmarter/main ... Adding YaK	2025-07-28 18:29:57 +00:00
Glyn Normington	44e58a5ece	Merge pull request #164 from GlitzSmarter/main ... Adding YaK	2025-07-28 19:29:48 +01:00
Glyn Normington	12c5368e04	Fix syntax	2025-07-28 19:29:19 +01:00
Glitz Smarter	c15065544a	Rewriting function in YaK ... Rewrote the function to make it clear the quote is according to the page on the Clearwater website.	2025-07-28 12:54:25 -04:00
Cory Dransfeldt	9171625db6	fix: remove summaly	2025-07-28 08:56:21 -07:00
Gregory	8b188d0612	Added YaK user-agent to list of robots ... Added Meltwater's AI to list of bots that use AI and scrape websites.	2025-07-26 23:50:11 -04:00
ai.robots.txt	47769c8429	Update from Dark Visitors v1.38	2025-07-17 01:04:20 +00:00
dark-visitors	9843c31303	Update from Dark Visitors	2025-07-16 01:03:43 +00:00
ai.robots.txt	567e94a6cf	Merge pull request #160 from sdavids/users/sdavids/feat/htaccess ... Simplify htaccess rewrite rule	2025-07-08 16:00:16 +00:00
Glyn Normington	30c4c037d9	Merge pull request #160 from sdavids/users/sdavids/feat/htaccess ... Simplify htaccess rewrite rule	2025-07-08 16:59:52 +01:00
Sebastian Davids	c7407e721d	Simplify htaccess rewrite rule ... https://httpd.apache.org/docs/2.4/rewrite/flags.html#flag_f fix #159 Signed-off-by: Sebastian Davids <sdavids@gmx.de>	2025-07-08 12:33:24 +02:00
dark-visitors	f6f504012f	Update from Dark Visitors	2025-07-08 01:01:25 +00:00
ai.robots.txt	63f1f56307	Merge pull request #158 from ai-robots-txt/thinkbot-summaly ... chore(robots.json): add SummalyBot + Thinkbot	2025-07-07 16:55:31 +00:00
Glyn Normington	91a5b3c995	Merge pull request #158 from ai-robots-txt/thinkbot-summaly ... chore(robots.json): add SummalyBot + Thinkbot	2025-07-07 17:55:22 +01:00
Cory Dransfeldt	c5dd4e98b6	chore(robots.json): add SummalyBot + Thinkbot	2025-07-07 09:49:26 -07:00
dark-visitors	70ff0c9fbd	Update from Dark Visitors	2025-07-03 01:00:49 +00:00
ai.robots.txt	20a242d390	Merge pull request #156 from nisbet-hubbard/patch-9 ... Update funtion and description for SemrushBot	2025-07-02 15:58:19 +00:00
Cory Dransfeldt	91ad302d68	Merge pull request #156 from nisbet-hubbard/patch-9 ... Update funtion and description for SemrushBot	2025-07-02 08:58:08 -07:00
dark-visitors	79bc22523f	Update from Dark Visitors	2025-07-02 01:01:05 +00:00
ai.robots.txt	2bfcee794d	Update from Dark Visitors	2025-06-29 01:07:47 +00:00
dark-visitors	7eb2099d3f	Update from Dark Visitors	2025-06-28 00:58:57 +00:00
nisbet-hubbard	accc48f327	Update funtion and description for SemrushBot	2025-06-21 08:48:58 +08:00
dark-visitors	4ed17b8e4a	Update from Dark Visitors	2025-06-17 01:00:21 +00:00
ai.robots.txt	5326c202b5	Merge pull request #154 from paulrudy/main ... re-add facebookexternalhit	2025-06-16 15:12:42 +00:00
Cory Dransfeldt	a31ae1e6d0	Merge pull request #154 from paulrudy/main ... re-add facebookexternalhit	2025-06-16 08:12:31 -07:00
paulrudy	7535893aec	re-add facebookexternalhit	2025-06-15 16:49:07 -07:00
ai.robots.txt	eb05f2f527	Merge pull request #153 from sergiospagnuolo/Poseidon ... v1.37 Update robots.json with new crawler	2025-06-14 14:04:03 +00:00
Cory Dransfeldt	26a46c409d	Merge pull request #153 from sergiospagnuolo/Poseidon ... Update robots.json with new crawler	2025-06-14 07:03:52 -07:00
dark-visitors	2b68568ac2	Update from Dark Visitors	2025-06-14 00:58:11 +00:00
Sérgio Spagnuolo	b05f2fee00	Update robots.json with new crawler ... Update with Poseidon Research Crawler as found in nytimes.com/robots.txt	2025-06-13 17:15:13 -03:00
ai.robots.txt	e53d81c66d	Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ... chore(robots.json): adds MyCentralAIScraperBot	2025-06-13 09:28:41 +00:00
Glyn Normington	20e327e74e	Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ... chore(robots.json): adds MyCentralAIScraperBot	2025-06-13 10:28:32 +01:00
Glyn Normington	8f17718e76	Fix typo	2025-06-13 10:28:12 +01:00
Cory Dransfeldt	d760f9216f	chore(robots.json): adds MyCentralAIScraperBot	2025-06-12 13:08:29 -07:00
ai.robots.txt	842e2256e8	Merge pull request #150 from ai-robots-txt/semrush-bots ... chore(robots.json): adds additional SemrushBot user agents	2025-06-12 07:12:00 +00:00
Glyn Normington	229ea20426	Merge pull request #150 from ai-robots-txt/semrush-bots ... chore(robots.json): adds additional SemrushBot user agents	2025-06-12 08:11:51 +01:00
Cory Dransfeldt	14d68f05ba	chore(robots.json): adds additional SemrushBot user agents	2025-06-11 13:50:53 -07:00
dark-visitors	cf598b6b71	Update from Dark Visitors v1.36	2025-06-10 01:00:37 +00:00