91 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Cory Dransfeldt	55b4505e30	chore: add Timpibot v1.9	2024-07-29 12:38:22 -07:00
Cory Dransfeldt	d49e860b74	chore: add VelenPublicWebCrawler v1.8	2024-07-29 12:12:42 -07:00
Cory Dransfeldt	6e323554c6	chore: add Meta-ExternalAgent v1.7	2024-07-29 08:27:31 -07:00
Cory Dransfeldt	2972926532	chore: add OAI-SearchBot v1.6	2024-07-26 09:06:10 -07:00
Glyn Normington	c17cae6e9d	link to bot metrics table ... Make it easier to view the table.	2024-07-17 02:28:32 +01:00
Cory Dransfeldt	3692d66918	chore: update bots table v1.5	2024-07-16 12:07:01 -07:00
Cory Dransfeldt	af52578965	chore: drop google adbot; add GoogleOther bots	2024-07-16 12:05:34 -07:00
Cory Dransfeldt	570fd36ea2	chore: update bots table	2024-07-10 19:47:23 -07:00
Cory Dransfeldt	0ca6bce87e	chore: add ImagesiftBot v1.4	2024-07-09 17:41:32 -07:00
Cory Dransfeldt	0971af19b6	chore: peer39 unrelated to ai	2024-07-09 17:39:51 -07:00
Cory Dransfeldt	74fa789985	Merge pull request #18 from glyn/add-ref ... Add reference	2024-06-29 12:22:23 -07:00
Glyn Normington	4a8ce0b51d	Add reference ... This links to https://github.com/glyn/nginx_robot_access	2024-06-29 12:20:38 +01:00
Cory Dransfeldt	70e8985622	chore: remove unused image v1.3	2024-06-22 12:44:02 -07:00
Cory Dransfeldt	fd4ade555c	Merge pull request #17 from corbindavenport/main ... More bot information and improved README	2024-06-22 12:42:52 -07:00
Corbin Davenport	fe98e41546	Update Perplexity information	2024-06-21 22:17:45 -04:00
Corbin Davenport	20dc4e16ad	Update releases feed links	2024-06-21 22:02:54 -04:00
Corbin Davenport	149a72be0c	Add bot info for omgili, peer39, and youbot	2024-06-21 21:43:41 -04:00
Corbin Davenport	ec4610a118	Add information for Google, Meta, and img2dataset bots	2024-06-21 20:49:25 -04:00
Cory Dransfeldt	4163ca92a5	chore: clean up unnecessary ai.txt	2024-06-21 08:58:44 -07:00
Cory Dransfeldt	89de2d2d91	chore: resolve conflict v1.2	2024-06-20 08:12:24 -07:00
Cory Dransfeldt	a980ace544	chore: update readme	2024-06-20 08:11:38 -07:00
Cory Dransfeldt	a90ee5e9f0	chore: clean up bots and narrow scope	2024-06-20 08:09:21 -07:00
Cory Dransfeldt	4b8499d718	Merge pull request #16 from nisbet-hubbard/patch-1 ... Remove potential false positives	2024-06-20 07:57:16 -07:00
nisbet-hubbard	56c2285462	Update robots.txt	2024-06-20 11:31:08 +08:00
Cory Dransfeldt	429336d725	fix: resolve conflict v1.1	2024-06-15 09:26:14 -07:00
Cory Dransfeldt	3f65a93891	chore: keeps Applebot-Extended in favor of Applebot as the latter is simply for search	2024-06-15 09:25:17 -07:00
Cory Dransfeldt	ca730a81ca	Merge pull request #15 from christiansievers/main ... Update table-of-bot-metrics.md	2024-06-13 10:50:10 -07:00
Christian Sievers	ce0b99124b	Update table-of-bot-metrics.md ... add link to Apple docs and description for Applebot-Extended	2024-06-13 14:17:31 +02:00
Cory Dransfeldt	1a4868f1a2	chore: update readme	2024-06-12 19:05:15 -07:00
Cory Dransfeldt	b5d4d5bc74	chore: update readme v1	2024-06-12 19:01:23 -07:00
Cory Dransfeldt	548a6686ab	chore: add rss badge	2024-06-12 18:56:09 -07:00
Cory Dransfeldt	1ac25b9135	chore: propagate applebot-extended out	2024-06-12 18:53:26 -07:00
Cory Dransfeldt	c546dc048c	Merge pull request #14 from chriskirknielsen/patch-1 ... Block Applebot-Extended	2024-06-12 18:51:09 -07:00
Christopher Kirk-Nielsen	39363fc813	Block Applebot-Extended ... Per [Apple's docs](https://support.apple.com/en-us/119829#datausage) ([via Matthew Bogart](https://matthewbogart.com/@matt/112605297864483766))	2024-06-12 16:43:12 -04:00
Cory Dransfeldt	d464f260e0	Merge branch 'main' of github.com:ai-robots-txt/ai.robots.txt	2024-06-10 13:23:55 -07:00
Cory Dransfeldt	fd10fbda94	chore: add cloudflare waf expression	2024-06-10 13:23:47 -07:00
Cory Dransfeldt	7a5a21fdad	chore: update bots table	2024-05-29 09:21:31 -07:00
Cory Dransfeldt	2ca4154d4e	chore: update perplexity entry	2024-05-29 09:14:35 -07:00
Cory Dransfeldt	e152c110c5	Merge pull request #10 from kg583/main ... Fix Markdown in `table-of-bot-metrics.md`	2024-05-20 19:43:29 -07:00
KG	04688e56fb	Update table-of-bot-metrics.md ... Fix Markdown	2024-05-16 18:25:46 -05:00
Cory Dransfeldt	696d2acef8	chore: add cohere-ai	2024-05-06 10:39:07 -07:00
Cory Dransfeldt	6940db2412	chore: add anthropic bots to table	2024-05-06 10:36:16 -07:00
Cory Dransfeldt	ec8e10a56c	chore: add bytespider to table	2024-05-06 10:34:43 -07:00
Cory Dransfeldt	262fe71ad9	chore: add ccbot data to table	2024-05-06 10:32:05 -07:00
Cory Dransfeldt	dea035365f	chore: add Diffbot and scoopit	2024-05-05 14:50:04 -07:00
Cory Dransfeldt	29e0799332	chore: add img2dataset to table-of-bot-metrics.md	2024-04-22 09:30:00 -07:00
Cory Dransfeldt	5025939e5d	chore: add img2dataset to edge func	2024-04-22 09:28:39 -07:00
Cory Dransfeldt	8ddd2a0bb3	chore: add img2dataset to nginx.conf	2024-04-22 09:28:21 -07:00
Cory Dransfeldt	2b77b0b0ad	chore: add img2dataset to apache.conf	2024-04-22 09:27:41 -07:00
Cory Dransfeldt	118ec00126	chore: add img2dataset to robots.txt	2024-04-22 09:26:59 -07:00