49 commits

Author	SHA1	Message	Date
ai.robots.txt	e12ddc0f42	Merge pull request #29 from jbowdre/dev ... only build on changes to robots.json	2024-08-06 15:44:54 +00:00
ai.robots.txt	b54e274bbc	Removing previously generated files	2024-08-06 15:44:53 +00:00
ai.robots.txt	eb924b9856	Merge pull request #28 from jsheard/patch-2 ... Add Cloudflares first-party scraper blocking to FAQ	2024-08-04 21:54:17 +00:00
ai.robots.txt	1cfc071498	Removing previously generated files	2024-08-04 21:54:16 +00:00
ai.robots.txt	c2f177870f	Merge pull request #27 from jsheard/patch-1 ... Fix Imagesift user agent	2024-08-04 21:53:48 +00:00
ai.robots.txt	0072b8f5f0	Removing previously generated files	2024-08-04 21:53:47 +00:00
ai.robots.txt	c7b781034e	chore: restore FriendlyCrawler + ImageSift	2024-08-04 19:29:01 +00:00
ai.robots.txt	9a8fa66772	Removing previously generated files	2024-08-04 19:29:00 +00:00
ai.robots.txt	8de5bc8e01	Merge pull request #25 from mirium999/add_icc_crawler ... Add ICC-Crawler	2024-08-04 01:21:56 +00:00
ai.robots.txt	8c632e1ba4	Removing previously generated files	2024-08-04 01:21:55 +00:00
ai.robots.txt	ffbad453f3	Merge pull request #24 from nisbet-hubbard/patch-5 ... Add last line of defence to FAQ	2024-08-03 14:27:47 +00:00
ai.robots.txt	b1907d86be	Removing previously generated files	2024-08-03 14:27:46 +00:00
ai.robots.txt	d8de1ebdd5	chore: contribution note	2024-08-02 16:32:00 +00:00
ai.robots.txt	9d8d3de8ed	Removing previously generated files	2024-08-02 16:31:59 +00:00
ai.robots.txt	b144225ece	chore: drop in additional data	2024-08-01 22:33:23 +00:00
ai.robots.txt	06b950bce9	Removing previously generated files	2024-08-01 22:33:23 +00:00
ai.robots.txt	f18f0d99b9	chore: remove test data	2024-08-01 22:29:02 +00:00
ai.robots.txt	747cc834c4	Removing previously generated files	2024-08-01 22:29:01 +00:00
Cory Dransfeldt	17a84f2c2d	chore: update robots table	2024-08-01 15:06:49 -07:00
Cory Dransfeldt	fa7b64ae4b	chore: add Scrapy	2024-07-30 10:28:46 -07:00
Cory Dransfeldt	55b4505e30	chore: add Timpibot	2024-07-29 12:38:22 -07:00
Cory Dransfeldt	d49e860b74	chore: add VelenPublicWebCrawler	2024-07-29 12:12:42 -07:00
Cory Dransfeldt	6e323554c6	chore: add Meta-ExternalAgent	2024-07-29 08:27:31 -07:00
Cory Dransfeldt	2972926532	chore: add OAI-SearchBot	2024-07-26 09:06:10 -07:00
Cory Dransfeldt	3692d66918	chore: update bots table	2024-07-16 12:07:01 -07:00
Cory Dransfeldt	570fd36ea2	chore: update bots table	2024-07-10 19:47:23 -07:00
Corbin Davenport	fe98e41546	Update Perplexity information	2024-06-21 22:17:45 -04:00
Corbin Davenport	149a72be0c	Add bot info for omgili, peer39, and youbot	2024-06-21 21:43:41 -04:00
Corbin Davenport	ec4610a118	Add information for Google, Meta, and img2dataset bots	2024-06-21 20:49:25 -04:00
Cory Dransfeldt	a90ee5e9f0	chore: clean up bots and narrow scope	2024-06-20 08:09:21 -07:00
Cory Dransfeldt	429336d725	fix: resolve conflict	2024-06-15 09:26:14 -07:00
Cory Dransfeldt	3f65a93891	chore: keeps Applebot-Extended in favor of Applebot as the latter is simply for search	2024-06-15 09:25:17 -07:00
Christian Sievers	ce0b99124b	Update table-of-bot-metrics.md ... add link to Apple docs and description for Applebot-Extended	2024-06-13 14:17:31 +02:00
Cory Dransfeldt	1ac25b9135	chore: propagate applebot-extended out	2024-06-12 18:53:26 -07:00
Cory Dransfeldt	7a5a21fdad	chore: update bots table	2024-05-29 09:21:31 -07:00
Cory Dransfeldt	2ca4154d4e	chore: update perplexity entry	2024-05-29 09:14:35 -07:00
KG	04688e56fb	Update table-of-bot-metrics.md ... Fix Markdown	2024-05-16 18:25:46 -05:00
Cory Dransfeldt	696d2acef8	chore: add cohere-ai	2024-05-06 10:39:07 -07:00
Cory Dransfeldt	6940db2412	chore: add anthropic bots to table	2024-05-06 10:36:16 -07:00
Cory Dransfeldt	ec8e10a56c	chore: add bytespider to table	2024-05-06 10:34:43 -07:00
Cory Dransfeldt	262fe71ad9	chore: add ccbot data to table	2024-05-06 10:32:05 -07:00
Cory Dransfeldt	dea035365f	chore: add Diffbot and scoopit	2024-05-05 14:50:04 -07:00
Cory Dransfeldt	29e0799332	chore: add img2dataset to table-of-bot-metrics.md	2024-04-22 09:30:00 -07:00
Cory Dransfeldt	9be338094b	chore: populate gptbot	2024-04-10 10:53:16 -07:00
Cory Dransfeldt	cc5966af5f	chore: populate ChatGPT-User	2024-04-10 10:39:34 -07:00
Cory Dransfeldt	0a8952b34f	chore: add applebot	2024-04-10 10:18:51 -07:00
Cory Dransfeldt	b402ec681c	chore: add anthropic + amazonbot	2024-04-10 10:15:22 -07:00
Matthew Linares	aa0edf05b3	Add Google Gemini response on AbsBot-Google for bot metrics table. ... Reference: https://g.co/gemini/share/5bef4db5f87d	2024-04-10 15:40:45 +01:00
Matthew Linares	048045ac6d	Adds table of bot details and metrics.	2024-04-08 20:29:45 +01:00