118 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
ai.robots.txt	9a8fa66772	Removing previously generated files	2024-08-04 19:29:00 +00:00
Cory Dransfeldt	1ca936ce11	chore: restore FriendlyCrawler + ImageSift	2024-08-04 12:28:48 -07:00
ai.robots.txt	8de5bc8e01	Merge pull request #25 from mirium999/add_icc_crawler ... Add ICC-Crawler	2024-08-04 01:21:56 +00:00
ai.robots.txt	8c632e1ba4	Removing previously generated files	2024-08-04 01:21:55 +00:00
Adam Newbold	8d4d52cdab	Merge pull request #25 from mirium999/add_icc_crawler ... Add ICC-Crawler	2024-08-03 21:21:45 -04:00
Mirium999	5826c18909	Add ICC-Crawler	2024-08-04 10:11:25 +09:00
ai.robots.txt	ffbad453f3	Merge pull request #24 from nisbet-hubbard/patch-5 ... Add last line of defence to FAQ	2024-08-03 14:27:47 +00:00
ai.robots.txt	b1907d86be	Removing previously generated files	2024-08-03 14:27:46 +00:00
Cory Dransfeldt	55c585e9e3	Merge pull request #24 from nisbet-hubbard/patch-5 ... Add last line of defence to FAQ	2024-08-03 07:27:37 -07:00
nisbet-hubbard	2b56c72bac	Update FAQ.md	2024-08-03 14:27:25 +08:00
nisbet-hubbard	b24e5cb3bb	Update FAQ.md	2024-08-03 14:12:50 +08:00
nisbet-hubbard	74b1502839	Update FAQ.md	2024-08-03 14:04:58 +08:00
ai.robots.txt	d8de1ebdd5	chore: contribution note	2024-08-02 16:32:00 +00:00
ai.robots.txt	9d8d3de8ed	Removing previously generated files	2024-08-02 16:31:59 +00:00
Cory Dransfeldt	349c35eed6	chore: contribution note	2024-08-02 09:31:48 -07:00
ai.robots.txt	b144225ece	chore: drop in additional data	2024-08-01 22:33:23 +00:00
ai.robots.txt	06b950bce9	Removing previously generated files	2024-08-01 22:33:23 +00:00
Cory Dransfeldt	b20dfec1e4	chore: drop in additional data	2024-08-01 15:33:07 -07:00
ai.robots.txt	f18f0d99b9	chore: remove test data	2024-08-01 22:29:02 +00:00
ai.robots.txt	747cc834c4	Removing previously generated files	2024-08-01 22:29:01 +00:00
Cory Dransfeldt	efabf3e721	chore: remove test data	2024-08-01 15:25:55 -07:00
Adam Newbold	1fdc79dacb	Adding GitHub Action	2024-08-01 18:17:19 -04:00
Cory Dransfeldt	17a84f2c2d	chore: update robots table	2024-08-01 15:06:49 -07:00
Cory Dransfeldt	6c596a50ea	chore: move FAQ into repo v1.11	2024-08-01 07:53:43 -07:00
Cory Dransfeldt	6a8e7a8eb0	Merge pull request #22 from nisbet-hubbard/patch-4 ... Add `PetalBot` (and `facebookexternalhit`?)	2024-08-01 07:49:30 -07:00
nisbet-hubbard	df89722038	Add PetalBot (and facebookexternalhit?)	2024-07-31 18:27:29 +08:00
Cory Dransfeldt	fa7b64ae4b	chore: add Scrapy v1.10	2024-07-30 10:28:46 -07:00
Cory Dransfeldt	55b4505e30	chore: add Timpibot v1.9	2024-07-29 12:38:22 -07:00
Cory Dransfeldt	d49e860b74	chore: add VelenPublicWebCrawler v1.8	2024-07-29 12:12:42 -07:00
Cory Dransfeldt	6e323554c6	chore: add Meta-ExternalAgent v1.7	2024-07-29 08:27:31 -07:00
Cory Dransfeldt	2972926532	chore: add OAI-SearchBot v1.6	2024-07-26 09:06:10 -07:00
Glyn Normington	c17cae6e9d	link to bot metrics table ... Make it easier to view the table.	2024-07-17 02:28:32 +01:00
Cory Dransfeldt	3692d66918	chore: update bots table v1.5	2024-07-16 12:07:01 -07:00
Cory Dransfeldt	af52578965	chore: drop google adbot; add GoogleOther bots	2024-07-16 12:05:34 -07:00
Cory Dransfeldt	570fd36ea2	chore: update bots table	2024-07-10 19:47:23 -07:00
Cory Dransfeldt	0ca6bce87e	chore: add ImagesiftBot v1.4	2024-07-09 17:41:32 -07:00
Cory Dransfeldt	0971af19b6	chore: peer39 unrelated to ai	2024-07-09 17:39:51 -07:00
Cory Dransfeldt	74fa789985	Merge pull request #18 from glyn/add-ref ... Add reference	2024-06-29 12:22:23 -07:00
Glyn Normington	4a8ce0b51d	Add reference ... This links to https://github.com/glyn/nginx_robot_access	2024-06-29 12:20:38 +01:00
Cory Dransfeldt	70e8985622	chore: remove unused image v1.3	2024-06-22 12:44:02 -07:00
Cory Dransfeldt	fd4ade555c	Merge pull request #17 from corbindavenport/main ... More bot information and improved README	2024-06-22 12:42:52 -07:00
Corbin Davenport	fe98e41546	Update Perplexity information	2024-06-21 22:17:45 -04:00
Corbin Davenport	20dc4e16ad	Update releases feed links	2024-06-21 22:02:54 -04:00
Corbin Davenport	149a72be0c	Add bot info for omgili, peer39, and youbot	2024-06-21 21:43:41 -04:00
Corbin Davenport	ec4610a118	Add information for Google, Meta, and img2dataset bots	2024-06-21 20:49:25 -04:00
Cory Dransfeldt	4163ca92a5	chore: clean up unnecessary ai.txt	2024-06-21 08:58:44 -07:00
Cory Dransfeldt	89de2d2d91	chore: resolve conflict v1.2	2024-06-20 08:12:24 -07:00
Cory Dransfeldt	a980ace544	chore: update readme	2024-06-20 08:11:38 -07:00
Cory Dransfeldt	a90ee5e9f0	chore: clean up bots and narrow scope	2024-06-20 08:09:21 -07:00
Cory Dransfeldt	4b8499d718	Merge pull request #16 from nisbet-hubbard/patch-1 ... Remove potential false positives	2024-06-20 07:57:16 -07:00