14 commits

Author	SHA1	Message	Date
Cory Dransfeldt	89de2d2d91	chore: resolve conflict	2024-06-20 08:12:24 -07:00
Cory Dransfeldt	a90ee5e9f0	chore: clean up bots and narrow scope	2024-06-20 08:09:21 -07:00
nisbet-hubbard	56c2285462	Update robots.txt	2024-06-20 11:31:08 +08:00
Cory Dransfeldt	3f65a93891	chore: keeps Applebot-Extended in favor of Applebot as the latter is simply for search	2024-06-15 09:25:17 -07:00
Christopher Kirk-Nielsen	39363fc813	Block Applebot-Extended ... Per [Apple's docs](https://support.apple.com/en-us/119829#datausage) ([via Matthew Bogart](https://matthewbogart.com/@matt/112605297864483766))	2024-06-12 16:43:12 -04:00
Cory Dransfeldt	dea035365f	chore: add Diffbot and scoopit	2024-05-05 14:50:04 -07:00
Cory Dransfeldt	118ec00126	chore: add img2dataset to robots.txt	2024-04-22 09:26:59 -07:00
Cory Dransfeldt	d6d40989f4	chore: add FriendlyCrawler to robots.txt	2024-04-08 12:40:59 -07:00
Cory Dransfeldt	47fc45f2f9	chore: add PiplBot	2024-04-06 20:25:28 -07:00
Cory Dransfeldt	46c8c9adb3	chore: add Meltwater	2024-04-03 08:56:30 -07:00
Cory Dransfeldt	c8a6d7f02d	chore: add Seekr	2024-04-03 08:56:12 -07:00
--Explosion--	3e57b5ab5d	Add GoogleOther ... Used by Google to crawl for internal research and development. It’s unknown what exactly this entails, but is a generic user agent that is used when no other appropriate user agent is available. Documentation available from Google: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers	2024-03-28 10:00:58 -05:00
Cory Dransfeldt	297071a664	Update robots.txt	2024-03-27 11:41:12 -07:00
Cory Dransfeldt	8aeddbdce8	Create robots.txt	2024-03-27 10:59:01 -07:00