github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity
300 commits 6 branches 46 tags 797 KiB
Commit graph

30 commits

Author SHA1 Message Date
dark-visitors
2036a68c1f Update from Dark Visitors 2024-12-04 00:55:50 +00:00
dark-visitors
37065f9118 Update from Dark Visitors 2024-11-24 00:57:05 +00:00
Glyn Normington
80002f5e17 Allow facebookexternalhit 
At the time of writing, this crawler does not
appear to be for the purpose of AI.

See: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
(accessed on 19 November 2024).

Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/40
 2024-11-19 03:33:45 +00:00
dark-visitors
bc0a0ad0e9 Update from Dark Visitors 2024-10-29 00:52:12 +00:00
dark-visitors
fe5f407673 Update from Dark Visitors 2024-10-27 00:54:47 +00:00
Glyn Normington
38a388097c Fix typo and trigger rerun of main job 2024-10-19 04:42:27 +01:00
dark-visitors
faf81efb12 Daily update from Dark Visitors 2024-10-19 01:17:15 +00:00
dark-visitors
b1491d2694 Daily update from Dark Visitors 2024-10-09 01:17:37 +00:00
Laker Turner
dc15afe847
 		Update robots.json with Claude respect link 2024-10-07 17:38:01 +01:00
Cory Dransfeldt
9c2394f23b
 		chore: add ISSCyberRiskCrawler 2024-09-30 16:25:20 -07:00
Cory Dransfeldt
6a988be27f
 		chore: add sidetrade bot 2024-09-28 13:58:00 -07:00
dark-visitors
7851cea4fd Daily update from Dark Visitors 2024-09-27 01:18:04 +00:00
Greg Lindahl
a6de89e6bd feat: make CCBot entry more accurate 2024-09-26 21:41:28 +00:00
dark-visitors
5963cbf9f7 Daily update from Dark Visitors 2024-09-08 01:19:31 +00:00
Cory Dransfeldt
8373294404
 		chore: add iaskspider/2.0 2024-09-06 19:05:26 -07:00
Cory Dransfeldt
0f8723558f
 		chore: add ai2bot 2024-08-28 20:07:32 -07:00
dark-visitors
7bfc1647a8 Daily update from Dark Visitors 2024-08-22 01:11:43 +00:00
dark-visitors
5937434aff Daily update from Dark Visitors 2024-08-15 01:07:15 +00:00
dark-visitors
6a275366be Daily update from Dark Visitors 2024-08-07 10:50:45 +00:00
Chenghao Mou
944bee0f56 call main after update 2024-08-07 11:31:58 +01:00
dark-visitors
cebf809391 Daily update from Dark Visitors 2024-08-07 00:14:26 +00:00
Chenghao Mou
4cf82b703f restore original robots.json 2024-08-06 19:50:38 +01:00
dark-visitors
63c7e742c3 Daily update from Dark Visitors 2024-08-06 16:54:29 +00:00
dark-visitors
fdd261dad4 Daily update from Dark Visitors 2024-08-06 16:27:02 +00:00
Joshua Sheard
146fd4ffba
 		Fix Imagesift user agent 2024-08-04 21:33:04 +01:00
Cory Dransfeldt
1ca936ce11
 		chore: restore FriendlyCrawler + ImageSift 2024-08-04 12:28:48 -07:00
Mirium999
5826c18909 Add ICC-Crawler 2024-08-04 10:11:25 +09:00
Cory Dransfeldt
b20dfec1e4
 		chore: drop in additional data 2024-08-01 15:33:07 -07:00
Cory Dransfeldt
efabf3e721
 		chore: remove test data 2024-08-01 15:25:55 -07:00
Adam Newbold
1fdc79dacb Adding GitHub Action 2024-08-01 18:17:19 -04:00