github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity

Commit graph

  • 379c339f97 skip push if no change Chenghao Mou 2024-08-06 19:40:01 +01:00
  • 01edb6c78c
         Merge branch 'ai-robots-txt:main' into main Chenghao Mou 2024-08-06 19:35:03 +01:00
  • 2a3685385c restrict scope Chenghao Mou 2024-08-06 19:33:49 +01:00
  • 85275e55b8
         Merge pull request #31 from glyn/addfaq Cory Dransfeldt 2024-08-06 11:15:29 -07:00
  • 8c6482fb45 restore the cron Chenghao Mou 2024-08-06 18:12:41 +01:00
  • 63c7e742c3 Daily update from Dark Visitors dark-visitors 2024-08-06 16:54:29 +00:00
  • 55e92f4324 update existing ones Chenghao Mou 2024-08-06 17:48:06 +01:00
  • 52d54cf127 restore the cron Chenghao Mou 2024-08-06 17:28:00 +01:00
  • fdd261dad4 Daily update from Dark Visitors dark-visitors 2024-08-06 16:27:02 +00:00
  • 6d2285f5e0 Add FAQ ai.robots.txt 2024-08-06 16:21:01 +00:00
  • 83d9397f17 Removing previously generated files ai.robots.txt 2024-08-06 16:21:00 +00:00
  • b4d25bf0cb Add FAQ Glyn Normington 2024-08-06 17:20:26 +01:00
  • 8ab1e30a6c test workflow Chenghao Mou 2024-08-06 17:12:26 +01:00
  • 192bf67631 add dark visitor workflow Chenghao Mou 2024-08-06 17:01:56 +01:00
  • e12ddc0f42 Merge pull request #29 from jbowdre/dev ai.robots.txt 2024-08-06 15:44:54 +00:00
  • b54e274bbc Removing previously generated files ai.robots.txt 2024-08-06 15:44:53 +00:00
  • 3e91a84d11
         Merge pull request #29 from jbowdre/dev Cory Dransfeldt 2024-08-04 16:04:59 -07:00
  • b0a93aeb70 only build on changes to robots.json John Bowdre 2024-08-04 17:45:18 -05:00
  • eb924b9856 Merge pull request #28 from jsheard/patch-2 v1.13 ai.robots.txt 2024-08-04 21:54:17 +00:00
  • 1cfc071498 Removing previously generated files ai.robots.txt 2024-08-04 21:54:16 +00:00
  • 24c3509a6e
         Merge pull request #28 from jsheard/patch-2 Cory Dransfeldt 2024-08-04 14:54:06 -07:00
  • c2f177870f Merge pull request #27 from jsheard/patch-1 ai.robots.txt 2024-08-04 21:53:48 +00:00
  • 0072b8f5f0 Removing previously generated files ai.robots.txt 2024-08-04 21:53:47 +00:00
  • 9c7257e7cf
         Merge pull request #27 from jsheard/patch-1 Cory Dransfeldt 2024-08-04 14:53:36 -07:00
  • 8dbbdbf44c
         Add Cloudflares first-party scraper blocking to FAQ Joshua Sheard 2024-08-04 21:38:02 +01:00
  • 146fd4ffba
         Fix Imagesift user agent Joshua Sheard 2024-08-04 21:33:04 +01:00
  • c7b781034e chore: restore FriendlyCrawler + ImageSift v1.12 ai.robots.txt 2024-08-04 19:29:01 +00:00
  • 9a8fa66772 Removing previously generated files ai.robots.txt 2024-08-04 19:29:00 +00:00
  • 1ca936ce11
         chore: restore FriendlyCrawler + ImageSift Cory Dransfeldt 2024-08-04 12:28:48 -07:00
  • 8de5bc8e01 Merge pull request #25 from mirium999/add_icc_crawler ai.robots.txt 2024-08-04 01:21:56 +00:00
  • 8c632e1ba4 Removing previously generated files ai.robots.txt 2024-08-04 01:21:55 +00:00
  • 8d4d52cdab
         Merge pull request #25 from mirium999/add_icc_crawler Adam Newbold 2024-08-03 21:21:45 -04:00
  • 5826c18909 Add ICC-Crawler Mirium999 2024-08-04 10:11:25 +09:00
  • ffbad453f3 Merge pull request #24 from nisbet-hubbard/patch-5 ai.robots.txt 2024-08-03 14:27:47 +00:00
  • b1907d86be Removing previously generated files ai.robots.txt 2024-08-03 14:27:46 +00:00
  • 55c585e9e3
         Merge pull request #24 from nisbet-hubbard/patch-5 Cory Dransfeldt 2024-08-03 07:27:37 -07:00
  • 2b56c72bac
         Update FAQ.md nisbet-hubbard 2024-08-03 14:27:25 +08:00
  • b24e5cb3bb
         Update FAQ.md nisbet-hubbard 2024-08-03 14:12:50 +08:00
  • 74b1502839
         Update FAQ.md nisbet-hubbard 2024-08-03 14:04:58 +08:00
  • d8de1ebdd5 chore: contribution note ai.robots.txt 2024-08-02 16:32:00 +00:00
  • 9d8d3de8ed Removing previously generated files ai.robots.txt 2024-08-02 16:31:59 +00:00
  • 349c35eed6
         chore: contribution note Cory Dransfeldt 2024-08-02 09:31:48 -07:00
  • b144225ece chore: drop in additional data ai.robots.txt 2024-08-01 22:33:23 +00:00
  • 06b950bce9 Removing previously generated files ai.robots.txt 2024-08-01 22:33:23 +00:00
  • b20dfec1e4
         chore: drop in additional data Cory Dransfeldt 2024-08-01 15:33:07 -07:00
  • f18f0d99b9 chore: remove test data ai.robots.txt 2024-08-01 22:29:02 +00:00
  • 747cc834c4 Removing previously generated files ai.robots.txt 2024-08-01 22:29:01 +00:00
  • efabf3e721
         chore: remove test data Cory Dransfeldt 2024-08-01 15:25:55 -07:00
  • 1fdc79dacb Adding GitHub Action Adam Newbold 2024-08-01 18:17:19 -04:00
  • 17a84f2c2d
         chore: update robots table Cory Dransfeldt 2024-08-01 15:06:49 -07:00
  • 6c596a50ea
         chore: move FAQ into repo v1.11 Cory Dransfeldt 2024-08-01 07:53:43 -07:00
  • 6a8e7a8eb0
         Merge pull request #22 from nisbet-hubbard/patch-4 Cory Dransfeldt 2024-08-01 07:49:30 -07:00
  • df89722038
         Add PetalBot (and facebookexternalhit?) nisbet-hubbard 2024-07-31 18:27:29 +08:00
  • fa7b64ae4b
         chore: add Scrapy v1.10 Cory Dransfeldt 2024-07-30 10:28:46 -07:00
  • 55b4505e30
         chore: add Timpibot v1.9 Cory Dransfeldt 2024-07-29 12:38:22 -07:00
  • d49e860b74
         chore: add VelenPublicWebCrawler v1.8 Cory Dransfeldt 2024-07-29 12:12:42 -07:00
  • 6e323554c6
         chore: add Meta-ExternalAgent v1.7 Cory Dransfeldt 2024-07-29 08:27:31 -07:00
  • 1df6d3bec1
         Add meta-externalagent nisbet-hubbard 2024-07-29 20:04:59 +08:00
  • 1c9810d8c5
         Add facebookexternalhit nisbet-hubbard 2024-07-29 19:37:58 +08:00
  • 2972926532
         chore: add OAI-SearchBot v1.6 Cory Dransfeldt 2024-07-26 09:06:10 -07:00
  • c17cae6e9d
         link to bot metrics table Glyn Normington 2024-07-17 02:28:32 +01:00
  • 3692d66918
         chore: update bots table v1.5 Cory Dransfeldt 2024-07-16 12:07:01 -07:00
  • af52578965
         chore: drop google adbot; add GoogleOther bots Cory Dransfeldt 2024-07-16 12:05:34 -07:00
  • 570fd36ea2
         chore: update bots table Cory Dransfeldt 2024-07-10 19:47:23 -07:00
  • 0ca6bce87e
         chore: add ImagesiftBot v1.4 Cory Dransfeldt 2024-07-09 17:41:32 -07:00
  • 0971af19b6
         chore: peer39 unrelated to ai Cory Dransfeldt 2024-07-09 17:39:51 -07:00
  • 74fa789985
         Merge pull request #18 from glyn/add-ref Cory Dransfeldt 2024-06-29 12:22:23 -07:00
  • 4a8ce0b51d Add reference Glyn Normington 2024-06-29 12:20:38 +01:00
  • 70e8985622
         chore: remove unused image v1.3 Cory Dransfeldt 2024-06-22 12:44:02 -07:00
  • fd4ade555c
         Merge pull request #17 from corbindavenport/main Cory Dransfeldt 2024-06-22 12:42:52 -07:00
  • fe98e41546
         Update Perplexity information Corbin Davenport 2024-06-21 22:17:45 -04:00
  • 20dc4e16ad
         Update releases feed links Corbin Davenport 2024-06-21 22:02:54 -04:00
  • 149a72be0c
         Add bot info for omgili, peer39, and youbot Corbin Davenport 2024-06-21 21:43:41 -04:00
  • ec4610a118
         Add information for Google, Meta, and img2dataset bots Corbin Davenport 2024-06-21 20:49:25 -04:00
  • 4163ca92a5
         chore: clean up unnecessary ai.txt Cory Dransfeldt 2024-06-21 08:58:44 -07:00
  • 89de2d2d91
         chore: resolve conflict v1.2 Cory Dransfeldt 2024-06-20 08:12:24 -07:00
  • a980ace544
         chore: update readme Cory Dransfeldt 2024-06-20 08:11:38 -07:00
  • a90ee5e9f0
         chore: clean up bots and narrow scope Cory Dransfeldt 2024-06-20 08:09:21 -07:00
  • 4b8499d718
         Merge pull request #16 from nisbet-hubbard/patch-1 Cory Dransfeldt 2024-06-20 07:57:16 -07:00
  • 56c2285462
         Update robots.txt nisbet-hubbard 2024-06-20 11:31:08 +08:00
  • 429336d725
         fix: resolve conflict v1.1 Cory Dransfeldt 2024-06-15 09:26:14 -07:00
  • 3f65a93891
         chore: keeps Applebot-Extended in favor of Applebot as the latter is simply for search Cory Dransfeldt 2024-06-15 09:25:17 -07:00
  • ca730a81ca
         Merge pull request #15 from christiansievers/main Cory Dransfeldt 2024-06-13 10:50:10 -07:00
  • ce0b99124b
         Update table-of-bot-metrics.md Christian Sievers 2024-06-13 14:17:31 +02:00
  • 1a4868f1a2
         chore: update readme Cory Dransfeldt 2024-06-12 19:05:15 -07:00
  • b5d4d5bc74
         chore: update readme v1 Cory Dransfeldt 2024-06-12 19:01:23 -07:00
  • 548a6686ab
         chore: add rss badge Cory Dransfeldt 2024-06-12 18:56:09 -07:00
  • 1ac25b9135
         chore: propagate applebot-extended out Cory Dransfeldt 2024-06-12 18:53:26 -07:00
  • c546dc048c
         Merge pull request #14 from chriskirknielsen/patch-1 Cory Dransfeldt 2024-06-12 18:51:09 -07:00
  • 39363fc813
         Block Applebot-Extended Christopher Kirk-Nielsen 2024-06-12 16:43:12 -04:00
  • d464f260e0
         Merge branch 'main' of github.com:ai-robots-txt/ai.robots.txt Cory Dransfeldt 2024-06-10 13:23:55 -07:00
  • fd10fbda94
         chore: add cloudflare waf expression Cory Dransfeldt 2024-06-10 13:23:47 -07:00
  • 7a5a21fdad
         chore: update bots table Cory Dransfeldt 2024-05-29 09:21:31 -07:00
  • 2ca4154d4e
         chore: update perplexity entry Cory Dransfeldt 2024-05-29 09:14:35 -07:00
  • e152c110c5
         Merge pull request #10 from kg583/main Cory Dransfeldt 2024-05-20 19:43:29 -07:00
  • 04688e56fb
         Update table-of-bot-metrics.md KG 2024-05-16 18:25:46 -05:00
  • 696d2acef8
         chore: add cohere-ai Cory Dransfeldt 2024-05-06 10:39:07 -07:00
  • 6940db2412
         chore: add anthropic bots to table Cory Dransfeldt 2024-05-06 10:36:16 -07:00
  • ec8e10a56c
         chore: add bytespider to table Cory Dransfeldt 2024-05-06 10:34:43 -07:00
  • 262fe71ad9
         chore: add ccbot data to table Cory Dransfeldt 2024-05-06 10:32:05 -07:00