Commit graph

  • 085dd1071e Update from Dark Visitors dark-visitors 2025-07-29 01:11:26 +00:00
  • 9565c11d4c Merge pull request #166 from ai-robots-txt/summaly-bot ai.robots.txt 2025-07-28 19:19:15 +00:00
  • 8869442615
    Merge pull request #166 from ai-robots-txt/summaly-bot Glyn Normington 2025-07-28 20:19:06 +01:00
  • 27420d6fed Merge pull request #164 from GlitzSmarter/main ai.robots.txt 2025-07-28 18:29:57 +00:00
  • 44e58a5ece
    Merge pull request #164 from GlitzSmarter/main Glyn Normington 2025-07-28 19:29:48 +01:00
  • 12c5368e04
    Fix syntax Glyn Normington 2025-07-28 19:29:19 +01:00
  • c15065544a
    Rewriting function in YaK Glitz Smarter 2025-07-28 12:54:25 -04:00
  • 9171625db6
    fix: remove summaly Cory Dransfeldt 2025-07-28 08:56:21 -07:00
  • 8b188d0612 Added YaK user-agent to list of robots Gregory 2025-07-26 23:50:11 -04:00
  • 47769c8429 Update from Dark Visitors v1.38 ai.robots.txt 2025-07-17 01:04:20 +00:00
  • 9843c31303 Update from Dark Visitors dark-visitors 2025-07-16 01:03:43 +00:00
  • 567e94a6cf Merge pull request #160 from sdavids/users/sdavids/feat/htaccess ai.robots.txt 2025-07-08 16:00:16 +00:00
  • 30c4c037d9
    Merge pull request #160 from sdavids/users/sdavids/feat/htaccess Glyn Normington 2025-07-08 16:59:52 +01:00
  • c7407e721d
    Simplify htaccess rewrite rule Sebastian Davids 2025-07-08 12:31:53 +02:00
  • f6f504012f Update from Dark Visitors dark-visitors 2025-07-08 01:01:25 +00:00
  • 63f1f56307 Merge pull request #158 from ai-robots-txt/thinkbot-summaly ai.robots.txt 2025-07-07 16:55:31 +00:00
  • 91a5b3c995
    Merge pull request #158 from ai-robots-txt/thinkbot-summaly Glyn Normington 2025-07-07 17:55:22 +01:00
  • c5dd4e98b6
    chore(robots.json): add SummalyBot + Thinkbot Cory Dransfeldt 2025-07-05 10:32:54 -07:00
  • 70ff0c9fbd Update from Dark Visitors dark-visitors 2025-07-03 01:00:49 +00:00
  • 20a242d390 Merge pull request #156 from nisbet-hubbard/patch-9 ai.robots.txt 2025-07-02 15:58:19 +00:00
  • 91ad302d68
    Merge pull request #156 from nisbet-hubbard/patch-9 Cory Dransfeldt 2025-07-02 08:58:08 -07:00
  • 79bc22523f Update from Dark Visitors dark-visitors 2025-07-02 01:01:05 +00:00
  • 2bfcee794d Update from Dark Visitors ai.robots.txt 2025-06-29 01:07:47 +00:00
  • 7eb2099d3f Update from Dark Visitors dark-visitors 2025-06-28 00:58:57 +00:00
  • accc48f327
    Update funtion and description for SemrushBot nisbet-hubbard 2025-06-21 08:48:58 +08:00
  • 4ed17b8e4a Update from Dark Visitors dark-visitors 2025-06-17 01:00:21 +00:00
  • 5326c202b5 Merge pull request #154 from paulrudy/main ai.robots.txt 2025-06-16 15:12:42 +00:00
  • a31ae1e6d0
    Merge pull request #154 from paulrudy/main Cory Dransfeldt 2025-06-16 08:12:31 -07:00
  • 7535893aec re-add facebookexternalhit paulrudy 2025-06-15 16:39:17 -07:00
  • eb05f2f527 Merge pull request #153 from sergiospagnuolo/Poseidon v1.37 ai.robots.txt 2025-06-14 14:04:03 +00:00
  • 26a46c409d
    Merge pull request #153 from sergiospagnuolo/Poseidon Cory Dransfeldt 2025-06-14 07:03:52 -07:00
  • 2b68568ac2 Update from Dark Visitors dark-visitors 2025-06-14 00:58:11 +00:00
  • b05f2fee00
    Update robots.json with new crawler Sérgio Spagnuolo 2025-06-13 17:15:13 -03:00
  • e53d81c66d Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ai.robots.txt 2025-06-13 09:28:41 +00:00
  • 20e327e74e
    Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot Glyn Normington 2025-06-13 10:28:32 +01:00
  • 8f17718e76
    Fix typo Glyn Normington 2025-06-13 10:28:12 +01:00
  • d760f9216f
    chore(robots.json): adds MyCentralAIScraperBot Cory Dransfeldt 2025-06-12 13:08:29 -07:00
  • 5c170bb6e3
    Merge branch 'ai-robots-txt:main' into feature/ua-synonyms-support Brendan Ngwa Nforbi 2025-06-12 08:49:22 -07:00
  • 3148669363 feat: add ua-synonyms support with tests, scoped to json_to_* functions brndnprog 2025-06-12 08:31:54 -07:00
  • 842e2256e8 Merge pull request #150 from ai-robots-txt/semrush-bots ai.robots.txt 2025-06-12 07:12:00 +00:00
  • 229ea20426
    Merge pull request #150 from ai-robots-txt/semrush-bots Glyn Normington 2025-06-12 08:11:51 +01:00
  • f0606f0eac feat: support ua-synonyms in robots.json to reduce duplication (#144) brndnprog 2025-06-11 22:04:39 -07:00
  • 14d68f05ba
    chore(robots.json): adds additional SemrushBot user agents semrush-bots Cory Dransfeldt 2025-06-11 13:50:53 -07:00
  • cf598b6b71 Update from Dark Visitors v1.36 dark-visitors 2025-06-10 01:00:37 +00:00
  • 3759a6bf14 chore(robots.json): adds EchoboxBot (#148) ai.robots.txt 2025-06-09 15:44:36 +00:00
  • 7867c3e26c
    chore(robots.json): adds EchoboxBot (#148) Cory Dransfeldt 2025-06-09 08:44:25 -07:00
  • e038658aff
    chore(robots.json): adds EchoboxBot Cory Dransfeldt 2025-06-09 08:36:59 -07:00
  • e21f6ae1b6 Update from Dark Visitors v1.35 dark-visitors 2025-06-06 00:59:25 +00:00
  • ac7ed17e71 Merge pull request #145 from ai-robots-txt/aws-bedrockbot ai.robots.txt 2025-06-05 16:51:17 +00:00
  • 81747e6772
    Merge pull request #145 from ai-robots-txt/aws-bedrockbot Glyn Normington 2025-06-05 17:51:03 +01:00
  • 528d77bf07
    chore(robots.json): adds bedrockbot Cory Dransfeldt 2025-06-05 09:14:23 -07:00
  • 77393df5aa Update from Dark Visitors dark-visitors 2025-06-05 00:59:28 +00:00
  • 75ea75a95b Merge pull request #143 from ai-robots-txt/panscient v1.34 ai.robots.txt 2025-06-04 18:04:06 +00:00
  • 2fca1ddcf1
    Merge pull request #143 from ai-robots-txt/panscient Glyn Normington 2025-06-04 19:03:53 +01:00
  • 9c28c63a0c Merge pull request #142 from ai-robots-txt/quillbot ai.robots.txt 2025-06-04 17:54:57 +00:00
  • 395c013eea
    Merge pull request #142 from ai-robots-txt/quillbot Cory Dransfeldt 2025-06-04 10:54:46 -07:00
  • 4568d69b0e
    chore(robots.json): adds Panscient Cory Dransfeldt 2025-06-04 10:54:14 -07:00
  • 03831a7eb5
    chore(robots.json): adds Quillbot Cory Dransfeldt 2025-06-04 10:46:58 -07:00
  • 2b5a59a303 Update from Dark Visitors dark-visitors 2025-06-04 01:00:07 +00:00
  • 3efabc603d Merge pull request #141 from Ivan-Chupin/patch-1 v1.33 ai.robots.txt 2025-06-03 23:28:48 +00:00
  • b35f9a31d7
    Merge pull request #141 from Ivan-Chupin/patch-1 Cory Dransfeldt 2025-06-03 16:28:36 -07:00
  • 8f75f4a2f5
    Add SBIntuitionsBot Ivan Chupin 2025-06-04 03:48:42 +05:00
  • 080946c360 Merge pull request #140 from ai-robots-txt/yandex-bots v1.32 ai.robots.txt 2025-06-03 19:51:25 +00:00
  • 7eec033cad
    Merge pull request #140 from ai-robots-txt/yandex-bots Glyn Normington 2025-06-03 20:51:14 +01:00
  • 3187fd8a32
    chore(robots.json): adds YandexAdditional crawlers Cory Dransfeldt 2025-06-03 12:24:16 -07:00
  • d239e7e5ad Merge pull request #139 from ai-robots-txt/workflow-fix ai.robots.txt 2025-06-03 01:52:35 +00:00
  • 9dbf34010a
    Merge pull request #139 from ai-robots-txt/workflow-fix Glyn Normington 2025-06-03 02:52:23 +01:00
  • 87016d1504 Update from Dark Visitors dark-visitors 2025-06-03 01:00:29 +00:00
  • 899ce01c55
    chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard Cory Dransfeldt 2025-06-02 14:47:31 -07:00
  • fa3ee881ac chore(ai_robots_update.yml): possible workflow fix ai.robots.txt 2025-06-02 21:49:47 +00:00
  • 6135251465
    chore(ai_robots_update.yml): possible workflow fix Cory Dransfeldt 2025-06-02 14:49:27 -07:00
  • b2ff663255
    chore(ai_robots_update.yml): possible workflow fix Cory Dransfeldt 2025-06-02 14:47:31 -07:00
  • 4af776f0a0
    Merge pull request #136 from ai-robots-txt/imgproxy-revert Glyn Normington 2025-06-02 20:21:10 +01:00
  • 1dd66b6969
    Revert "chore(robots.json): adds imgproxy crawler" Cory Dransfeldt 2025-06-02 11:53:06 -07:00
  • 814df6b9a0
    Merge pull request #134 from not-not-the-imp/patch-1 Cory Dransfeldt 2025-05-31 16:03:16 -07:00
  • 268922f8f2
    Update robots.json Cory Dransfeldt 2025-05-31 16:02:05 -07:00
  • 4259b25ccc
    Update robots.json Cory Dransfeldt 2025-05-31 16:01:09 -07:00
  • d22b9ec51a
    Update robots.json Cory Dransfeldt 2025-05-31 16:00:13 -07:00
  • 3e8edd083e
    Add AndiBot and PhindBot imp 2025-05-23 13:03:49 +01:00
  • 093ab81d78 Update from Dark Visitors ai.robots.txt 2025-05-23 00:58:57 +00:00
  • 7bf7f9164d Update from Dark Visitors v1.31 dark-visitors 2025-05-22 00:58:45 +00:00
  • fedb658cc0 Merge pull request #133 from ai-robots-txt/wpbot ai.robots.txt 2025-05-21 21:06:05 +00:00
  • 851eabe059
    Merge pull request #133 from ai-robots-txt/wpbot Glyn Normington 2025-05-21 22:05:51 +01:00
  • 7c5389f4a0 Merge pull request #98 from kylebuckingham/main ai.robots.txt 2025-05-21 19:00:23 +00:00
  • af597586b6
    Merge pull request #98 from kylebuckingham/main Cory Dransfeldt 2025-05-21 12:00:11 -07:00
  • b1d9a60a38
    chore(robots.json): adds wpbot Cory Dransfeldt 2025-05-21 11:40:33 -07:00
  • 1c2acd75b7 Merge pull request #126 from ai-robots-txt/mistral-bot ai.robots.txt 2025-05-21 15:27:26 +00:00
  • 202d3c3b9a
    Merge pull request #126 from ai-robots-txt/mistral-bot Glyn Normington 2025-05-21 16:27:14 +01:00
  • 0a78fe1e76
    Merge pull request #132 from ai-robots-txt/crawler-policy-update Glyn Normington 2025-05-21 15:13:35 +01:00
  • 8b151b2cdc
    Update README.md Cory Dransfeldt 2025-05-21 06:52:36 -07:00
  • 8a8001cbec
    chore(README): updates the opening line of our README to clarify the types of agents we block Cory Dransfeldt 2025-05-20 13:55:25 -07:00
  • fe1267e290
    Merge pull request #131 from Mihitoko/mention-x-robots-tag-for-bing Glyn Normington 2025-05-20 07:52:32 +01:00
  • 9297c7dfa3
    Mention X-Robots-Tag header as alternative for bing Mihitoko 2025-05-19 23:56:57 +02:00
  • 08909d958a
    fix(robots.json): remove duplicate entry for Meta-ExternalFetcher Cory Dransfeldt 2025-05-17 13:34:58 -07:00
  • 7a2e6cba52 Update from Dark Visitors dark-visitors 2025-05-17 00:57:28 +00:00
  • 6c3ae6eb20
    Merge branch 'ai-robots-txt:main' into main Falko Zurell 2025-05-16 13:52:02 +02:00
  • 684d11d889 moved traefik manual setup into docs Falko Zurell 2025-05-16 13:50:58 +02:00
  • dd1ed174b7 Merge pull request #129 from ai-robots-txt/google-cloudvertexbot ai.robots.txt 2025-05-16 11:35:15 +00:00
  • 89c0fbaf86
    Merge pull request #129 from ai-robots-txt/google-cloudvertexbot Glyn Normington 2025-05-16 12:35:04 +01:00
  • 9b5f75e2f3
    Create traefik-manual-setup.md Falko Zurell 2025-05-16 13:27:31 +02:00