Commit graph

  • 8e6bf36738
    Merge 7d6827bc72 into aa8519ec10 fiskhandlarn 2025-12-21 12:46:59 +00:00
  • aa8519ec10 Update from Dark Visitors main ai.robots.txt 2025-12-21 01:07:06 +00:00
  • 83485effdb Update from Dark Visitors dark-visitors 2025-12-20 00:58:49 +00:00
  • 8b8bf9da5d Update from Dark Visitors ai.robots.txt 2025-12-06 00:58:16 +00:00
  • f1c752ef12 Update from Dark Visitors dark-visitors 2025-12-05 01:00:44 +00:00
  • 51afa7113a
    Update ai_robots_update.yml with rebase command to fix scheduled run Adam Newbold 2025-12-03 20:15:10 -05:00
  • 7598d77e4a Update from Dark Visitors ai.robots.txt 2025-12-04 01:10:02 +00:00
  • 45b071b29f Update from Dark Visitors dark-visitors 2025-12-04 01:00:27 +00:00
  • f61b3496f7 Update from Dark Visitors dark-visitors 2025-12-03 01:00:32 +00:00
  • 8363d4fdd4 Update from Dark Visitors dark-visitors 2025-12-02 01:25:24 +00:00
  • 2ccd443581
    Update ai_robots_update.yml with workflow_dispatch Adam Newbold 2025-12-01 20:24:41 -05:00
  • 6d75f3c1c9
    Update robots.py to address error on line 57 Adam Newbold 2025-12-01 20:18:29 -05:00
  • 56010ef913
    Merge pull request #205 from fiskhandlarn/fix/editorconfig Glyn Normington 2025-11-29 10:02:08 +00:00
  • 3fadc88a23 Merge pull request #206 from newbold/main ai.robots.txt 2025-11-29 10:00:52 +00:00
  • 47c077a8ef
    Merge pull request #206 from newbold/main Glyn Normington 2025-11-29 10:00:42 +00:00
  • f5d7ccb243
    Fixed invalid JSON Adam Newbold 2025-11-28 14:10:38 -05:00
  • 30d719a09a
    Adding LAIONDownloader Adam Newbold 2025-11-28 14:08:18 -05:00
  • 05bbdebeaa feat: disallow final newline for files generated by python fiskhandlarn 2025-11-28 10:47:22 +01:00
  • c6ce9329a1 fix: ensure whitespace as defined in .editorconfig fiskhandlarn 2025-11-28 10:46:22 +01:00
  • 7d6827bc72 fix: generate requirements.txt for this repo only fiskhandlarn 2025-11-28 10:30:17 +01:00
  • 10d5ae2870 Merge pull request #200 from glyn/deepseek ai.robots.txt 2025-11-27 14:00:50 +00:00
  • 4467002298
    Merge pull request #200 from glyn/deepseek Glyn Normington 2025-11-27 14:00:40 +00:00
  • 4a159a818f
    Merge pull request #199 from glyn/editorconfig Glyn Normington 2025-11-27 13:54:03 +00:00
  • 3d6b33a71a
    Merge pull request #202 from glyn/formatting Glyn Normington 2025-11-27 12:42:31 +00:00
  • c26c8c0911 Tidy README Glyn Normington 2025-11-27 12:41:37 +00:00
  • 91959fe791
    Merge pull request #201 from Anshita-18H/add-requirements-file Glyn Normington 2025-11-27 12:38:28 +00:00
  • b75163e796
    Ensure Python3 is used Glyn Normington 2025-11-27 12:38:12 +00:00
  • f46754d280
    Order deps Glyn Normington 2025-11-27 12:37:35 +00:00
  • c3f2fe758e
    Whitespace Glyn Normington 2025-11-27 12:37:21 +00:00
  • 7521c3af50
    Fix link Glyn Normington 2025-11-27 12:37:07 +00:00
  • 4302fd1aca Improve contributing section and fix formatting in README Anshita-18H 2025-11-27 17:56:47 +05:30
  • 9ca5033927 Deduplicate requirements.txt and add installation instructions to README Anshita-18H 2025-11-27 17:44:47 +05:30
  • 9cc3dbc05f Add requirements.txt with project dependencies Anshita-18H 2025-11-27 17:06:59 +05:30
  • c322d7852d
    Merge pull request #130 from maxheadroom/main Glyn Normington 2025-11-26 12:08:11 +00:00
  • 35588b2ddb
    Typo. Glyn Normington 2025-11-26 12:07:39 +00:00
  • ab24e41106 Clarify that DeepSeekBot does not respect robots.txt Glyn Normington 2025-11-26 12:02:42 +00:00
  • b681c0c0d8 Standardise editor options Glyn Normington 2025-11-26 11:54:10 +00:00
  • 4e7e28335f
    Merge pull request #195 from fiskhandlarn/patch-2 Glyn Normington 2025-11-26 11:50:15 +00:00
  • a6cf6b204b test: update test nginx conf fiskhandlarn 2025-11-25 16:47:04 +01:00
  • 2679fcad34 feat: update nginx generator fiskhandlarn 2025-11-25 16:46:31 +01:00
  • ef8eda4fe6 chore: normalize quote style fiskhandlarn 2025-11-25 16:39:35 +01:00
  • a29102f0fc
    Merge pull request #196 from glyn/releasing v1.44 Adam Newbold 2025-11-25 10:34:25 -05:00
  • 0b3266b35f feat: allow robots access to /robots.txt in nginx fiskhandlarn 2025-11-20 11:50:37 +01:00
  • 6225e3e98e Document how to ship a new release Glyn Normington 2025-11-23 04:01:43 +00:00
  • be4d74412c Merge pull request #194 from glyn/193-kendra ai.robots.txt 2025-11-21 16:32:11 +00:00
  • 729be4693a
    Merge pull request #194 from glyn/193-kendra Cory Dransfeldt 2025-11-21 08:32:01 -08:00
  • efb4d260da delete extraneous hyphen Glyn Normington 2025-11-20 10:23:09 +00:00
  • e2726ac160 Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo v1.43 ai.robots.txt 2025-11-14 21:01:25 +00:00
  • 663d030f96
    Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo Glyn Normington 2025-11-14 21:01:13 +00:00
  • 28b45ea08d
    chore: adds NotebookLM and KlaviyoAIBot agents Cory Dransfeldt 2025-11-14 11:03:03 -08:00
  • 443dd27527 Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots ai.robots.txt 2025-11-05 23:50:59 +00:00
  • 60b6a0829d
    Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots Glyn Normington 2025-11-05 23:50:48 +00:00
  • 00bf2b0e13
    chore(robots.json): add AmazonBuyForMe and atlassian-bot Cory Dransfeldt 2025-11-05 14:02:37 -08:00
  • e87eb706e3 Merge pull request #188 from ai-robots-txt/cdransf/buddybot ai.robots.txt 2025-11-04 05:16:28 +00:00
  • 3d41350256
    Merge pull request #188 from ai-robots-txt/cdransf/buddybot Glyn Normington 2025-11-04 05:16:15 +00:00
  • 808451055c
    chore: adds BuddyBot Cory Dransfeldt 2025-11-03 13:39:07 -08:00
  • 5cad0ee389 Merge pull request #187 from ai-robots-txt/cdransf/add-Linguee-Bot ai.robots.txt 2025-10-24 18:05:01 +00:00
  • d55c9980cd
    Merge pull request #187 from ai-robots-txt/cdransf/add-Linguee-Bot Glyn Normington 2025-10-24 19:04:42 +01:00
  • 192b0a2eef
    chore: adds Linguee Bot Cory Dransfeldt 2025-10-24 10:35:04 -07:00
  • 97e19445ce Update from Dark Visitors dark-visitors 2025-10-24 00:53:30 +00:00
  • 0bc2361be8 Merge pull request #186 from ai-robots-txt/third-party ai.robots.txt 2025-10-21 15:23:46 +00:00
  • 511d8c955d
    Merge pull request #186 from ai-robots-txt/third-party Cory Dransfeldt 2025-10-21 08:23:28 -07:00
  • b89a9eae6a Use third party in text, rather than first third-party Glyn Normington 2025-10-21 01:11:32 +01:00
  • 646ab08e15 Merge pull request #185 from ai-robots-txt/cdransf/add-IbouBot ai.robots.txt 2025-10-21 00:09:24 +00:00
  • 5c3da1c1af
    Merge pull request #185 from ai-robots-txt/cdransf/add-IbouBot Glyn Normington 2025-10-21 01:09:06 +01:00
  • d22b8dfd7b
    Description is third party, not first Glyn Normington 2025-10-21 01:08:42 +01:00
  • 19c1d346c3
    chore: adds IbouBot Cory Dransfeldt 2025-10-20 09:28:31 -07:00
  • 2fa0e9119c Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare v1.42 ai.robots.txt 2025-10-20 14:34:43 +00:00
  • 0874a92503
    Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare Glyn Normington 2025-10-20 15:34:30 +01:00
  • 28d2d09633
    chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot cdransf/bot-additions-cloudflare Cory Dransfeldt 2025-10-19 14:26:05 -07:00
  • 25ba28c219
    chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot cdransf/bot-additions Cory Dransfeldt 2025-10-19 14:26:05 -07:00
  • 260f5029fe Update from Dark Visitors ai.robots.txt 2025-10-16 00:56:33 +00:00
  • 91bf905fa9 Update from Dark Visitors dark-visitors 2025-10-15 00:56:30 +00:00
  • 56d03d46fb Update from Dark Visitors v1.41 dark-visitors 2025-09-26 00:54:30 +00:00
  • 38d60b928c Merge pull request #179 from ai-robots-txt/deepseekbot ai.robots.txt 2025-09-25 10:17:38 +00:00
  • e2266bbc1d
    Merge pull request #179 from ai-robots-txt/deepseekbot Glyn Normington 2025-09-25 11:17:23 +01:00
  • bf347bdf91
    chore(robots.json): add DeepSeekBot Cory Dransfeldt 2025-09-24 09:42:01 -07:00
  • c6e7d69dd5
    Update README.md nisbet-hubbard 2025-09-14 19:41:36 +08:00
  • 9e6a479bd3
    Update README.md nisbet-hubbard 2025-09-14 19:41:36 +08:00
  • 8906f6b447 Update from Dark Visitors ai.robots.txt 2025-09-12 00:53:06 +00:00
  • 2fd93029ca Update from Dark Visitors dark-visitors 2025-09-11 00:55:06 +00:00
  • b6338ddc73 Merge pull request #176 from emersion/TerraCotta ai.robots.txt 2025-09-10 14:11:16 +00:00
  • 7ffbf33baf
    Merge pull request #176 from emersion/TerraCotta Cory Dransfeldt 2025-09-10 07:11:04 -07:00
  • 50870ba911 Add TerraCotta Simon Ser 2025-09-10 12:04:08 +02:00
  • dd391bf960 Update from Dark Visitors dark-visitors 2025-09-10 00:53:56 +00:00
  • 229d1b4dbc Merge pull request #174 from karolyi/master ai.robots.txt 2025-09-09 02:45:32 +00:00
  • 4d506ca322
    Merge pull request #174 from karolyi/master Glyn Normington 2025-09-09 03:45:23 +01:00
  • ec508ab434
    Update Brightbot operator and details; add meta-webindexer entry László Károlyi 2025-09-08 21:42:06 +02:00
  • d1e0a9a757
    Add meta-webindexer bot and update Brightbot operator info László Károlyi 2025-09-07 12:12:35 +02:00
  • 0ed29412c9 Update from Dark Visitors v1.40 dark-visitors 2025-08-28 00:55:47 +00:00
  • 2ad1c3e831 Merge pull request #168 from ai-robots-txt/google-firebase-shapbot 1.40 ai.robots.txt 2025-08-27 10:13:17 +00:00
  • 1677278c5a
    Merge pull request #168 from ai-robots-txt/google-firebase-shapbot Glyn Normington 2025-08-27 11:13:04 +01:00
  • 1a8edfa84a
    chore: add Google-Firebase and ShapBot Cory Dransfeldt 2025-08-26 19:39:42 -07:00
  • cf073d49f2 Update from Dark Visitors dark-visitors 2025-08-15 01:01:42 +00:00
  • 6d3f3e1712 Merge pull request #167 from ai-robots-txt/OpenAI-bot v1.39 ai.robots.txt 2025-08-14 01:45:05 +00:00
  • 784b8440a5
    Merge pull request #167 from ai-robots-txt/OpenAI-bot Glyn Normington 2025-08-14 02:44:53 +01:00
  • 0e687a5b58
    chore: adds OpenAI but user agent Cory Dransfeldt 2025-08-13 09:01:55 -07:00
  • ff9fc26404 Update from Dark Visitors ai.robots.txt 2025-08-01 01:12:53 +00:00
  • 146a229662 Update from Dark Visitors dark-visitors 2025-07-31 01:05:21 +00:00
  • 64f9d6ce9c Update from Dark Visitors ai.robots.txt 2025-07-30 01:05:21 +00:00