From 3b67c9c54ef25c499c58f586861be4e731ff90c8 Mon Sep 17 00:00:00 2001 From: Ideenschmiede CEO Date: Tue, 25 Aug 2026 12:15:02 +0000 Subject: [PATCH 1/2] Add KI-Zugangsindex to Related resources --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index da93433..5313a4e 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,8 @@ file on-the-fly. - [Bot Ledger](https://farrelldan.github.io/ai-bot-directory/): free, static directory of verified AI crawlers with a one-click `robots.txt` and `llms.txt` generator. No signup required. +- [KI-Zugangsindex](https://peppe1337.github.io/ki-zugangsindex/): open dataset on how widely this kind of blocking is actually deployed in the German (`.de`) web, measured on a fixed panel of 600 domains so the same sites can be re-checked over time. First measurement 2026-08-25: of the domains serving a parseable `robots.txt`, 32.8% in the top 300 (77/235) disallow at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, against 14.8% (31/210) of domains outside the top 50,000. All 600 raw responses and the measurement code are included. + ## Contributing A note about contributing: updates should be added/made to `robots.json`. A GitHub action will then generate the updated `robots.txt`, `table-of-bot-metrics.md`, `.htaccess` and `nginx-block-ai-bots.conf`. From 5e73986001b29bbcac27020a9059906bf0058df8 Mon Sep 17 00:00:00 2001 From: Ideenschmiede CEO Date: Wed, 26 Aug 2026 16:02:12 +0000 Subject: [PATCH 2/2] Apply maintainer's suggestion: trim entry to avoid stale figures --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5313a4e..1e042de 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ file on-the-fly. - [Bot Ledger](https://farrelldan.github.io/ai-bot-directory/): free, static directory of verified AI crawlers with a one-click `robots.txt` and `llms.txt` generator. No signup required. -- [KI-Zugangsindex](https://peppe1337.github.io/ki-zugangsindex/): open dataset on how widely this kind of blocking is actually deployed in the German (`.de`) web, measured on a fixed panel of 600 domains so the same sites can be re-checked over time. First measurement 2026-08-25: of the domains serving a parseable `robots.txt`, 32.8% in the top 300 (77/235) disallow at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, against 14.8% (31/210) of domains outside the top 50,000. All 600 raw responses and the measurement code are included. +- [KI-Zugangsindex](https://peppe1337.github.io/ki-zugangsindex/): open dataset on how widely this kind of blocking is actually deployed in the German (`.de`) web, measured on a fixed panel of 600 domains so the same sites can be re-checked over time. ## Contributing