Applies @glyn's suggestions: less emphatic phrasing, and the SLCC1 detail
replaced by a link to issue 208 rather than restated here.
Co-authored-by: glyn <glyn@users.noreply.github.com>
The non-AI user agent list covers synthetic prefix/suffix variants
(NotCursor, CursorNot). It had no real-world agent that embeds a listed
name mid-string, which is what #208 actually reported: SLCC1 and SLCC2 in
older Internet Explorer and Trident agents span the listed agent LCC.
Verified the case is load-bearing: removing the word boundaries from
list_to_pcre fails this test with
AssertionError: <re.Match object; span=(65, 68), match='LCC'> is not None
Also document in the FAQ that agent names are matched as whole words, for
anyone consuming robots.json directly and writing their own matcher.
The Lightpanda browser is an AI-centric headless browser used for scraping and automation.
It is currently being used by a botnet of millions of IP addresses, including over 20k SpaceX IPs, to scrape sites such as the official Haskell Gitlab:
https://github.com/lightpanda-io/browser/issues/3156
Free, static directory of verified AI crawlers with a one-click robots.txt/llms.txt generator, thematically aligned with this project's goal of helping site owners manage AI crawler access.
- nginx and lighttpd allow single quoted strings, so use python's
built-in methods to escape quotes
- only apache requires parentheses on the outside (and only for obscure
reasons)
- caddy expects double quoted strings. at least escape inner quotes
directly (re.escape does not do this since python 3.7), should any
appear
Re-submission of #248, which was merged then reverted in 80c19fc. The revert was
caused by main.yml's commit-message handling, not by this data — that is fixed
separately in #254.
Six entries move from "Unclear at this time." to the operator's own documented
value, each citing the primary source inline:
Applebot Yes support.apple.com/en-us/119829#retrieval
Meta-ExternalAgent Yes developers.facebook.com/docs/sharing/webmasters/web-crawlers/
Meta-ExternalFetcher No (same Meta page — user-initiated fetch, documented
meta-externalfetcher No as not checking robots.txt)
DuckAssistBot Yes duckduckgo.com/duckduckgo-help-pages/results/duckassistbot/
Google-Agent Yes developers.google.com/search/docs/crawling-indexing/google-common-crawlers#google-agent
table-of-bot-metrics.md is regenerated and included, so `robots.py --convert` has
nothing left to write and the workflow's commit step is skipped entirely.
Verified: code/tests.py 13/13 · robots.py --convert exits 0 and is idempotent ·
robots.txt, .htaccess, the nginx/lighttpd/haproxy configs and the Caddyfile are
byte-identical · the JSON diff touches only the six `respect` fields, nothing else.
`git commit -m "${{ github.event.head_commit.message }}"` splices arbitrary text
straight into a double-quoted bash string. A message containing a double quote
closes the string early and the remainder is re-parsed as shell words.
This is what broke main after #248 merged. That PR's title contained
"Unclear at this time." (with quotes), so the merge commit message did too, and
the line bash actually ran was:
git commit -m "Fill "Unclear at this time." from primary operator docs ..."
which git received as:
-m "Fill Unclear" at this "time. from primary operator docs ..."
Hence `error: pathspec 'at' did not match any file(s) known to git`, a failed
run, and the revert in 80c19fc. The data in that PR was fine — robots.py
--convert exits 0 against it and code/tests.py passes 13/13.
The same interpolation is also a script-injection vector, which is the more
important reason to change it: a PR title is attacker-controlled, and
chore: tidy"; <any command>; echo "
executes that command on the runner with the workflow's token. `inputs.message`
has the same shape in the `if [ -n ... ]` test and its own `git commit -m`, so
all three are moved.
Passing through `env:` and quoting the shell variable is GitHub's documented
recommendation for untrusted values. The variable is expanded by bash after
parsing, so quotes, newlines and `$(...)` stay literal text.
Verified: code/tests.py 13/13 · python code/robots.py --convert exits 0 and
leaves robots.txt, table-of-bot-metrics.md and every server-config output
byte-identical · reproduced both the parse failure and the injection locally
against the old form, and confirmed the env form commits the same message
verbatim, quotes included.
