QwenBot, ERNIEBot, DoubaoBot and MistralAI-Index appear in the robots.txt
of Dutch news sites, measured against 83 domains in August 2026, but were
missing from this list.
MistralAI-Index is documented by the vendor at https://docs.mistral.ai/robots
and is the third of Mistral's three crawlers.
For QwenBot, ERNIEBot and DoubaoBot the evidence is different in kind and
the fields say so: the operator is identifiable from the name and the
company's own model family, but none of the three is documented on a
vendor crawler page I could find, so "respect" stays "Unclear at this
time." They are included because publishers are already naming them,
which is the same reason this list carries other undocumented agents.
Only robots.json is changed; the other files are regenerated
automatically.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141w8LsSMs7AHbon2ySf84n
All three are documented by the vendor itself:
- KimiBot and Kimi-SearchBot: https://www.kimi.ai/policies/kimi-crawlers
The list already has Kimi-User; Moonshot documents three crawlers.
- MistralAI-Training: https://docs.mistral.ai/robots/
The list already has MistralAI-User. Mistral documents three; the
third, MistralAI-Index, is proposed separately in #274.
Only robots.json is changed, per review: the other files are regenerated
automatically. Rebased on main so that Diffbot-User, merged in the
meantime, is preserved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141w8LsSMs7AHbon2ySf84n
Added at a maintainer's request in #277, where I raised it but did not
propose it.
OpenAI documents it at https://developers.openai.com/api/docs/bots as the
fourth of its four crawlers. Two things in that entry are worth recording
in the fields rather than leaving to the reader:
- The page does not say OAI-AdsBot obeys robots.txt, and gives no
disallow example, so "respect" is "Unclear at this time." rather than
the "Yes" the other three OpenAI crawlers carry.
- OpenAI states the data is not used to train foundation models, and that
the crawler only visits pages submitted as ads. What makes it belong on
this list is the other half of the same paragraph: content from the
landing page is also used to "determine when it's most relevant to show
the ad to users".
Only robots.json is changed; the other files are regenerated
automatically.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141w8LsSMs7AHbon2ySf84n
Diffbot documents two separate user agents. The list currently has only
the proactive crawler; this adds the user-triggered one.
From Diffbot's own documentation:
Diffbot-User — This is used by requests originating on behalf of a
human user browsing a URL using Diffbot software, in response to
their input.
Source: https://docs.diffbot.com/docs/does-crawl-respect-robotstxt
Same distinction as GPTBot / ChatGPT-User and PerplexityBot /
Perplexity-User, so it is classified as an AI Assistant fetching only
when prompted by a user.
Observed in the robots.txt of five independent publishers in four
countries: nytimes.com, bbc.co.uk, lefigaro.fr, corriere.it and
usatoday.com.
Generated files updated with code/robots.py --convert.
Applies @glyn's suggestions: less emphatic phrasing, and the SLCC1 detail
replaced by a link to issue 208 rather than restated here.
Co-authored-by: glyn <glyn@users.noreply.github.com>
The non-AI user agent list covers synthetic prefix/suffix variants
(NotCursor, CursorNot). It had no real-world agent that embeds a listed
name mid-string, which is what #208 actually reported: SLCC1 and SLCC2 in
older Internet Explorer and Trident agents span the listed agent LCC.
Verified the case is load-bearing: removing the word boundaries from
list_to_pcre fails this test with
AssertionError: <re.Match object; span=(65, 68), match='LCC'> is not None
Also document in the FAQ that agent names are matched as whole words, for
anyone consuming robots.json directly and writing their own matcher.
The Lightpanda browser is an AI-centric headless browser used for scraping and automation.
It is currently being used by a botnet of millions of IP addresses, including over 20k SpaceX IPs, to scrape sites such as the official Haskell Gitlab:
https://github.com/lightpanda-io/browser/issues/3156
Free, static directory of verified AI crawlers with a one-click robots.txt/llms.txt generator, thematically aligned with this project's goal of helping site owners manage AI crawler access.
