robots.txt user-agent matching is case-insensitive, so two entries whose
names differ only in case are the same crawler. updated_robots_json keyed
off the scraped name, so when knownagents.com changed the capitalisation
of a name the ingest added a second entry instead of updating the first.
consolidate() also looks the name up by exact key, so the curated operator
and respect values were left behind on the old key rather than carried over.
robots.json currently carries three such pairs, and each one emits a
duplicate User-agent line in every generated file.
This resolves an incoming name against the keys already present before
using it, so an existing entry is updated in place. No generated file
changes as a result.
QwenBot, ERNIEBot, DoubaoBot and MistralAI-Index appear in the robots.txt
of Dutch news sites, measured against 83 domains in August 2026, but were
missing from this list.
MistralAI-Index is documented by the vendor at https://docs.mistral.ai/robots
and is the third of Mistral's three crawlers.
For QwenBot, ERNIEBot and DoubaoBot the evidence is different in kind and
the fields say so: the operator is identifiable from the name and the
company's own model family, but none of the three is documented on a
vendor crawler page I could find, so "respect" stays "Unclear at this
time." They are included because publishers are already naming them,
which is the same reason this list carries other undocumented agents.
Only robots.json is changed; the other files are regenerated
automatically.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141w8LsSMs7AHbon2ySf84n
All three are documented by the vendor itself:
- KimiBot and Kimi-SearchBot: https://www.kimi.ai/policies/kimi-crawlers
The list already has Kimi-User; Moonshot documents three crawlers.
- MistralAI-Training: https://docs.mistral.ai/robots/
The list already has MistralAI-User. Mistral documents three; the
third, MistralAI-Index, is proposed separately in #274.
Only robots.json is changed, per review: the other files are regenerated
automatically. Rebased on main so that Diffbot-User, merged in the
meantime, is preserved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141w8LsSMs7AHbon2ySf84n
Added at a maintainer's request in #277, where I raised it but did not
propose it.
OpenAI documents it at https://developers.openai.com/api/docs/bots as the
fourth of its four crawlers. Two things in that entry are worth recording
in the fields rather than leaving to the reader:
- The page does not say OAI-AdsBot obeys robots.txt, and gives no
disallow example, so "respect" is "Unclear at this time." rather than
the "Yes" the other three OpenAI crawlers carry.
- OpenAI states the data is not used to train foundation models, and that
the crawler only visits pages submitted as ads. What makes it belong on
this list is the other half of the same paragraph: content from the
landing page is also used to "determine when it's most relevant to show
the ad to users".
Only robots.json is changed; the other files are regenerated
automatically.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141w8LsSMs7AHbon2ySf84n
