Added at a maintainer's request in #277, where I raised it but did not
propose it.
OpenAI documents it at https://developers.openai.com/api/docs/bots as the
fourth of its four crawlers. Two things in that entry are worth recording
in the fields rather than leaving to the reader:
- The page does not say OAI-AdsBot obeys robots.txt, and gives no
disallow example, so "respect" is "Unclear at this time." rather than
the "Yes" the other three OpenAI crawlers carry.
- OpenAI states the data is not used to train foundation models, and that
the crawler only visits pages submitted as ads. What makes it belong on
this list is the other half of the same paragraph: content from the
landing page is also used to "determine when it's most relevant to show
the ad to users".
Only robots.json is changed; the other files are regenerated
automatically.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141w8LsSMs7AHbon2ySf84n
Diffbot documents two separate user agents. The list currently has only
the proactive crawler; this adds the user-triggered one.
From Diffbot's own documentation:
Diffbot-User — This is used by requests originating on behalf of a
human user browsing a URL using Diffbot software, in response to
their input.
Source: https://docs.diffbot.com/docs/does-crawl-respect-robotstxt
Same distinction as GPTBot / ChatGPT-User and PerplexityBot /
Perplexity-User, so it is classified as an AI Assistant fetching only
when prompted by a user.
Observed in the robots.txt of five independent publishers in four
countries: nytimes.com, bbc.co.uk, lefigaro.fr, corriere.it and
usatoday.com.
Generated files updated with code/robots.py --convert.
Applies @glyn's suggestions: less emphatic phrasing, and the SLCC1 detail
replaced by a link to issue 208 rather than restated here.
Co-authored-by: glyn <glyn@users.noreply.github.com>
The non-AI user agent list covers synthetic prefix/suffix variants
(NotCursor, CursorNot). It had no real-world agent that embeds a listed
name mid-string, which is what #208 actually reported: SLCC1 and SLCC2 in
older Internet Explorer and Trident agents span the listed agent LCC.
Verified the case is load-bearing: removing the word boundaries from
list_to_pcre fails this test with
AssertionError: <re.Match object; span=(65, 68), match='LCC'> is not None
Also document in the FAQ that agent names are matched as whole words, for
anyone consuming robots.json directly and writing their own matcher.
The Lightpanda browser is an AI-centric headless browser used for scraping and automation.
It is currently being used by a botnet of millions of IP addresses, including over 20k SpaceX IPs, to scrape sites such as the official Haskell Gitlab:
https://github.com/lightpanda-io/browser/issues/3156
Free, static directory of verified AI crawlers with a one-click robots.txt/llms.txt generator, thematically aligned with this project's goal of helping site owners manage AI crawler access.
- nginx and lighttpd allow single quoted strings, so use python's
built-in methods to escape quotes
- only apache requires parentheses on the outside (and only for obscure
reasons)
- caddy expects double quoted strings. at least escape inner quotes
directly (re.escape does not do this since python 3.7), should any
appear
