bcc4e408e2
The non-AI user agent list covers synthetic prefix/suffix variants (NotCursor, CursorNot). It had no real-world agent that embeds a listed name mid-string, which is what #208 actually reported: SLCC1 and SLCC2 in older Internet Explorer and Trident agents span the listed agent LCC. Verified the case is load-bearing: removing the word boundaries from list_to_pcre fails this test with AssertionError: <re.Match object; span=(65, 68), match='LCC'> is not None Also document in the FAQ that agent names are matched as whole words, for anyone consuming robots.json directly and writing their own matcher.
5.8 KiB
Frequently asked questions
Why should we block these crawlers?
They're extractive, confer no benefit to the creators of data they're ingesting and also have wide-ranging negative externalities: particularly copyright abuse and environmental impact.
How Tech Giants Cut Corners to Harvest Data for A.I.
OpenAI, Google and Meta ignored corporate policies, altered their own rules and discussed skirting copyright law as they sought online information to train their newest artificial intelligence systems.
How AI copyright lawsuits could make the whole industry go extinct
The New York Times' lawsuit against OpenAI is part of a broader, industry-shaking copyright challenge that could define the future of AI.
Reconciling the contrasting narratives on the environmental impact of large language models
Studies have shown that the training of just one LLM can consume as much energy as five cars do across their lifetimes. The water footprint of AI is also substantial; for example, recent work has highlighted that water consumption associated with AI models involves data centers using millions of gallons of water per day for cooling. Additionally, the energy consumption and carbon emissions of AI are projected to grow quickly in the coming years [...].
Scientists Predict AI to Generate Millions of Tons of E-Waste
we could end up with between 1.2 million and 5 million metric tons of additional electronic waste by the end of this decade [the 2020's].
How do we know AI companies/bots respect robots.txt?
The short answer is that we don't. robots.txt is a well-established standard, but compliance is voluntary. There is no enforcement mechanism.
Why might AI web crawlers respect robots.txt?
Larger and/or reputable companies developing AI models probably wouldn't want to damage their reputation by ignoring robots.txt.
Also, given the contentious nature of AI and the possibility of legislation limiting its development, companies developing AI models will probably want to be seen to be behaving ethically, and so should (eventually) respect robots.txt.
Can we block crawlers based on user agent strings?
Yes, provided the crawlers identify themselves and your application/hosting supports doing so.
Some crawlers — such as Perplexity — do not identify themselves via their user agent strings and, as such, are difficult to block.
Can I use robots.json directly in my own tooling?
You're welcome to, with a caveat. robots.json isn't intended as a primary
deliverable of this project — the generated configuration files are. If you consume
it yourself, note that the agent names are matched as whole words, not as
substrings.
The generated configs wrap the list in \b(...)\b for exactly this reason. Without
word boundaries, short agent names match inside unrelated strings: the listed agent
LCC appears inside SLCC1, a Windows licensing component present in the user
agent of older Internet Explorer and Trident-based browsers. A naive substring match
would block those real visitors (#208).
If you build your own matcher from robots.json, use word-boundary matching.
What can we do if a bot doesn't respect robots.txt?
That depends on your stack.
- Nginx
- Blocking Bots with Nginx by Robb Knight
- Blocking AI web crawlers by Glyn Normington
- Apache httpd
- Blockin' bots. by Ethan Marcotte
- Blocking Bots With 11ty And Apache by fLaMEd fury
Tip
The snippets in these articles all use
mod_rewrite, which should be considered a last resort. A good alternative that's less resource-intensive ismod_setenvif; see httpd docs for an example. You should also consider setting this up inhttpd.confinstead of.htaccessif it's available to you. - Netlify
- Blockin' bots on Netlify by Jeremia Kimelman
- Cloudflare
- Block AI bots, scrapers and crawlers with a single click by Cloudflare
- I’m blocking AI crawlers by Roelant
- Vercel
- Block AI Bots Firewall Rule by Vercel
How can I contribute?
Open a pull request. It will be reviewed and acted upon appropriately. We really appreciate contributions — this is a community effort.
I'd like to donate money
That's kind of you, but we don't need your money. If you insist, we'd love you to make a donation to the American Civil Liberties Union, the Disasters Emergency Committee, or a similar organisation.
Can my company sponsor ai.robots.txt?
No, thank you. We do not accept sponsorship of any kind. We prefer to maintain our independence. Our costs are negligible as we are entirely volunteer-based and community-driven.