17 commits

Author	SHA1	Message	Date
Saliu Jamiu Olamilekan	66a8329a61	docs: trim the FAQ entry per review ... Applies @glyn's suggestions: less emphatic phrasing, and the SLCC1 detail replaced by a link to issue 208 rather than restated here. Co-authored-by: glyn <glyn@users.noreply.github.com>	2026-08-18 21:13:10 -07:00
Saliu Jamiu Olamilekan	bcc4e408e2	test: pin the SLCC1/SLCC2 false positive from #208 ... The non-AI user agent list covers synthetic prefix/suffix variants (NotCursor, CursorNot). It had no real-world agent that embeds a listed name mid-string, which is what #208 actually reported: SLCC1 and SLCC2 in older Internet Explorer and Trident agents span the listed agent LCC. Verified the case is load-bearing: removing the word boundaries from list_to_pcre fails this test with AssertionError: <re.Match object; span=(65, 68), match='LCC'> is not None Also document in the FAQ that agent names are matched as whole words, for anyone consuming robots.json directly and writing their own matcher.	2026-08-18 20:28:19 -07:00
Glyn Normington	b1856e6988	Donations	2025-04-18 18:40:44 +01:00
Glyn Normington	d05ede8fe1	Clarify our position on sponsorship ... Some firms, including those with .ai domains, have offered to sponsor this project. So make our position clear.	2025-04-18 17:46:56 +01:00
Glyn Normington	d50615d394	Improve formatting ... This clarifies the scope of the tip is Apache httpd.	2024-11-10 01:06:13 +00:00
Glyn Normington	2c88909be3	Fix formatting	2024-11-10 01:02:18 +00:00
Glyn Normington	9295b6a963	Clarify our rationale ... I deleted the point about excessive load on crawled sites as any other crawler could potentially be guilty of this and I wouldn't want our scope to creep to all crawlers. Ref: https://github.com/ai-robots-txt/ai.robots.txt/issues/53#issuecomment-2466042550	2024-11-09 04:45:47 +00:00
Glyn Normington	e6bb7cae9e	Augment the "why" FAQ ... Ref: https://github.com/ai-robots-txt/ai.robots.txt/issues/40#issuecomment-2419078796	2024-10-17 12:27:05 +01:00
Glyn Normington	b229f5b936	Re-order the FAQ ... The "why" question should come first.	2024-10-17 12:25:54 +01:00
Malte Ubl	809851ae88	Add instructions for AI bot blocking on Vercel	2024-09-07 15:59:25 -07:00
Cory Dransfeldt	bc66d10afd	chore: update faq	2024-08-14 09:21:26 -07:00
Glyn Normington	b4d25bf0cb	Add FAQ	2024-08-06 17:20:26 +01:00
Joshua Sheard	8dbbdbf44c	Add Cloudflares first-party scraper blocking to FAQ	2024-08-04 21:38:02 +01:00
nisbet-hubbard	2b56c72bac	Update FAQ.md	2024-08-03 14:27:25 +08:00
nisbet-hubbard	b24e5cb3bb	Update FAQ.md	2024-08-03 14:12:50 +08:00
nisbet-hubbard	74b1502839	Update FAQ.md	2024-08-03 14:04:58 +08:00
Cory Dransfeldt	6c596a50ea	chore: move FAQ into repo	2024-08-01 07:53:43 -07:00