remote/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2026-08-31 05:14:06 +02:00
Code Issues Projects Releases Packages Wiki Activity
707 commits 7 branches 53 tags 2 MiB
Commit graph

165 commits

Author SHA1 Message Date
MLuc24
024c9c2d6b Add ExaSearchBot to robots.json 2026-08-17 10:30:41 +07:00
Anthony M. Cook
a5d17db8b8
 		add Lightpanda user agent 
The Lightpanda browser is an AI-centric headless browser used for scraping and automation.

It is currently being used by a botnet of millions of IP addresses, including over 20k SpaceX IPs, to scrape sites such as the official Haskell Gitlab:
https://github.com/lightpanda-io/browser/issues/3156
 2026-08-07 11:22:16 -05:00
Özden und Julia
2eb9315743 Fill "Unclear at this time." from primary operator docs for 6 entries 
Re-submission of #248, which was merged then reverted in 80c19fc. The revert was
caused by main.yml's commit-message handling, not by this data — that is fixed
separately in #254.

Six entries move from "Unclear at this time." to the operator's own documented
value, each citing the primary source inline:

  Applebot              Yes   support.apple.com/en-us/119829#retrieval
  Meta-ExternalAgent    Yes   developers.facebook.com/docs/sharing/webmasters/web-crawlers/
  Meta-ExternalFetcher  No    (same Meta page — user-initiated fetch, documented
  meta-externalfetcher  No     as not checking robots.txt)
  DuckAssistBot         Yes   duckduckgo.com/duckduckgo-help-pages/results/duckassistbot/
  Google-Agent          Yes   developers.google.com/search/docs/crawling-indexing/google-common-crawlers#google-agent

table-of-bot-metrics.md is regenerated and included, so `robots.py --convert` has
nothing left to write and the workflow's commit step is skipped entirely.

Verified: code/tests.py 13/13 · robots.py --convert exits 0 and is idempotent ·
robots.txt, .htaccess, the nginx/lighttpd/haproxy configs and the Caddyfile are
byte-identical · the JSON diff touches only the six `respect` fields, nothing else.
 2026-08-04 09:11:42 +02:00
Glyn Normington
69a87566de Delete IbouBot 
Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/250
 2026-07-31 13:04:31 +01:00
Glyn Normington
80c19fcad5 Revert "Fill operator and respect from primary docs for 6 entries" 
This reverts commit e62a95bb8a.

Reverted because of CI failure noted in PR.
 2026-07-31 03:30:04 +01:00
Glyn Normington
bbe0579a23
 		Merge pull request #248 from INXPRNCD/fill-unclear-operator-and-respect-from-primary-docs 
Fill "Unclear at this time." from primary operator docs for 6 entries
 2026-07-31 03:20:58 +01:00
Known Agents
2155906baa Update from knownagents.com 2026-07-31 01:57:31 +00:00
Erdinc Özden
e62a95bb8a Fill operator and respect from primary docs for 6 entries 
Applebot, Meta-ExternalAgent, Meta-ExternalFetcher, meta-externalfetcher,
DuckAssistBot and Google-Agent carried 'Unclear at this time.' in both
operator and respect, but each operator documents the behaviour on its
own site. Values are linked to the primary source inline, matching the
style already used by Applebot-Extended and meta-externalagent.

robots.txt output is unchanged — no keys added or renamed.
 2026-07-30 22:05:49 +02:00
Dhruv Maniya
49669178ef fix: configure exact user-agent matching 
Signed-off-by: Dhruv Maniya <dhruvmaniya1998@gmail.com>
 2026-07-29 21:47:42 +05:30
Known Agents
a0fed45f0e Update from knownagents.com 2026-07-10 02:03:20 +00:00
Glyn Normington
51f7fd5e3a
 		Merge pull request #240 from bobmatyas/add/tabstack 
Add: Mozilla-Tabstack
 2026-07-09 16:00:59 +01:00
Known Agents
a71b27a232 Update from knownagents.com 2026-07-09 02:04:28 +00:00
bob
c793fa0a8c Add: Mozilla-Tabstack. Closes #226 2026-07-08 21:16:52 -04:00
Gavin King
dad7db5168 Fixed broken URLs 2026-07-08 20:38:48 -04:00
Gavin King
0c5de5b90b Updating the name 2026-07-08 15:30:25 -04:00
dark-visitors
a394130efd Update from Dark Visitors 2026-06-26 02:32:55 +00:00
dark-visitors
5cb3ec765f Update from Dark Visitors 2026-06-17 11:21:12 +00:00
dark-visitors
118bb772c0 Update from Dark Visitors 2026-06-17 11:01:38 +00:00
dark-visitors
3091ad0a23 Update from Dark Visitors 2026-06-16 02:52:42 +00:00
dark-visitors
f420408eee Update from Dark Visitors 2026-06-04 02:48:49 +00:00
Justin Ribeiro
7673437733
 		add: AgentTimes bot 2026-06-02 11:26:06 -07:00
dark-visitors
ee3a96ee5c Update from Dark Visitors 2026-05-22 02:33:30 +00:00
dark-visitors
aea7db9e34 Update from Dark Visitors 2026-05-13 02:24:20 +00:00
dark-visitors
851ce068cf Update from Dark Visitors 2026-05-03 02:01:44 +00:00
Georg Sauthoff
a9f686064f Add NagetBot and newsai user-agents 
User-agent header values seen in the wild over the last days:

    Mozilla/5.0 (compatible; NagetBot/1.0; +https://naget.ai/bot)
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 newsai/1.0 Safari/537.36
 2026-05-02 13:04:51 +02:00
dark-visitors
ad47cafc97 Update from Dark Visitors 2026-03-26 01:26:23 +00:00
dark-visitors
2d2ec4ce8d Update from Dark Visitors 2026-03-25 01:20:42 +00:00
dark-visitors
3de8865f00 Update from Dark Visitors 2026-03-06 01:21:01 +00:00
dark-visitors
a762015e58 Update from Dark Visitors 2026-02-13 01:23:17 +00:00
dark-visitors
83485effdb Update from Dark Visitors 2025-12-20 00:58:49 +00:00
dark-visitors
f1c752ef12 Update from Dark Visitors 2025-12-05 01:00:44 +00:00
dark-visitors
45b071b29f Update from Dark Visitors 2025-12-04 01:00:27 +00:00
dark-visitors
f61b3496f7 Update from Dark Visitors 2025-12-03 01:00:32 +00:00
dark-visitors
8363d4fdd4 Update from Dark Visitors 2025-12-02 01:25:24 +00:00
Adam Newbold
f5d7ccb243
 		Fixed invalid JSON 2025-11-28 14:10:38 -05:00
Adam Newbold
30d719a09a
 		Adding LAIONDownloader 2025-11-28 14:08:18 -05:00
Glyn Normington
ab24e41106 Clarify that DeepSeekBot does not respect robots.txt 
Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/198
 2025-11-26 12:02:42 +00:00
Glyn Normington
efb4d260da delete extraneous hyphen 
Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/193
 2025-11-20 10:23:48 +00:00
Glyn Normington
663d030f96
 		Merge pull request #192 from ai-robots-txt/cdransf/notebooklm-klaviyo 
chore: adds NotebookLM and KlaviyoAIBot agents
 2025-11-14 21:01:13 +00:00
Cory Dransfeldt
28b45ea08d
 		chore: adds NotebookLM and KlaviyoAIBot agents 2025-11-14 11:03:03 -08:00
Glyn Normington
60b6a0829d
 		Merge pull request #189 from ai-robots-txt/cdransf/atlassian-amazon-bots 
chore(robots.json): add AmazonBuyForMe and atlassian-bot
 2025-11-05 23:50:48 +00:00
Cory Dransfeldt
00bf2b0e13
 		chore(robots.json): add AmazonBuyForMe and atlassian-bot 2025-11-05 14:02:37 -08:00
Cory Dransfeldt
808451055c
 		chore: adds BuddyBot 2025-11-03 13:39:07 -08:00
Cory Dransfeldt
192b0a2eef
 		chore: adds Linguee Bot 2025-10-24 10:35:04 -07:00
dark-visitors
97e19445ce Update from Dark Visitors 2025-10-24 00:53:30 +00:00
Glyn Normington
b89a9eae6a Use third party in text, rather than first 2025-10-21 01:11:32 +01:00
Glyn Normington
d22b8dfd7b
 		Description is third party, not first 2025-10-21 01:08:42 +01:00
Cory Dransfeldt
19c1d346c3
 		chore: adds IbouBot 2025-10-20 09:28:31 -07:00
Glyn Normington
0874a92503
 		Merge pull request #183 from ai-robots-txt/cdransf/bot-additions-cloudflare 
chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot
 2025-10-20 15:34:30 +01:00
Cory Dransfeldt
28d2d09633
 		chore: add amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot 2025-10-19 15:57:28 -07:00