github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #156 from nisbet-hubbard/patch-9

Browse source 
Update funtion and description for SemrushBot
This commit is contained in:
ai.robots.txt 2025-07-02 15:58:19 +00:00
commit 20a242d390
6 changed files with 11 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

6
haproxy-block-ai-bots.txt
View file

@ -31,6 +31,7 @@ facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
GoogleOther
@ -50,6 +51,7 @@ Meta-ExternalFetcher
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
@ -68,11 +70,7 @@ QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot
SemrushBot-BA
SemrushBot-CT
SemrushBot-OCOB
SemrushBot-SI
SemrushBot-SWA
Sidetrade indexer bot
TikTokSpider