github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #73 from nisbet-hubbard/patch-8

Browse source 
Actually block Semrush’s AI tools
This commit is contained in:
ai.robots.txt 2025-02-01 10:51:01 +00:00
commit 89d4c6e5ca
3 changed files with 5 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

3
robots.txt
View file

@ -36,7 +36,8 @@ User-agent: PanguBot
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: Scrapy
User-agent: SemrushBot
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA
User-agent: Sidetrade indexer bot
User-agent: Timpibot
User-agent: VelenPublicWebCrawler