github/ai.robots.txt
1
0
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-12-29 12:18:33 +01:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #168 from ai-robots-txt/google-firebase-shapbot

Browse source 
chore: add Google-Firebase and ShapBot
This commit is contained in:
ai.robots.txt 2025-08-27 10:13:17 +00:00
commit 2ad1c3e831
6 changed files with 10 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

2
robots.txt
View file

@ -38,6 +38,7 @@ User-agent: FriendlyCrawler
User-agent: Gemini-Deep-Research
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: Google-Firebase
User-agent: GoogleAgent-Mariner
User-agent: GoogleOther
User-agent: GoogleOther-Image
@ -79,6 +80,7 @@ User-agent: SBIntuitionsBot
User-agent: Scrapy
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA
User-agent: ShapBot
User-agent: Sidetrade indexer bot
User-agent: Thinkbot
User-agent: TikTokSpider