What Is Google-Extended?
Google-Extended is a user-agent token that allows website owners to control whether their content is used to train and improve Google's AI products — including Gemini, Google AI Mode, and other AI services — independently of standard Google search crawling.
Prior to Google-Extended, website owners had a binary choice: allow Googlebot (allowing both search indexing and AI training) or block Googlebot (blocking both). Google-Extended allows a middle path: opt out of AI training while remaining indexed in Google search.
How Google-Extended Works
Google-Extended is set in your robots.txt file:
Allow Google search indexing AND AI training (default):
# No Google-Extended entry needed — this is the default
Allow Google search indexing, but opt out of AI training:
User-agent: Google-Extended
Disallow: /
Allow AI training on specific directories:
User-agent: Google-Extended
Disallow: /
Allow: /public-content/
Impact on AI Visibility
Blocking Google-Extended has implications for your AI visibility:
- Positive: Your content won't be used to train Google's AI without your permission
- Negative: Content blocked from Google-Extended may not appear in Google AI Mode responses, AI Overviews, or Gemini recommendations as readily
For most businesses trying to improve AI visibility, allowing Google-Extended is recommended. The AI training opt-out matters more for publishers, authors, and creators concerned about copyright and revenue sharing.
Google-Extended vs. GPTBot vs. Other AI Crawlers
| Crawler | Company | Controls | |---|---|---| | Google-Extended | Google | Google's AI training and AI services (Gemini, AI Mode) | | GPTBot | OpenAI | ChatGPT training data | | anthropic-ai | Anthropic | Claude training data | | PerplexityBot | Perplexity | Perplexity real-time retrieval | | CCBot | Common Crawl | Various AI datasets |
Q: Does blocking Google-Extended affect my Google search rankings? A: No. Google-Extended is separate from standard Googlebot. Blocking Google-Extended does not affect how your pages rank in traditional Google search results — only how Google's AI products use your content.
Q: How do I know if Google-Extended is crawling my site?
A: Check your web server access logs for the user agent string Google-Extended. You can also review crawl statistics in Google Search Console under Settings → Crawl Stats.