Google-Extended
Google-Extended collects content for model training, and the community reference lists its operator as Google. Our October 2026 survey found 748 sites blocking it, or 12.1% of the files we could read.
Who runs Google-Extended?
| Fact | Google-Extended reference entry |
|---|---|
| Operator | “Google” |
| Purpose | “LLM training.” |
| Does Google-Extended obey robots.txt? | “Yes” |
| Frequency | “No information.” |
| Cloudflare managed list for Google-Extended | Yes |
These facts quote the stored community ai.robots.txt reference.
“Used to train Gemini and Vertex AI generative APIs. Does not impact a site's inclusion or ranking in Google Search.”
Model training is this crawler’s listed use. An owner may permit an assistant to read a page for someone and deny bulk collection that would use it as training material.
How many sites block it?
A User-agent line names it in 875 files, 14.2% of 6,171 readable files. Whole-site bans, written Disallow: /, total 748, or 12.1%. That leaves 5,423 without a recorded ban.
For the top 1,000 ranks, its name appears in 117 files out of 552, or 21.2%. Whole-site blocks total 96. Blocking is more common among the leading sites, at 17.4%. Lower in the ranking, blocking reaches 652 files out of 5,619, a 11.6% share.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Google-Extended, all ranks | 875 | 14.2% | 748 | 12.1% |
| Google-Extended, top 1,000 | 117 | 21.2% | 96 | 17.4% |
| Google-Extended, remaining ranks | 758 | 13.5% | 652 | 11.6% |
A general User-agent: * ban contributes to this total whenever it applies to this crawler. Of 748 blocks, 181 come only from that general rule, without naming this crawler. The counted overlap is 567 files, with 308 files naming it without banning all paths. Missing names account for 5,296 files. Repeating a name or changing its capitalization does not increase the naming total.
Has that changed since August?
Between August and October, 176 sites that had blocked it stopped and 55 started, so 121 fewer of the same 5,889 sites block it. A further 647 blocked it in both months, and 5,011 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 933 blocks among 6,236 readable files. October’s was 748 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 55 |
| Stopped blocking | 176 |
| Blocked in both | 647 |
| Blocked in neither | 5,011 |
| Net change | -121 |
How to block or allow it
Add these lines to robots.txt at your site root. For Google-Extended, the robots.txt standard gives a named group priority over User-agent: *.
Block the whole site
User-agent: Google-Extended
Disallow: /Allow access
User-agent: Google-Extended
Allow: /It's the 4th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
- Google-CloudVertexBot
- Google-Firebase
- GoogleAgent-Mariner
- GoogleOther
- GoogleOther-Image
- GoogleOther-Video
How we counted
For Google-Extended, we surveyed Tranco’s top 10,000 sites on 19 October 2026. Requests could run for 8 seconds, with a 262,144-byte file limit. Missing, unreachable and oversized files do not contribute to these shares. Later readings are included in results from scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
The 875 files naming it include 308 files without a whole-site ban.
How do the shares compare by rank?
At the top of the ranking, 21.2% of readable files name it and 17.4% block it. Across all ranks, those shares are 14.2% and 12.1%.
What other entry is close in the index?
The nearby Bytespider entry reports 826 files that name it and 930 whole-site blocks, 15.1% of readable files. Here, 875 files mention the name and 748 block access, or 12.1%.