Google-Agent
Few readable files block Google-Agent in October 2026, at 19 sites or 0.3%. Its operator is unclear in the community reference.
Who runs Google-Agent?
| Fact | Google-Agent reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Agents” |
| Does Google-Agent obey robots.txt? | “Yes” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Google-Agent | No |
These facts quote the stored community ai.robots.txt reference.
“Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.”
The reference places this crawler among AI agents. A requested page visit can answer a person now, whereas training collects material for a model. Owners may apply different access rules.
How many sites block it?
The name occurs in 30 out of 6,171 readable files, a 0.5% share. Across the full sample, the whole-site Disallow: / block count is 19, or 0.3%. Files without that ban number 6,152.
Breaking out the leading 1,000 sites gives 552 readable files, with 10 naming the crawler, or 1.8%. Whole-site blocking appears in 5 of them. Blocking is more common among the leading sites, at 0.9%. Beyond those ranks, 5,619 files have a blocking total of 14, 0.2% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Google-Agent, all ranks | 30 | 0.5% | 19 | 0.3% |
| Google-Agent, top 1,000 | 10 | 1.8% | 5 | 0.9% |
| Google-Agent, remaining ranks | 20 | 0.4% | 14 | 0.2% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. Of 19 blocks, 1 does not also appear in our count of files naming it. Looking just at named files gives 18 blocks and 12 without a whole-site block. For 6,141 readable files, we found no mention by name. We ignore capitalization and repeated names.
How to block or allow it
Add these lines to robots.txt at your site root. For Google-Agent, the robots.txt standard gives a named group priority over User-agent: *.
Block the whole site
User-agent: Google-Agent
Disallow: /Allow access
User-agent: Google-Agent
Allow: /It's the 136th most-named crawler of the 171 we counted in the crawler index.
How we counted
For Google-Agent, our 19 October 2026 collection used Tranco’s top 10,000 sites. An 8-second wait or a file over 262,144 bytes ended a read. Shares exclude unreachable, unavailable and oversized files. Reading files again used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) as well. See the full survey method.
Common questions
Can a file name it and still allow it?
A name identifies the crawler addressed by a group. The survey found 12 files with that name and no recorded whole-site ban, out of 30.
How do the shares compare by rank?
For naming rules, 1.8% of the leading sample mentions it, compared with 0.5% overall. Whole-site blocking is 0.9% at leading ranks and 0.3% across the full sample.
What other entry is close in the index?
For comparison, Kimi-User has 30 files naming it and 253 blocks, or 4.1%. Our counts for this entry are 30 files with the name and 19 blocking the whole site, or 0.3%.