Google-CloudVertexBot
Google-CloudVertexBot is blocked by 462 sites in the October 2026 survey, or 7.5% of readable files. The community reference lists its operator as Google.
Who runs Google-CloudVertexBot?
| Fact | Google-CloudVertexBot reference entry |
|---|---|
| Operator | “Google” |
| Purpose | “Build and manage AI models for businesses employing Vertex AI” |
| Does Google-CloudVertexBot obey robots.txt? | “Yes” |
| Frequency | “No information.” |
| Cloudflare managed list for Google-CloudVertexBot | No |
These facts quote the stored community ai.robots.txt reference.
“Google-CloudVertexBot crawls sites on the site owners' request when building Vertex AI Agents.”
This entry’s stated use is quoted above. A site can welcome a page fetch prompted by a person while declining the collection of that text to build future models.
How many sites block it?
Of 6,171 robots.txt files we could read, 462 block the whole site with Disallow: /, or 7.5%. Another way to read the sample is by name, with 275 files naming it in a User-agent line, or 4.5%. The 5,709 remaining files have no recorded whole-site block.
Breaking out the leading 1,000 sites gives 552 readable files, with 43 naming the crawler, or 7.8%. Whole-site blocking appears in 72 of them. Blocking is more common among the leading sites, at 13.0%. Beyond those ranks, 5,619 files have a blocking total of 390, 6.9% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Google-CloudVertexBot, all ranks | 275 | 4.5% | 462 | 7.5% |
| Google-CloudVertexBot, top 1,000 | 43 | 7.8% | 72 | 13.0% |
| Google-CloudVertexBot, remaining ranks | 232 | 4.1% | 390 | 6.9% |
The survey includes relevant User-agent: * instructions alongside instructions addressed to this crawler. Of 462 blocks, 222 come only from that general rule, without naming this crawler. The counted overlap is 240 files, with 35 files naming it without banning all paths. Missing names account for 5,896 files. Repeating a name or changing its capitalization does not increase the naming total.
Has that changed since August?
Between August and October, 18 sites that had blocked it stopped and 72 started, so 54 more of the same 5,889 sites block it. A further 353 blocked it in both months, and 5,446 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 395 blocks among 6,236 readable files. October’s was 462 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 72 |
| Stopped blocking | 18 |
| Blocked in both | 353 |
| Blocked in neither | 5,446 |
| Net change | +54 |
How to block or allow it
Place the group in your root robots.txt file. Per the robots.txt standard, a named group for Google-CloudVertexBot overrides User-agent: *.
Block the whole site
User-agent: Google-CloudVertexBot
Disallow: /Allow access
User-agent: Google-CloudVertexBot
Allow: /It's the 33rd most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For Google-CloudVertexBot, our 19 October 2026 collection used Tranco’s top 10,000 sites. Each file had 8 seconds to arrive, up to 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. Repeat readings are included, using scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
The 275 files that mention its name do not all record a whole-site ban. Those without one total 35.
What do the top-ranked sites show?
At the top of the ranking, 7.8% of readable files name it and 13.0% block it. Across all ranks, those shares are 4.5% and 7.5%.
Which nearby entry can I compare?
For comparison, cohere-training-data-crawler has 250 files naming it and 246 blocks, or 4.0%. Our counts for this entry are 275 files with the name and 462 blocking the whole site, or 7.5%.