CloudVertexBot
Few readable files block CloudVertexBot in October 2026, at 57 sites or 0.9%. Its operator is unclear in the community reference.
Who runs CloudVertexBot?
| Fact | CloudVertexBot reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Data Scrapers” |
| Does CloudVertexBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for CloudVertexBot | No |
These facts quote the stored community ai.robots.txt reference.
“CloudVertexBot is a Google-operated crawler available to site owners to request targeted crawls of their own sites for AI training purposes on the Vertex AI platform.”
This entry is classed as a crawler collecting data for AI. Reading a page to answer a user has a different purpose from building a model with it. Owners may allow the requested visit.
How many sites block it?
A User-agent line names it in 58 files, 0.9% of 6,171 readable files. Whole-site bans, written Disallow: /, total 57, or 0.9%. That leaves 6,114 without a recorded ban.
Breaking out the leading 1,000 sites gives 552 readable files, with 4 naming the crawler, or 0.7%. Whole-site blocking appears in 4 of them. The leading sites have a lower block share, 0.7%. Beyond those ranks, 5,619 files have a blocking total of 53, 0.9% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| CloudVertexBot, all ranks | 58 | 0.9% | 57 | 0.9% |
| CloudVertexBot, top 1,000 | 4 | 0.7% | 4 | 0.7% |
| CloudVertexBot, remaining ranks | 54 | 1.0% | 53 | 0.9% |
This is a count of blocks attached to the named crawler, covering the whole site. Blanket rules for every crawler do not enter it. Of 57 blocks, 3 do not also appear in our count of files naming it. The split among files that name it is 54 with a whole-site block and 4 without one. Its name is absent from 6,113 readable files. We ignore capitalization and repeated names.
How to block or allow it
Put these in robots.txt at the site root. Per the robots.txt standard, a group naming CloudVertexBot takes precedence over User-agent: *.
Block the whole site
User-agent: CloudVertexBot
Disallow: /Allow access
User-agent: CloudVertexBot
Allow: /It's the 107th most-named crawler of the 171 we counted in the crawler index.
How we counted
For CloudVertexBot, we surveyed Tranco’s top 10,000 sites on 19 October 2026. Files had an 8-second timeout and a 262,144-byte limit. Only available files we could read enter the percentages. Later readings are included in results from scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
The 58 files naming it include 4 files without a whole-site ban.
How do the shares compare by rank?
For naming rules, 0.7% of the leading sample mentions it, compared with 0.9% overall. Whole-site blocking is 0.7% at leading ranks and 0.9% across the full sample.
What other entry is close in the index?
Poggio-Citations is named by 58 files and blocked by 57, or 0.9%. For this crawler, 58 files name it and 57 block it, a 0.9% share.