Cloudflare-AutoRAG
Cloudflare-AutoRAG is listed as a search crawler, and the community reference lists its operator as Cloudflare. In October 2026, 52 sites block it, a 0.8% share of readable files.
Who runs Cloudflare-AutoRAG?
| Fact | Cloudflare-AutoRAG reference entry |
|---|---|
| Operator | “Cloudflare” |
| Purpose | “Collects data for AI search” |
| Does Cloudflare-AutoRAG obey robots.txt? | “Yes” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Cloudflare-AutoRAG | No |
These facts quote the stored community ai.robots.txt reference.
“AutoRAG is an all-in-one AI search solution.”
The listed use here is search. An owner may want a page included in search and available on request without offering its contents for future model development.
How many sites block it?
The name occurs in 54 out of 6,171 readable files, a 0.9% share. Across the full sample, the whole-site Disallow: / block count is 52, or 0.8%. Files without that ban number 6,119.
Among 552 readable files from the top 1,000 sites, we found 19 files naming it, a 3.4% share, with 18 files blocking the whole site. Blocking is more common among the leading sites, at 3.3%. Outside this group, 5,619 files contribute 34 blocks, or 0.6%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Cloudflare-AutoRAG, all ranks | 54 | 0.9% | 52 | 0.8% |
| Cloudflare-AutoRAG, top 1,000 | 19 | 3.4% | 18 | 3.3% |
| Cloudflare-AutoRAG, remaining ranks | 35 | 0.6% | 34 | 0.6% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. Of 52 blocks, 1 does not also appear in our count of files naming it. The naming total includes 51 files with a ban on every path and 3 files without that ban. Elsewhere, 6,117 files do not name it. We ignore capitalization and repeated names.
How to block or allow it
Use your site’s root file, robots.txt. Specific groups for Cloudflare-AutoRAG take precedence over User-agent: *, according to the robots.txt standard.
Block the whole site
User-agent: Cloudflare-AutoRAG
Disallow: /Allow access
User-agent: Cloudflare-AutoRAG
Allow: /It's the 115th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For Cloudflare-AutoRAG, our 19 October 2026 collection used Tranco’s top 10,000 sites. Requests could run for 8 seconds, with a 262,144-byte file limit. Missing, unreachable and oversized files do not contribute to these shares. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
The 54 files naming it include 3 files without a whole-site ban.
How do the shares compare by rank?
Leading files record 3.4% with the name and 3.3% blocking it. Across all readable files, the corresponding shares are 0.9% and 0.8%.
What other entry is close in the index?
The nearby Spider entry reports 52 files that name it and 50 whole-site blocks, 0.8% of readable files. Here, 54 files mention the name and 52 block access, or 0.8%.