CloudflareBrowserRenderingCrawler
CloudflareBrowserRenderingCrawler is blocked by 68 sites in the October 2026 survey, or 1.1% of readable files. The community reference lists its operator as Cloudflare that returns rendered website content for research, monitoring, and AI data workflows.
Who runs CloudflareBrowserRenderingCrawler?
| Fact | CloudflareBrowserRenderingCrawler reference entry |
|---|---|
| Operator | “Cloudflare that returns rendered website content for research, monitoring, and AI data workflows” |
| Purpose | “AI Data Providers” |
| Does CloudflareBrowserRenderingCrawler obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for CloudflareBrowserRenderingCrawler | No |
These facts quote the stored community ai.robots.txt reference.
“CloudflareBrowserRenderingCrawler is a web crawler operated by Cloudflare that returns rendered website content for research, monitoring, and AI data workflows.”
Its listed role is supplying data for AI tools. Indexing helps readers find pages, and a user’s request can fetch one. Training reuses the material in models, which an owner may choose to block.
How many sites block it?
Our readable sample has 68 whole-site blocks, 1.1% of 6,171 files, with Disallow: / as the instruction. Of that full sample, 65 name this crawler, or 1.1%. The count without a whole-site block is 6,103.
From the highest 1,000 ranks, 552 files could be read. Its naming count is 22, or 4.0%, and the block count is 21. Blocking is more common among the leading sites, at 3.8%. Compare 47 blocks among the 5,619 readable files below those ranks, or 0.8%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| CloudflareBrowserRenderingCrawler, all ranks | 65 | 1.1% | 68 | 1.1% |
| CloudflareBrowserRenderingCrawler, top 1,000 | 22 | 4.0% | 21 | 3.8% |
| CloudflareBrowserRenderingCrawler, remaining ranks | 43 | 0.8% | 47 | 0.8% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. Of 68 blocks, 8 do not also appear in our count of files naming it. The overlap between naming and blocking is 60 files. Naming without a whole-site ban accounts for 5, and 6,106 files never name it. We ignore capitalization and repeated names.
How to block or allow it
Add these lines to robots.txt at your site root. For CloudflareBrowserRenderingCrawler, the robots.txt standard gives a named group priority over User-agent: *.
Block the whole site
User-agent: CloudflareBrowserRenderingCrawler
Disallow: /Allow access
User-agent: CloudflareBrowserRenderingCrawler
Allow: /It's the 94th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For CloudflareBrowserRenderingCrawler, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. Requests could run for 8 seconds, with a 262,144-byte file limit. Shares exclude unreachable, unavailable and oversized files. Reading files again used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) as well. See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
A name identifies the crawler addressed by a group. The survey found 5 files with that name and no recorded whole-site ban, out of 65.
What do the top-ranked sites show?
At the top of the ranking, 4.0% of readable files name it and 3.8% block it. Across all ranks, those shares are 1.1% and 1.1%.
Which nearby entry can I compare?
NovaAct is named by 65 files and blocked by 56, or 0.9%. For this crawler, 65 files name it and 68 block it, a 1.1% share.