Claude-Web
Claude-Web is blocked by 619 sites in the October 2026 survey, or 10.0% of readable files. The community reference lists its operator as Anthropic.
Who runs Claude-Web?
| Fact | Claude-Web reference entry |
|---|---|
| Operator | “Anthropic” |
| Purpose | “Undocumented AI Agents” |
| Does Claude-Web obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Claude-Web | No |
These facts quote the stored community ai.robots.txt reference.
“Claude-Web is an AI-related agent operated by Anthropic. It's currently unclear exactly what it's used for, since there's no official documentation. If you can provide more detail, please contact us.”
The reference leaves this agent’s purpose undocumented. Keeping a site available to readers may include allowing search and requested fetches. An owner may still keep the text out of training collections.
How many sites block it?
Our readable sample has 619 whole-site blocks, 10.0% of 6,171 files, with Disallow: / as the instruction. Of that full sample, 511 name this crawler, or 8.3%. The count without a whole-site block is 5,552.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 64 files carry the name, or 11.6%, with 87 files blocking the whole site. Blocking is more common among the leading sites, at 15.8%. Farther down, 532 of 5,619 files record a block, 9.5% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Claude-Web, all ranks | 511 | 8.3% | 619 | 10.0% |
| Claude-Web, top 1,000 | 64 | 11.6% | 87 | 15.8% |
| Claude-Web, remaining ranks | 447 | 8.0% | 532 | 9.5% |
Rules for every crawler, written User-agent: *, count here too. Of 619 blocks, 215 come only from that general rule, without naming this crawler. We found 404 files naming it and blocking the whole site, along with 107 files naming it without that ban. Another 5,660 files omit the name. We ignore capitalization and repeated names.
Has that changed since August?
Between August and October, 36 sites that had blocked it stopped and 68 started, so 32 more of the same 5,889 sites block it. A further 508 blocked it in both months, and 5,277 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 572 blocks among 6,236 readable files. October’s was 619 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 68 |
| Stopped blocking | 36 |
| Blocked in both | 508 |
| Blocked in neither | 5,277 |
| Net change | +32 |
How to block or allow it
Add these lines to robots.txt at your site root. For Claude-Web, the robots.txt standard gives a named group priority over User-agent: *.
Block the whole site
User-agent: Claude-Web
Disallow: /Allow access
User-agent: Claude-Web
Allow: /It's the 16th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For Claude-Web, our 19 October 2026 collection used Tranco’s top 10,000 sites. Requests could run for 8 seconds, with a 262,144-byte file limit. Missing, unreachable and oversized files do not contribute to these shares. scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) parsed both initial and repeat readings. See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
We found 107 files without a whole-site ban among 511 that name it.
What do the top-ranked sites show?
Leading files record 11.6% with the name and 15.8% blocking it. Across all readable files, the corresponding shares are 8.3% and 10.0%.
Which nearby entry can I compare?
The nearby PetalBot entry reports 487 files that name it and 433 whole-site blocks, 7.0% of readable files. Here, 511 files mention the name and 619 block access, or 10.0%.