Claude-Code
Claude-Code is blocked by 256 sites in the October 2026 survey, or 4.1% of readable files. Its operator is unclear in the community reference.
Who runs Claude-Code?
| Fact | Claude-Code reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Coding Agents” |
| Does Claude-Code obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Claude-Code | No |
These facts quote the stored community ai.robots.txt reference.
“Claude Code is an AI coding agent by Anthropic that can build, debug, and ship code directly from the terminal, handling tasks like codebase onboarding, multi-file edits,…”
The listed coding role concerns tools that help with software work. A requested page visit can answer a person now, whereas training collects material for a model. Owners may apply different access rules.
How many sites block it?
The 256 files blocking the whole site account for 4.1% of 6,171 readable robots.txt files. We found its name in 28 User-agent lists, a 0.5% share. Without a whole-site Disallow: / block, 5,915 files remain.
From the highest 1,000 ranks, 552 files could be read. Its naming count is 3, or 0.5%, and the block count is 43. Blocking is more common among the leading sites, at 7.8%. Compare 213 blocks among the 5,619 readable files below those ranks, or 3.8%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Claude-Code, all ranks | 28 | 0.5% | 256 | 4.1% |
| Claude-Code, top 1,000 | 3 | 0.5% | 43 | 7.8% |
| Claude-Code, remaining ranks | 25 | 0.4% | 213 | 3.8% |
Counts include a whole-site ban in a User-agent: * group if no named group overrides it. Of 256 blocks, 236 come only from that general rule, without naming this crawler. Names and bans overlap in 20 files. We also found 8 files with a name and no whole-site ban, plus 6,143 files omitting the name. Case changes and repeated spellings are merged.
Has that changed since August?
Between August and October, 16 sites that had blocked it stopped and 17 started, so 1 more of the same 5,889 sites block it. A further 208 blocked it in both months, and 5,648 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 244 blocks among 6,236 readable files. October’s was 256 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 17 |
| Stopped blocking | 16 |
| Blocked in both | 208 |
| Blocked in neither | 5,648 |
| Net change | +1 |
How to block or allow it
Place the group in your root robots.txt file. Per the robots.txt standard, a named group for Claude-Code overrides User-agent: *.
Block the whole site
User-agent: Claude-Code
Disallow: /Allow access
User-agent: Claude-Code
Allow: /It's the 141st most-named crawler of the 171 we counted in the crawler index.
How we counted
For Claude-Code, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. Requests could run for 8 seconds, with a 262,144-byte file limit. Shares exclude unreachable, unavailable and oversized files. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
The 28 files naming it include 8 files without a whole-site ban.
How do the shares compare by rank?
Within readable robots.txt files near the top of the ranking, 0.5% name it and 7.8% block it. Overall, naming reaches 0.5% and blocking 4.1%.
What other entry is close in the index?
For comparison, Diffbot-User has 28 files naming it and 513 blocks, or 8.3%. Our counts for this entry are 28 files with the name and 256 blocking the whole site, or 4.1%.