anthropic-ai
anthropic-ai collects content for model training, and the community reference lists its operator as Anthropic. Our October 2026 survey found 718 sites blocking it, or 11.6% of the files we could read.
Who runs anthropic-ai?
| Fact | anthropic-ai reference entry |
|---|---|
| Operator | “Anthropic” |
| Purpose | “Scrapes data to train Anthropic's AI products.” |
| Does anthropic-ai obey robots.txt? | “Unclear at this time.” |
| Frequency | “No information provided.” |
| Cloudflare managed list for anthropic-ai | No |
These facts quote the stored community ai.robots.txt reference.
A matching description is used for 1 other entry. Counts below are specific to this crawler.
Model training is this crawler’s listed use. An owner may permit an assistant to read a page for someone and deny bulk collection that would use it as training material.
How many sites block it?
The 718 files blocking the whole site account for 11.6% of 6,171 readable robots.txt files. We found its name in 642 User-agent lists, a 10.4% share. Without a whole-site Disallow: / block, 5,453 files remain.
For the top 1,000 ranks, its name appears in 75 files out of 552, or 13.6%. Whole-site blocks total 87. Blocking is more common among the leading sites, at 15.8%. Lower in the ranking, blocking reaches 631 files out of 5,619, a 11.2% share.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| anthropic-ai, all ranks | 642 | 10.4% | 718 | 11.6% |
| anthropic-ai, top 1,000 | 75 | 13.6% | 87 | 15.8% |
| anthropic-ai, remaining ranks | 567 | 10.1% | 631 | 11.2% |
Counts include a whole-site ban in a User-agent: * group if no named group overrides it. Of 718 blocks, 214 come only from that general rule, without naming this crawler. The overlap between naming and blocking is 504 files. Naming without a whole-site ban accounts for 138, and 5,529 files never name it. We ignore capitalization and repeated names.
Has that changed since August?
Between August and October, 32 sites that had blocked it stopped and 82 started, so 50 more of the same 5,889 sites block it. A further 590 blocked it in both months, and 5,185 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 651 blocks among 6,236 readable files. October’s was 718 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 82 |
| Stopped blocking | 32 |
| Blocked in both | 590 |
| Blocked in neither | 5,185 |
| Net change | +50 |
How to block or allow it
Put these in robots.txt at the site root. Per the robots.txt standard, a group naming anthropic-ai takes precedence over User-agent: *.
Block the whole site
User-agent: anthropic-ai
Disallow: /Allow access
User-agent: anthropic-ai
Allow: /It's the 11th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For anthropic-ai, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. Each file had 8 seconds to arrive, up to 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) parsed both initial and repeat readings. See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
We found 138 files without a whole-site ban among 642 that name it.
What do the top-ranked sites show?
Leading files record 13.6% with the name and 15.8% blocking it. Across all readable files, the corresponding shares are 10.4% and 11.6%.
Which nearby entry can I compare?
For comparison, OAI-SearchBot has 608 files naming it and 427 blocks, or 6.9%. Our counts for this entry are 642 files with the name and 718 blocking the whole site, or 11.6%.