CCBot
CCBot is blocked by 945 sites in the October 2026 survey, or 15.3% of readable files. The community reference lists its operator as Common Crawl Foundation.
Who runs CCBot?
| Fact | CCBot reference entry |
|---|---|
| Operator | “Common Crawl Foundation” |
| Purpose | “Provides open crawl dataset, used for many purposes, including Machine Learning/AI.” |
| Does CCBot obey robots.txt? | “Yes” |
| Frequency | “Monthly at present.” |
| Cloudflare managed list for CCBot | Yes |
These facts quote the stored community ai.robots.txt reference.
“Web archive going back to 2008. Cited in thousands of research papers per year.”
This entry’s stated use is quoted above. Keeping a site available to readers may include allowing search and requested fetches. An owner may still keep the text out of training collections.
How many sites block it?
Our readable sample has 945 whole-site blocks, 15.3% of 6,171 files, with Disallow: / as the instruction. Of that full sample, 934 name this crawler, or 15.1%. The count without a whole-site block is 5,226.
At the top 1,000 ranks, we could read 552 files. Naming covers 121, a 21.9% share, and blocking covers 124. Blocking is more common among the leading sites, at 22.5%. Elsewhere, 5,619 readable files give 821 blocks, or 14.6%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| CCBot, all ranks | 934 | 15.1% | 945 | 15.3% |
| CCBot, top 1,000 | 121 | 21.9% | 124 | 22.5% |
| CCBot, remaining ranks | 813 | 14.5% | 821 | 14.6% |
Counts include a whole-site ban in a User-agent: * group if no named group overrides it. Of 945 blocks, 199 come only from that general rule, without naming this crawler. The overlap between naming and blocking is 746 files. Naming without a whole-site ban accounts for 188, and 5,237 files never name it. We ignore capitalization and repeated names.
Has that changed since August?
Between August and October, 190 sites that had blocked it stopped and 51 started, so 139 fewer of the same 5,889 sites block it. A further 841 blocked it in both months, and 4,807 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 1,147 blocks among 6,236 readable files. October’s was 945 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 51 |
| Stopped blocking | 190 |
| Blocked in both | 841 |
| Blocked in neither | 4,807 |
| Net change | -139 |
How to block or allow it
Place the group in your root robots.txt file. Per the robots.txt standard, a named group for CCBot overrides User-agent: *.
Block the whole site
User-agent: CCBot
Disallow: /Allow access
User-agent: CCBot
Allow: /It's the 3rd most-named crawler of the 171 we counted in the crawler index.
How we counted
For CCBot, we surveyed Tranco’s top 10,000 sites on 19 October 2026. Each file had 8 seconds to arrive, up to 262,144 bytes. Missing, unreachable and oversized files do not contribute to these shares. Collection and repeat readings used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
A name identifies the crawler addressed by a group. The survey found 188 files with that name and no recorded whole-site ban, out of 934.
Are leading sites different?
Its name appears in 21.9% of readable leading files and 15.1% overall. Blocking affects 22.5% in the leading group and 15.3% in all ranks.
How does the nearby crawler compare?
The nearby Google-Extended entry reports 875 files that name it and 748 whole-site blocks, 12.1% of readable files. Here, 934 files mention the name and 945 block access, or 15.3%.