TerraCotta
TerraCotta is blocked by 70 sites in the October 2026 survey, or 1.1% of readable files. The community reference lists its operator as Ceramic AI.
Who runs TerraCotta?
| Fact | TerraCotta reference entry |
|---|---|
| Operator | “Unclear at this time.” / “Ceramic AI” |
| Purpose | “AI Data Providers” |
| Does TerraCotta obey robots.txt? | “Unclear at this time.” / “Yes” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for TerraCotta | No |
These facts quote the stored community ai.robots.txt reference.
“TerraCotta is Ceramic's web crawler that indexes public content to power their web-scale search API for AI and LLMs.”
Its listed role is supplying data for AI tools. Search referrals or a user’s question can give an owner a reason to allow access while declining collection of page text for training.
How many sites block it?
Within 6,171 readable robots.txt files, the name appears in 73, or 1.2%. Instructions blocking every path, Disallow: /, appear in 70, a 1.1% share. No whole-site ban is recorded for 6,101 files.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 21 files carry the name, or 3.8%, with 19 files blocking the whole site. Blocking is more common among the leading sites, at 3.4%. Farther down, 51 of 5,619 files record a block, 0.9% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| TerraCotta, all ranks | 73 | 1.2% | 70 | 1.1% |
| TerraCotta, top 1,000 | 21 | 3.8% | 19 | 3.4% |
| TerraCotta, remaining ranks | 52 | 0.9% | 51 | 0.9% |
Only whole-site blocks recorded against this crawler’s name count here. Rules addressed to every crawler are outside this count. The counted overlap is 70 files, with 3 files naming it without banning all paths. Missing names account for 6,098 files. Repeating a name or changing its capitalization does not increase the naming total.
How to block or allow it
The examples include the spelling variants counted together here. Place the group in your root robots.txt file. Per the robots.txt standard, a named group for TerraCotta overrides User-agent: *.
Block the whole site
User-agent: Terra Cotta
User-agent: TerraCotta
Disallow: /Allow access
User-agent: Terra Cotta
User-agent: TerraCotta
Allow: /It's the 80th most-named crawler of the 171 we counted in the crawler index.
How we counted
For TerraCotta, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. Files had an 8-second timeout and a 262,144-byte limit. Missing, unreachable and oversized files do not contribute to these shares. Collection and repeat readings used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
We found 3 files without a whole-site ban among 73 that name it.
How do the shares compare by rank?
Within readable robots.txt files near the top of the ranking, 3.8% name it and 3.4% block it. Overall, naming reaches 1.2% and blocking 1.1%.
What other entry is close in the index?
The nearby laion-huggingface-processor entry reports 72 files that name it and 93 whole-site blocks, 1.5% of readable files. Here, 73 files mention the name and 70 block access, or 1.1%.