Crawlspace
Few readable files block Crawlspace in October 2026, at 58 sites or 0.9%. The community reference lists its operator as Crawlspace.
Who runs Crawlspace?
| Fact | Crawlspace reference entry |
|---|---|
| Operator | “Crawlspace” |
| Purpose | “AI Data Providers” |
| Does Crawlspace obey robots.txt? | “Yes” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Crawlspace | No |
These facts quote the stored community ai.robots.txt reference.
“Crawlspace is a web crawler platform that fetches and extracts website content for AI agents, RAG applications, and structured data workflows.”
Its listed role is supplying data for AI tools. An owner may want a page included in search and available on request without offering its contents for future model development.
How many sites block it?
A User-agent line names it in 61 files, 1.0% of 6,171 readable files. Whole-site bans, written Disallow: /, total 58, or 0.9%. That leaves 6,113 without a recorded ban.
From the highest 1,000 ranks, 552 files could be read. Its naming count is 7, or 1.3%, and the block count is 7. Blocking is more common among the leading sites, at 1.3%. Compare 51 blocks among the 5,619 readable files below those ranks, or 0.9%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Crawlspace, all ranks | 61 | 1.0% | 58 | 0.9% |
| Crawlspace, top 1,000 | 7 | 1.3% | 7 | 1.3% |
| Crawlspace, remaining ranks | 54 | 1.0% | 51 | 0.9% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. Looking just at named files gives 58 blocks and 3 without a whole-site block. For 6,110 readable files, we found no mention by name. We ignore capitalization and repeated names.
How to block or allow it
Edit the root robots.txt file on your site. The robots.txt standard gives a Crawlspace group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: Crawlspace
Disallow: /Allow access
User-agent: Crawlspace
Allow: /It's the 102nd most-named crawler of the 171 we counted in the crawler index.
How we counted
For Crawlspace, we surveyed Tranco’s top 10,000 sites on 19 October 2026. An 8-second wait or a file over 262,144 bytes ended a read. Shares exclude unreachable, unavailable and oversized files. scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) parsed both initial and repeat readings. See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
We found 3 files without a whole-site ban among 61 that name it.
What do the top-ranked sites show?
Within readable robots.txt files near the top of the ranking, 1.3% name it and 1.3% block it. Overall, naming reaches 1.0% and blocking 0.9%.
Which nearby entry can I compare?
For comparison, Shap-User has 61 files naming it and 60 blocks, or 1.0%. Our counts for this entry are 61 files with the name and 58 blocking the whole site, or 0.9%.