ShapBot
ShapBot is blocked by 120 sites in the October 2026 survey, or 1.9% of readable files. The community reference lists its operator as Parallel.
Who runs ShapBot?
| Fact | ShapBot reference entry |
|---|---|
| Operator | “Parallel” |
| Purpose | “AI Data Providers” |
| Does ShapBot obey robots.txt? | “Yes” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for ShapBot | No |
These facts quote the stored community ai.robots.txt reference.
“ShapBot is a web crawler by Parallel that collects and structures web content to power its search, extraction, and deep research APIs, providing AI agents with high-accur…”
Its listed role is supplying data for AI tools. Search indexing is aimed at helping people find pages. Feeding text into model training is another use an owner may decline.
How many sites block it?
A User-agent line names it in 123 files, 2.0% of 6,171 readable files. Whole-site bans, written Disallow: /, total 120, or 1.9%. That leaves 6,051 without a recorded ban.
From the highest 1,000 ranks, 552 files could be read. Its naming count is 28, or 5.1%, and the block count is 27. Blocking is more common among the leading sites, at 4.9%. Compare 93 blocks among the 5,619 readable files below those ranks, or 1.7%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| ShapBot, all ranks | 123 | 2.0% | 120 | 1.9% |
| ShapBot, top 1,000 | 28 | 5.1% | 27 | 4.9% |
| ShapBot, remaining ranks | 95 | 1.7% | 93 | 1.7% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. Of 120 blocks, 1 does not also appear in our count of files naming it. We found 119 files naming it and blocking the whole site, along with 4 files naming it without that ban. Another 6,048 files omit the name. We ignore capitalization and repeated names.
How to block or allow it
Put these in robots.txt at the site root. Per the robots.txt standard, a group naming ShapBot takes precedence over User-agent: *.
Block the whole site
User-agent: ShapBot
Disallow: /Allow access
User-agent: ShapBot
Allow: /It's the 53rd most-named crawler of the 171 we counted in the crawler index.
How we counted
For ShapBot, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. Each file had 8 seconds to arrive, up to 262,144 bytes. Missing, unreachable and oversized files do not contribute to these shares. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
This survey includes 4 files with the name and no whole-site ban, out of 123 naming files.
What do the top-ranked sites show?
At the top of the ranking, 5.1% of readable files name it and 4.9% block it. Across all ranks, those shares are 2.0% and 1.9%.
Which nearby entry can I compare?
The nearby ChatGLM-Spider entry reports 117 files that name it and 123 whole-site blocks, 2.0% of readable files. Here, 123 files mention the name and 120 block access, or 1.9%.