Spider
Few readable files block Spider in October 2026, at 50 sites or 0.8%. Its operator is unclear in the community reference.
Who runs Spider?
| Fact | Spider reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Data Scrapers” |
| Does Spider obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Spider | No |
These facts quote the stored community ai.robots.txt reference.
This entry is classed as a crawler collecting data for AI. Keeping a site available to readers may include allowing search and requested fetches. An owner may still keep the text out of training collections.
How many sites block it?
The name occurs in 52 out of 6,171 readable files, a 0.8% share. Across the full sample, the whole-site Disallow: / block count is 50, or 0.8%. Files without that ban number 6,121.
Among 552 readable files from the top 1,000 sites, we found 4 files naming it, a 0.7% share, with 4 files blocking the whole site. The leading sites have a lower block share, 0.7%. Outside this group, 5,619 files contribute 46 blocks, or 0.8%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Spider, all ranks | 52 | 0.8% | 50 | 0.8% |
| Spider, top 1,000 | 4 | 0.7% | 4 | 0.7% |
| Spider, remaining ranks | 48 | 0.9% | 46 | 0.8% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. The naming total includes 50 files with a ban on every path and 2 files without that ban. Elsewhere, 6,119 files do not name it. We ignore capitalization and repeated names.
How to block or allow it
These lines go in robots.txt at the root. User-agent: * yields to a group naming Spider under the robots.txt standard.
Block the whole site
User-agent: Spider
Disallow: /Allow access
User-agent: Spider
Allow: /It's the 116th most-named crawler of the 171 we counted in the crawler index.
How we counted
For Spider, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. A request timed out after 8 seconds, with files capped at 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. Repeat readings are included, using scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
The 52 files naming it include 2 files without a whole-site ban.
How do the shares compare by rank?
At the top of the ranking, 0.7% of readable files name it and 0.7% block it. Across all ranks, those shares are 0.8% and 0.8%.
What other entry is close in the index?
The nearby MyCentralAIScraperBot entry reports 51 files that name it and 52 whole-site blocks, 0.8% of readable files. Here, 52 files mention the name and 50 block access, or 0.8%.