imageSpider
imageSpider is blocked by 104 sites in the October 2026 survey, or 1.7% of readable files. Its operator is unclear in the community reference.
Who runs imageSpider?
| Fact | imageSpider reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Data Scrapers” |
| Does imageSpider obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for imageSpider | No |
These facts quote the stored community ai.robots.txt reference.
This entry is classed as a crawler collecting data for AI. Allowing people to find and read a page need not include permission to gather it for model training. Owners can express those choices in crawler-specific groups.
How many sites block it?
The 104 files blocking the whole site account for 1.7% of 6,171 readable robots.txt files. We found its name in 90 User-agent lists, a 1.5% share. Without a whole-site Disallow: / block, 6,067 files remain.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 20 files carry the name, or 3.6%, with 20 files blocking the whole site. Blocking is more common among the leading sites, at 3.6%. Farther down, 84 of 5,619 files record a block, 1.5% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| imageSpider, all ranks | 90 | 1.5% | 104 | 1.7% |
| imageSpider, top 1,000 | 20 | 3.6% | 20 | 3.6% |
| imageSpider, remaining ranks | 70 | 1.2% | 84 | 1.5% |
This is a count of blocks attached to the named crawler, covering the whole site. Blanket rules for every crawler do not enter it. Of 104 blocks, 16 do not also appear in our count of files naming it. The overlap between naming and blocking is 88 files. Naming without a whole-site ban accounts for 2, and 6,081 files never name it. We ignore capitalization and repeated names.
How to block or allow it
Use your site’s root file, robots.txt. Specific groups for imageSpider take precedence over User-agent: *, according to the robots.txt standard.
Block the whole site
User-agent: imageSpider
Disallow: /Allow access
User-agent: imageSpider
Allow: /It's the 67th most-named crawler of the 171 we counted in the crawler index.
How we counted
For imageSpider, our 19 October 2026 collection used Tranco’s top 10,000 sites. Requests could run for 8 seconds, with a 262,144-byte file limit. Only available files we could read enter the percentages. The file readings, including repeats, use scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
We found 2 files without a whole-site ban among 90 that name it.
How do the shares compare by rank?
Its name appears in 3.6% of readable leading files and 1.5% overall. Blocking affects 3.6% in the leading group and 1.7% in all ranks.
What other entry is close in the index?
The nearby Datenbank Crawler entry reports 88 files that name it and 0 whole-site blocks, 0.0% of readable files. Here, 90 files mention the name and 104 block access, or 1.7%.