img2dataset
img2dataset is blocked by 181 sites in the October 2026 survey, or 2.9% of readable files. The community reference lists its operator as img2dataset.
Who runs img2dataset?
| Fact | img2dataset reference entry |
|---|---|
| Operator | “img2dataset” |
| Purpose | “Scrapes images for use in LLMs.” |
| Does img2dataset obey robots.txt? | “Unclear at this time.” |
| Frequency | “At the discretion of img2dataset users.” |
| Cloudflare managed list for img2dataset | No |
These facts quote the stored community ai.robots.txt reference.
“Downloads large sets of images into datasets for LLM training or other purposes.”
This entry’s stated use is quoted above. Allowing people to find and read a page need not include permission to gather it for model training. Owners can express those choices in crawler-specific groups.
How many sites block it?
A User-agent line names it in 200 files, 3.2% of 6,171 readable files. Whole-site bans, written Disallow: /, total 181, or 2.9%. That leaves 5,990 without a recorded ban.
Tranco’s top 1,000 sites yielded 552 readable files, including 37 files with its name, or 6.7%, and 33 files blocking it. Blocking is more common among the leading sites, at 6.0%. Outside those ranks, 148 of 5,619 files block it, or 2.6%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| img2dataset, all ranks | 200 | 3.2% | 181 | 2.9% |
| img2dataset, top 1,000 | 37 | 6.7% | 33 | 6.0% |
| img2dataset, remaining ranks | 163 | 2.9% | 148 | 2.6% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. Of 181 blocks, 1 does not also appear in our count of files naming it. The split among files that name it is 180 with a whole-site block and 20 without one. Its name is absent from 5,971 readable files. We ignore capitalization and repeated names.
How to block or allow it
Add these lines to robots.txt at your site root. For img2dataset, the robots.txt standard gives a named group priority over User-agent: *.
Block the whole site
User-agent: img2dataset
Disallow: /Allow access
User-agent: img2dataset
Allow: /It's the 40th most-named crawler of the 171 we counted in the crawler index.
How we counted
For img2dataset, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. Files had an 8-second timeout and a 262,144-byte limit. Missing, unreachable and oversized files do not contribute to these shares. scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) parsed both initial and repeat readings. See the full survey method.
Common questions
Can a file name it and still allow it?
We found 20 files without a whole-site ban among 200 that name it.
How do the shares compare by rank?
Leading files record 6.7% with the name and 6.0% blocking it. Across all readable files, the corresponding shares are 3.2% and 2.9%.
What other entry is close in the index?
VelenPublicWebCrawler is named by 200 files and blocked by 194, or 3.1%. For this crawler, 200 files name it and 181 block it, a 2.9% share.