LAIONDownloader
LAIONDownloader has 34 sites asking it to stay out in October 2026, or 0.6% of the files we could read. However, the reference says it does not obey robots.txt, and the community reference lists its operator as Large-scale Artificial Intelligence Open Network.
Who runs LAIONDownloader?
| Fact | LAIONDownloader reference entry |
|---|---|
| Operator | “Large-scale Artificial Intelligence Open Network” |
| Purpose | “AI tools and models for machine learning research.” |
| Does LAIONDownloader obey robots.txt? | “No” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for LAIONDownloader | No |
These facts quote the stored community ai.robots.txt reference.
“LAIONDownloader is a bot by LAION, a non-profit organization that provides datasets, tools and models to liberate machine learning research.”
The reference says this crawler ignores robots.txt. An owner may want a page included in search and available on request without offering its contents for future model development.
How many sites block it?
Of 6,171 robots.txt files we could read, 34 block the whole site with Disallow: /, or 0.6%. Another way to read the sample is by name, with 28 files naming it in a User-agent line, or 0.5%. The 6,137 remaining files have no recorded whole-site block.
From the highest 1,000 ranks, 552 files could be read. Its naming count is 1, or 0.2%, and the block count is 1. The leading sites have a lower block share, 0.2%. Compare 33 blocks among the 5,619 readable files below those ranks, or 0.6%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| LAIONDownloader, all ranks | 28 | 0.5% | 34 | 0.6% |
| LAIONDownloader, top 1,000 | 1 | 0.2% | 1 | 0.2% |
| LAIONDownloader, remaining ranks | 27 | 0.5% | 33 | 0.6% |
For this entry, we counted whole-site bans attached to its own name, leaving general rules for all crawlers out of the total. Of 34 blocks, 7 do not also appear in our count of files naming it. Looking just at named files gives 27 blocks and 1 without a whole-site block. For 6,143 readable files, we found no mention by name. We ignore capitalization and repeated names.
How to block or allow it
Put these in robots.txt at the site root. Per the robots.txt standard, a group naming LAIONDownloader takes precedence over User-agent: *.
Block the whole site
User-agent: LAIONDownloader
Disallow: /Allow access
User-agent: LAIONDownloader
Allow: /It's the 143rd most-named crawler of the 171 we counted in the crawler index.
How we counted
For LAIONDownloader, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. Files had an 8-second timeout and a 262,144-byte limit. Missing, unreachable and oversized files do not contribute to these shares. The file readings, including repeats, use scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
We found 1 file without a whole-site ban among 28 that name it.
What do the top-ranked sites show?
Within readable robots.txt files near the top of the ranking, 0.2% name it and 0.2% block it. Overall, naming reaches 0.5% and blocking 0.6%.
Which nearby entry can I compare?
The nearby TongyiBot entry reports 24 files that name it and 27 whole-site blocks, 0.4% of readable files. Here, 28 files mention the name and 34 block access, or 0.6%.