ICC-Crawler
ICC-Crawler collects content for model training, and the community reference lists its operator as NICT. Our October 2026 survey found 229 sites blocking it, or 3.7% of the files we could read.
Who runs ICC-Crawler?
| Fact | ICC-Crawler reference entry |
|---|---|
| Operator | “NICT” |
| Purpose | “Scrapes data to train and support AI technologies.” |
| Does ICC-Crawler obey robots.txt? | “Yes” |
| Frequency | “No information.” |
| Cloudflare managed list for ICC-Crawler | No |
These facts quote the stored community ai.robots.txt reference.
“Use the collected data for artificial intelligence technologies; provide data to third parties, including commercial companies; those companies can use the data for their own business.”
Model training is this crawler’s listed use. An owner may want search visitors and answers that link back to a page while keeping its text out of model-training datasets.
How many sites block it?
A User-agent line names it in 230 files, 3.7% of 6,171 readable files. Whole-site bans, written Disallow: /, total 229, or 3.7%. That leaves 5,942 without a recorded ban.
Tranco’s top 1,000 sites yielded 552 readable files, including 33 files with its name, or 6.0%, and 33 files blocking it. Blocking is more common among the leading sites, at 6.0%. Outside those ranks, 196 of 5,619 files block it, or 3.5%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| ICC-Crawler, all ranks | 230 | 3.7% | 229 | 3.7% |
| ICC-Crawler, top 1,000 | 33 | 6.0% | 33 | 6.0% |
| ICC-Crawler, remaining ranks | 197 | 3.5% | 196 | 3.5% |
This total covers whole-site bans recorded for this name. A blanket User-agent: * rule alone is outside this measure. Of 229 blocks, 7 do not also appear in our count of files naming it. The naming total includes 222 files with a ban on every path and 8 files without that ban. Elsewhere, 5,941 files do not name it. We ignore capitalization and repeated names.
How to block or allow it
Use your site’s root file, robots.txt. Specific groups for ICC-Crawler take precedence over User-agent: *, according to the robots.txt standard.
Block the whole site
User-agent: ICC-Crawler
Disallow: /Allow access
User-agent: ICC-Crawler
Allow: /It's the 37th most-named crawler of the 171 we counted in the crawler index.
How we counted
For ICC-Crawler, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. Each file had 8 seconds to arrive, up to 262,144 bytes. Missing, unreachable and oversized files do not contribute to these shares. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
A name identifies the crawler addressed by a group. The survey found 8 files with that name and no recorded whole-site ban, out of 230.
How do the shares compare by rank?
At the top of the ranking, 6.0% of readable files name it and 6.0% block it. Across all ranks, those shares are 3.7% and 3.7%.
What other entry is close in the index?
The nearby PanguBot entry reports 205 files that name it and 193 whole-site blocks, 3.1% of readable files. Here, 230 files mention the name and 229 block access, or 3.7%.