ISSCyberRiskCrawler
ISSCyberRiskCrawler has 111 sites asking it to stay out in October 2026, or 1.8% of the files we could read. However, the reference says it does not obey robots.txt, and the community reference lists its operator as ISS-Corporate.
Who runs ISSCyberRiskCrawler?
| Fact | ISSCyberRiskCrawler reference entry |
|---|---|
| Operator | “ISS-Corporate” |
| Purpose | “Scrapes data to train machine learning models.” |
| Does ISSCyberRiskCrawler obey robots.txt? | “No” |
| Frequency | “No information.” |
| Cloudflare managed list for ISSCyberRiskCrawler | No |
These facts quote the stored community ai.robots.txt reference.
“Used to train machine learning based models to quantify cyber risk.”
The reference says this crawler ignores robots.txt. Allowing people to find and read a page need not include permission to gather it for model training. Owners can express those choices in crawler-specific groups.
How many sites block it?
Of 6,171 robots.txt files we could read, 111 block the whole site with Disallow: /, or 1.8%. Another way to read the sample is by name, with 109 files naming it in a User-agent line, or 1.8%. The 6,060 remaining files have no recorded whole-site block.
Breaking out the leading 1,000 sites gives 552 readable files, with 26 naming the crawler, or 4.7%. Whole-site blocking appears in 26 of them. Blocking is more common among the leading sites, at 4.7%. Beyond those ranks, 5,619 files have a blocking total of 85, 1.5% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| ISSCyberRiskCrawler, all ranks | 109 | 1.8% | 111 | 1.8% |
| ISSCyberRiskCrawler, top 1,000 | 26 | 4.7% | 26 | 4.7% |
| ISSCyberRiskCrawler, remaining ranks | 83 | 1.5% | 85 | 1.5% |
This total covers whole-site bans recorded for this name. A blanket User-agent: * rule alone is outside this measure. Of 111 blocks, 5 do not also appear in our count of files naming it. The split among files that name it is 106 with a whole-site block and 3 without one. Its name is absent from 6,062 readable files. We ignore capitalization and repeated names.
How to block or allow it
Place the group in your root robots.txt file. Per the robots.txt standard, a named group for ISSCyberRiskCrawler overrides User-agent: *.
Block the whole site
User-agent: ISSCyberRiskCrawler
Disallow: /Allow access
User-agent: ISSCyberRiskCrawler
Allow: /It's the 57th most-named crawler of the 171 we counted in the crawler index.
How we counted
For ISSCyberRiskCrawler, we surveyed Tranco’s top 10,000 sites on 19 October 2026. A request timed out after 8 seconds, with files capped at 262,144 bytes. Only available files we could read enter the percentages. Reading files again used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) as well. See the full survey method.
Common questions
Can a file name it and still allow it?
The 109 files that mention its name do not all record a whole-site ban. Those without one total 3.
How do the shares compare by rank?
Leading files record 4.7% with the name and 4.7% blocking it. Across all readable files, the corresponding shares are 1.8% and 1.8%.
What other entry is close in the index?
Gemini-Deep-Research is named by 108 files and blocked by 312, or 5.1%. For this crawler, 109 files name it and 111 block it, a 1.8% share.