iaskspider
iaskspider has 120 sites asking it to stay out in October 2026, or 1.9% of the files we could read. However, the reference says it does not obey robots.txt, and the community reference lists its operator as iAsk.
Who runs iaskspider?
| Fact | iaskspider reference entry |
|---|---|
| Operator | “Unclear at this time.” / “iAsk” |
| Purpose | “Undocumented AI Agents” / “Crawls sites to provide answers to user queries.” |
| Does iaskspider obey robots.txt? | “Unclear at this time.” / “No” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for iaskspider | No |
These facts quote the stored community ai.robots.txt reference.
“Used to provide answers to user queries.”
The reference says this crawler ignores robots.txt. An owner who wants search traffic may allow indexing and visits requested by users, yet decline to have the same content gathered for training datasets.
How many sites block it?
Within 6,171 readable robots.txt files, the name appears in 138, or 2.2%. Instructions blocking every path, Disallow: /, appear in 120, a 1.9% share. No whole-site ban is recorded for 6,051 files.
At the top 1,000 ranks, we could read 552 files. Naming covers 30, a 5.4% share, and blocking covers 22. Blocking is more common among the leading sites, at 4.0%. Elsewhere, 5,619 readable files give 98 blocks, or 1.7%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| iaskspider, all ranks | 138 | 2.2% | 120 | 1.9% |
| iaskspider, top 1,000 | 30 | 5.4% | 22 | 4.0% |
| iaskspider, remaining ranks | 108 | 1.9% | 98 | 1.7% |
Only whole-site blocks recorded against this crawler’s name count here. Rules addressed to every crawler are outside this count. Of 120 blocks, 22 do not also appear in our count of files naming it. The counted overlap is 98 files, with 40 files naming it without banning all paths. Missing names account for 6,033 files. Repeating a name or changing its capitalization does not increase the naming total.
How to block or allow it
The examples include the spelling variants counted together here. Put these in robots.txt at the site root. Per the robots.txt standard, a group naming iaskspider takes precedence over User-agent: *.
Block the whole site
User-agent: iaskspider
User-agent: iaskspider/2.0
Disallow: /Allow access
User-agent: iaskspider
User-agent: iaskspider/2.0
Allow: /It's the 52nd most-named crawler of the 171 we counted in the crawler index.
How we counted
For iaskspider, we surveyed Tranco’s top 10,000 sites on 19 October 2026. A request timed out after 8 seconds, with files capped at 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
We found 40 files without a whole-site ban among 138 that name it.
How do the shares compare by rank?
Within readable robots.txt files near the top of the ranking, 5.4% name it and 4.0% block it. Overall, naming reaches 2.2% and blocking 1.9%.
What other entry is close in the index?
For comparison, ShapBot has 123 files naming it and 120 blocks, or 1.9%. Our counts for this entry are 138 files with the name and 120 blocking the whole site, or 1.9%.