HenkBot
HenkBot has 19 recorded blocks in October 2026, or 0.3%, and 18 files naming it. Its operator is unclear in the community reference.
Who runs HenkBot?
| Fact | HenkBot reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Data Providers” |
| Does HenkBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for HenkBot | No |
These facts quote the stored community ai.robots.txt reference.
“Henkbot crawls the web on behalf of Valyu, an AI search infrastructure provider that indexes content for use in AI-powered retrieval pipelines.”
Its listed role is supplying data for AI tools. Search referrals or a user’s question can give an owner a reason to allow access while declining collection of page text for training.
How many sites block it?
Instructions to stay out of the whole site appear in 19 files, or 0.3% of 6,171 readable files. We counted 18 with its name in a User-agent line, a 0.3% share. Another 6,152 lack a recorded Disallow: / ban.
For the top 1,000 ranks, its name appears in 2 files out of 552, or 0.4%. Whole-site blocks total 2. Blocking is more common among the leading sites, at 0.4%. Lower in the ranking, blocking reaches 17 files out of 5,619, a 0.3% share.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| HenkBot, all ranks | 18 | 0.3% | 19 | 0.3% |
| HenkBot, top 1,000 | 2 | 0.4% | 2 | 0.4% |
| HenkBot, remaining ranks | 16 | 0.3% | 17 | 0.3% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. Of 19 blocks, 1 does not also appear in our count of files naming it. All 18 files naming it also record a whole-site block. The remaining 6,153 files do not name it. Naming counts ignore capitalization and repeated mentions.
How to block or allow it
Add these lines to robots.txt at your site root. For HenkBot, the robots.txt standard gives a named group priority over User-agent: *.
Block the whole site
User-agent: HenkBot
Disallow: /Allow access
User-agent: HenkBot
Allow: /It's the 154th most-named crawler of the 171 we counted in the crawler index.
How we counted
For HenkBot, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. Each file had 8 seconds to arrive, up to 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. Later readings are included in results from scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
The 18 files with its name all deny whole-site access in these readings.
How do the shares compare by rank?
For naming rules, 0.4% of the leading sample mentions it, compared with 0.3% overall. Whole-site blocking is 0.4% at leading ranks and 0.3% across the full sample.
What other entry is close in the index?
For comparison, opencode has 18 files naming it and 16 blocks, or 0.3%. Our counts for this entry are 18 files with the name and 19 blocking the whole site, or 0.3%.