AIWebIndex
AIWebIndex is listed as a search crawler, and the community reference lists its operator as Lyrenth. In October 2026, 24 sites block it, a 0.4% share of readable files.
Who runs AIWebIndex?
| Fact | AIWebIndex reference entry |
|---|---|
| Operator | “Lyrenth” |
| Purpose | “AI Search Crawlers” |
| Does AIWebIndex obey robots.txt? | “Yes” |
| Frequency | “At most one request per domain every 2 seconds, and slower where robots.txt sets a longer Crawl-delay.” |
| Cloudflare managed list for AIWebIndex | No |
These facts quote the stored community ai.robots.txt reference.
“Builds an index of public pages and serves them to AI agents as extracted, readable text with attribution and a link back to the source. Does not train foundation models on crawled content. Identity can be checked three ways: published IP ranges at https://lyrenth.com/bot/ip-ranges.json, forward-confirmed reverse DNS under lyrenth.com, and Web Bot Auth signatures (RFC 9421). Full policy at https://lyrenth.com/crawler-policy”
The listed use here is search. Indexing helps readers find pages, and a user’s request can fetch one. Training reuses the material in models, which an owner may choose to block.
How many sites block it?
Instructions to stay out of the whole site appear in 24 files, or 0.4% of 6,171 readable files. We counted 18 with its name in a User-agent line, a 0.3% share. Another 6,147 lack a recorded Disallow: / ban.
For the top 1,000 ranks, its name appears in 3 files out of 552, or 0.5%. Whole-site blocks total 3. Blocking is more common among the leading sites, at 0.5%. Lower in the ranking, blocking reaches 21 files out of 5,619, a 0.4% share.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| AIWebIndex, all ranks | 18 | 0.3% | 24 | 0.4% |
| AIWebIndex, top 1,000 | 3 | 0.5% | 3 | 0.5% |
| AIWebIndex, remaining ranks | 15 | 0.3% | 21 | 0.4% |
For this entry, we counted whole-site bans attached to its own name, leaving general rules for all crawlers out of the total. Of 24 blocks, 7 do not also appear in our count of files naming it. The split among files that name it is 17 with a whole-site block and 1 without one. Its name is absent from 6,153 readable files. We ignore capitalization and repeated names.
How to block or allow it
Place the group in your root robots.txt file. Per the robots.txt standard, a named group for AIWebIndex overrides User-agent: *.
Block the whole site
User-agent: AIWebIndex
Disallow: /Allow access
User-agent: AIWebIndex
Allow: /It's the 153rd most-named crawler of the 171 we counted in the crawler index.
How we counted
For AIWebIndex, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. A request timed out after 8 seconds, with files capped at 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. The file readings, including repeats, use scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
We found 1 file without a whole-site ban among 18 that name it.
Are leading sites different?
For naming rules, 0.5% of the leading sample mentions it, compared with 0.3% overall. Whole-site blocking is 0.5% at leading ranks and 0.4% across the full sample.
How does the nearby crawler compare?
HenkBot is named by 18 files and blocked by 19, or 0.3%. For this crawler, 18 files name it and 24 block it, a 0.4% share.