Aranet-SearchBot
Few readable files block Aranet-SearchBot in October 2026, at 16 sites or 0.3%. Its operator is unclear in the community reference.
Who runs Aranet-SearchBot?
| Fact | Aranet-SearchBot reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “Undocumented AI Agents” |
| Does Aranet-SearchBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Aranet-SearchBot | No |
These facts quote the stored community ai.robots.txt reference.
The reference leaves this agent’s purpose undocumented. Helping people find a page through search is a different use from using its contents to train a model, so owners may allow those visits separately.
How many sites block it?
The 16 files blocking the whole site account for 0.3% of 6,171 readable robots.txt files. We found its name in 16 User-agent lists, a 0.3% share. Without a whole-site Disallow: / block, 6,155 files remain.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 3 files carry the name, or 0.5%, with 2 files blocking the whole site. Blocking is more common among the leading sites, at 0.4%. Farther down, 14 of 5,619 files record a block, 0.2% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Aranet-SearchBot, all ranks | 16 | 0.3% | 16 | 0.3% |
| Aranet-SearchBot, top 1,000 | 3 | 0.5% | 2 | 0.4% |
| Aranet-SearchBot, remaining ranks | 13 | 0.2% | 14 | 0.2% |
This total covers whole-site bans recorded for this name. A blanket User-agent: * rule alone is outside this measure. Of 16 blocks, 2 do not also appear in our count of files naming it. The counted overlap is 14 files, with 2 files naming it without banning all paths. Missing names account for 6,155 files. Repeating a name or changing its capitalization does not increase the naming total.
How to block or allow it
Put these in robots.txt at the site root. Per the robots.txt standard, a group naming Aranet-SearchBot takes precedence over User-agent: *.
Block the whole site
User-agent: Aranet-SearchBot
Disallow: /Allow access
User-agent: Aranet-SearchBot
Allow: /It's the 161st most-named crawler of the 171 we counted in the crawler index.
How we counted
For Aranet-SearchBot, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. An 8-second wait or a file over 262,144 bytes ended a read. Only available files we could read enter the percentages. scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) parsed both initial and repeat readings. See the full survey method.
Common questions
What does naming this crawler tell me?
We found 2 files without a whole-site ban among 16 that name it.
Are leading sites different?
Leading files record 0.5% with the name and 0.4% blocking it. Across all readable files, the corresponding shares are 0.3% and 0.3%.
How does the nearby crawler compare?
BixelBot is named by 16 files and blocked by 16, or 0.3%. For this crawler, 16 files name it and 16 block it, a 0.3% share.