YiyanBot
Few readable files block YiyanBot in October 2026, at 22 sites or 0.4%. The community reference lists its operator as Baidu that fetches web content for the yiyan.
Who runs YiyanBot?
| Fact | YiyanBot reference entry |
|---|---|
| Operator | “Baidu that fetches web content for the yiyan” |
| Purpose | “AI Assistants” |
| Does YiyanBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for YiyanBot | No |
These facts quote the stored community ai.robots.txt reference.
“YiyanBot is a web crawler operated by Baidu that fetches web content for the yiyan.baidu.com assistant and related ERNIE-generated answers.”
The stated role here is an AI assistant. Search referrals or a user’s question can give an owner a reason to allow access while declining collection of page text for training.
How many sites block it?
A User-agent line names it in 22 files, 0.4% of 6,171 readable files. Whole-site bans, written Disallow: /, total 22, or 0.4%. That leaves 6,149 without a recorded ban.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 3 files carry the name, or 0.5%, with 3 files blocking the whole site. Blocking is more common among the leading sites, at 0.5%. Farther down, 19 of 5,619 files record a block, 0.3% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| YiyanBot, all ranks | 22 | 0.4% | 22 | 0.4% |
| YiyanBot, top 1,000 | 3 | 0.5% | 3 | 0.5% |
| YiyanBot, remaining ranks | 19 | 0.3% | 19 | 0.3% |
Only whole-site blocks recorded against this crawler’s name count here. Rules addressed to every crawler are outside this count. Of 22 blocks, 1 does not also appear in our count of files naming it. Names and bans overlap in 21 files. We also found 1 file with a name and no whole-site ban, plus 6,149 files omitting the name. Case changes and repeated spellings are merged.
How to block or allow it
Place the group in your root robots.txt file. Per the robots.txt standard, a named group for YiyanBot overrides User-agent: *.
Block the whole site
User-agent: YiyanBot
Disallow: /Allow access
User-agent: YiyanBot
Allow: /It's the 147th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For YiyanBot, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. Each file had 8 seconds to arrive, up to 262,144 bytes. Only available files we could read enter the percentages. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
The 22 files that mention its name do not all record a whole-site ban. Those without one total 1.
Are leading sites different?
At the top of the ranking, 0.5% of readable files name it and 0.5% block it. Across all ranks, those shares are 0.4% and 0.4%.
How does the nearby crawler compare?
UseAI is named by 21 files and blocked by 25, or 0.4%. For this crawler, 22 files name it and 22 block it, a 0.4% share.