YiBot
YiBot is listed as a search crawler, and the community reference lists its operator as 01.ai, an AI company founded by Kai-Fu Lee.. In October 2026, 16 sites block it, a 0.3% share of readable files.
Who runs YiBot?
| Fact | YiBot reference entry |
|---|---|
| Operator | “01.ai, an AI company founded by Kai-Fu Lee.” |
| Purpose | “This bot crawls the web to gather data and content for indexation and AI model training.” |
| Does YiBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for YiBot | No |
These facts quote the stored community ai.robots.txt reference.
“YiBot/1.0 is an advanced artificial intelligence platform designed to enhance interaction and engagement across various applications. Its architecture allows for seamless integration into websites and applications, providing users with a responsive and intuitive interface.”
The listed use here is search. An owner may want search visitors and answers that link back to a page while keeping its text out of model-training datasets.
How many sites block it?
Within 6,171 readable robots.txt files, the name appears in 17, or 0.3%. Instructions blocking every path, Disallow: /, appear in 16, a 0.3% share. No whole-site ban is recorded for 6,155 files.
Breaking out the leading 1,000 sites gives 552 readable files, with 1 naming the crawler, or 0.2%. Whole-site blocking appears in 1 of them. The leading sites have a lower block share, 0.2%. Beyond those ranks, 5,619 files have a blocking total of 15, 0.3% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| YiBot, all ranks | 17 | 0.3% | 16 | 0.3% |
| YiBot, top 1,000 | 1 | 0.2% | 1 | 0.2% |
| YiBot, remaining ranks | 16 | 0.3% | 15 | 0.3% |
Only whole-site blocks recorded against this crawler’s name count here. Rules addressed to every crawler are outside this count. Of 16 blocks, 1 does not also appear in our count of files naming it. The split among files that name it is 15 with a whole-site block and 2 without one. Its name is absent from 6,154 readable files. We ignore capitalization and repeated names.
How to block or allow it
Place the group in your root robots.txt file. Per the robots.txt standard, a named group for YiBot overrides User-agent: *.
Block the whole site
User-agent: YiBot
Disallow: /Allow access
User-agent: YiBot
Allow: /It's the 159th most-named crawler of the 171 we counted in the crawler index.
How we counted
For YiBot, our 19 October 2026 collection used Tranco’s top 10,000 sites. An 8-second wait or a file over 262,144 bytes ended a read. Shares exclude unreachable, unavailable and oversized files. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
A name identifies the crawler addressed by a group. The survey found 2 files with that name and no recorded whole-site ban, out of 17.
Are leading sites different?
At the top of the ranking, 0.2% of readable files name it and 0.2% block it. Across all ranks, those shares are 0.3% and 0.3%.
How does the nearby crawler compare?
For comparison, AgentDataBot has 16 files naming it and 16 blocks, or 0.3%. Our counts for this entry are 17 files with the name and 16 blocking the whole site, or 0.3%.