DoubaoBot
DoubaoBot has 28 recorded blocks in October 2026, or 0.5%, and 15 files naming it. The community reference lists its operator as ByteDance.
Who runs DoubaoBot?
| Fact | DoubaoBot reference entry |
|---|---|
| Operator | “ByteDance” |
| Purpose | “AI crawler for ByteDance's Doubao AI assistant.” |
| Does DoubaoBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for DoubaoBot | No |
These facts quote the stored community ai.robots.txt reference.
“DoubaoBot is a crawler operated by ByteDance to collect web content for its Doubao AI assistant. Distinct from Bytespider (general/training) and TikTokSpider (video).”
The stated role here is an AI assistant. Helping people find a page through search is a different use from using its contents to train a model, so owners may allow those visits separately.
How many sites block it?
Instructions to stay out of the whole site appear in 28 files, or 0.5% of 6,171 readable files. We counted 15 with its name in a User-agent line, a 0.2% share. Another 6,143 lack a recorded Disallow: / ban.
Breaking out the leading 1,000 sites gives 552 readable files, with 1 naming the crawler, or 0.2%. Whole-site blocking appears in 4 of them. Blocking is more common among the leading sites, at 0.7%. Beyond those ranks, 5,619 files have a blocking total of 24, 0.4% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| DoubaoBot, all ranks | 15 | 0.2% | 28 | 0.5% |
| DoubaoBot, top 1,000 | 1 | 0.2% | 4 | 0.7% |
| DoubaoBot, remaining ranks | 14 | 0.2% | 24 | 0.4% |
This total covers whole-site bans recorded for this name. A blanket User-agent: * rule alone is outside this measure. Of 28 blocks, 13 do not also appear in our count of files naming it. All 15 files naming it also record a whole-site block. The remaining 6,156 files do not name it. Naming counts ignore capitalization and repeated mentions.
How to block or allow it
Add these lines to robots.txt at your site root. For DoubaoBot, the robots.txt standard gives a named group priority over User-agent: *.
Block the whole site
User-agent: DoubaoBot
Disallow: /Allow access
User-agent: DoubaoBot
Allow: /It's the 166th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For DoubaoBot, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. An 8-second wait or a file over 262,144 bytes ended a read. Only available files we could read enter the percentages. Collection and repeat readings used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
In the 15 files naming this crawler, every one also blocks the whole site.
How do the shares compare by rank?
At the top of the ranking, 0.2% of readable files name it and 0.7% block it. Across all ranks, those shares are 0.2% and 0.5%.
What other entry is close in the index?
For comparison, Hunyuan has 15 files naming it and 14 blocks, or 0.2%. Our counts for this entry are 15 files with the name and 28 blocking the whole site, or 0.5%.