QwenBot
Few readable files block QwenBot in October 2026, at 20 sites or 0.3%. The community reference lists its operator as Alibaba.
Who runs QwenBot?
| Fact | QwenBot reference entry |
|---|---|
| Operator | “Alibaba” |
| Purpose | “Collects public web content for Alibaba's Qwen (Tongyi Qianwen) large language models.” |
| Does QwenBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for QwenBot | No |
These facts quote the stored community ai.robots.txt reference.
“QwenBot is a crawler operated by Alibaba to collect public web content for its Qwen (Tongyi Qianwen) large language models. Alibaba also operates TongyiBot.”
This entry’s stated use is quoted above. An owner may want a page included in search and available on request without offering its contents for future model development.
How many sites block it?
The name occurs in 20 out of 6,171 readable files, a 0.3% share. Across the full sample, the whole-site Disallow: / block count is 20, or 0.3%. Files without that ban number 6,151.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 1 file carry the name, or 0.2%, with 2 files blocking the whole site. Blocking is more common among the leading sites, at 0.4%. Farther down, 18 of 5,619 files record a block, 0.3% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| QwenBot, all ranks | 20 | 0.3% | 20 | 0.3% |
| QwenBot, top 1,000 | 1 | 0.2% | 2 | 0.4% |
| QwenBot, remaining ranks | 19 | 0.3% | 18 | 0.3% |
This is a count of blocks attached to the named crawler, covering the whole site. Blanket rules for every crawler do not enter it. Of 20 blocks, 2 do not also appear in our count of files naming it. The split among files that name it is 18 with a whole-site block and 2 without one. Its name is absent from 6,151 readable files. We ignore capitalization and repeated names.
How to block or allow it
Put these in robots.txt at the site root. Per the robots.txt standard, a group naming QwenBot takes precedence over User-agent: *.
Block the whole site
User-agent: QwenBot
Disallow: /Allow access
User-agent: QwenBot
Allow: /It's the 149th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For QwenBot, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. Requests could run for 8 seconds, with a 262,144-byte file limit. Missing, unreachable and oversized files do not contribute to these shares. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
This survey includes 2 files with the name and no whole-site ban, out of 20 naming files.
Are leading sites different?
For naming rules, 0.2% of the leading sample mentions it, compared with 0.3% overall. Whole-site blocking is 0.4% at leading ranks and 0.3% across the full sample.
How does the nearby crawler compare?
CragCrawler is named by 19 files and blocked by 24, or 0.4%. For this crawler, 20 files name it and 20 block it, a 0.3% share.