PanguBot
PanguBot is blocked by 193 sites in the October 2026 survey, or 3.1% of readable files. The community reference lists its operator as the Chinese company Huawei.
Who runs PanguBot?
| Fact | PanguBot reference entry |
|---|---|
| Operator | “the Chinese company Huawei” |
| Purpose | “AI Data Scrapers” |
| Does PanguBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for PanguBot | No |
These facts quote the stored community ai.robots.txt reference.
“PanguBot is a web crawler operated by the Chinese company Huawei. It's used to download training data for its multimodal LLM (Large Language Model) called PanGu.”
This entry is classed as a crawler collecting data for AI. An owner may allow a search crawler to list a page or an assistant to retrieve it for someone, yet decline to supply the text for training.
How many sites block it?
A User-agent line names it in 205 files, 3.3% of 6,171 readable files. Whole-site bans, written Disallow: /, total 193, or 3.1%. That leaves 5,978 without a recorded ban.
From the highest 1,000 ranks, 552 files could be read. Its naming count is 36, or 6.5%, and the block count is 33. Blocking is more common among the leading sites, at 6.0%. Compare 160 blocks among the 5,619 readable files below those ranks, or 2.8%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| PanguBot, all ranks | 205 | 3.3% | 193 | 3.1% |
| PanguBot, top 1,000 | 36 | 6.5% | 33 | 6.0% |
| PanguBot, remaining ranks | 169 | 3.0% | 160 | 2.8% |
For this entry, we counted whole-site bans attached to its own name, leaving general rules for all crawlers out of the total. Of 193 blocks, 1 does not also appear in our count of files naming it. The overlap between naming and blocking is 192 files. Naming without a whole-site ban accounts for 13, and 5,966 files never name it. We ignore capitalization and repeated names.
How to block or allow it
These lines go in robots.txt at the root. User-agent: * yields to a group naming PanguBot under the robots.txt standard.
Block the whole site
User-agent: PanguBot
Disallow: /Allow access
User-agent: PanguBot
Allow: /It's the 38th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For PanguBot, our 19 October 2026 collection used Tranco’s top 10,000 sites. Requests could run for 8 seconds, with a 262,144-byte file limit. Shares exclude unreachable, unavailable and oversized files. Reading files again used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) as well. See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
We found 13 files without a whole-site ban among 205 that name it.
What do the top-ranked sites show?
For naming rules, 6.5% of the leading sample mentions it, compared with 3.3% overall. Whole-site blocking is 6.0% at leading ranks and 3.1% across the full sample.
Which nearby entry can I compare?
The nearby FriendlyCrawler entry reports 201 files that name it and 196 whole-site blocks, 3.2% of readable files. Here, 205 files mention the name and 193 block access, or 3.1%.