ChatGLM-Spider
ChatGLM-Spider is blocked by 123 sites in the October 2026 survey, or 2.0% of readable files. Its operator is unclear in the community reference.
Who runs ChatGLM-Spider?
| Fact | ChatGLM-Spider reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Data Scrapers” |
| Does ChatGLM-Spider obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for ChatGLM-Spider | No |
These facts quote the stored community ai.robots.txt reference.
This entry is classed as a crawler collecting data for AI. An owner who wants search traffic may allow indexing and visits requested by users, yet decline to have the same content gathered for training datasets.
How many sites block it?
Our readable sample has 123 whole-site blocks, 2.0% of 6,171 files, with Disallow: / as the instruction. Of that full sample, 117 name this crawler, or 1.9%. The count without a whole-site block is 6,048.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 26 files carry the name, or 4.7%, with 28 files blocking the whole site. Blocking is more common among the leading sites, at 5.1%. Farther down, 95 of 5,619 files record a block, 1.7% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| ChatGLM-Spider, all ranks | 117 | 1.9% | 123 | 2.0% |
| ChatGLM-Spider, top 1,000 | 26 | 4.7% | 28 | 5.1% |
| ChatGLM-Spider, remaining ranks | 91 | 1.6% | 95 | 1.7% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. Of 123 blocks, 11 do not also appear in our count of files naming it. Names and bans overlap in 112 files. We also found 5 files with a name and no whole-site ban, plus 6,054 files omitting the name. Case changes and repeated spellings are merged.
How to block or allow it
Edit the root robots.txt file on your site. The robots.txt standard gives a ChatGLM-Spider group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: ChatGLM-Spider
Disallow: /Allow access
User-agent: ChatGLM-Spider
Allow: /It's the 54th most-named crawler of the 171 we counted in the crawler index.
How we counted
For ChatGLM-Spider, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. Each file had 8 seconds to arrive, up to 262,144 bytes. Missing, unreachable and oversized files do not contribute to these shares. The file readings, including repeats, use scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
The 117 files naming it include 5 files without a whole-site ban.
How do the shares compare by rank?
Its name appears in 4.7% of readable leading files and 1.9% overall. Blocking affects 5.1% in the leading group and 2.0% in all ranks.
What other entry is close in the index?
The nearby atlassian-bot entry reports 110 files that name it and 103 whole-site blocks, 1.7% of readable files. Here, 117 files mention the name and 123 block access, or 2.0%.