Bytespider
Bytespider has 930 sites asking it to stay out in October 2026, or 15.1% of the files we could read. However, the reference says it does not obey robots.txt, and the community reference lists its operator as ByteDance.
Who runs Bytespider?
| Fact | Bytespider reference entry |
|---|---|
| Operator | “ByteDance” |
| Purpose | “LLM training.” |
| Does Bytespider obey robots.txt? | “No” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Bytespider | Yes |
These facts quote the stored community ai.robots.txt reference.
“Downloads data to train LLMS, including ChatGPT competitors.”
The reference says this crawler ignores robots.txt. Helping people find a page through search is a different use from using its contents to train a model, so owners may allow those visits separately.
How many sites block it?
The 930 files blocking the whole site account for 15.1% of 6,171 readable robots.txt files. We found its name in 826 User-agent lists, a 13.4% share. Without a whole-site Disallow: / block, 5,241 files remain.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 110 files carry the name, or 19.9%, with 126 files blocking the whole site. Blocking is more common among the leading sites, at 22.8%. Farther down, 804 of 5,619 files record a block, 14.3% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Bytespider, all ranks | 826 | 13.4% | 930 | 15.1% |
| Bytespider, top 1,000 | 110 | 19.9% | 126 | 22.8% |
| Bytespider, remaining ranks | 716 | 12.7% | 804 | 14.3% |
The survey includes relevant User-agent: * instructions alongside instructions addressed to this crawler. Of 930 blocks, 212 come only from that general rule, without naming this crawler. The split among files that name it is 718 with a whole-site block and 108 without one. Its name is absent from 5,345 readable files. We ignore capitalization and repeated names.
Has that changed since August?
Between August and October, 168 sites that had blocked it stopped and 66 started, so 102 fewer of the same 5,889 sites block it. A further 816 blocked it in both months, and 4,839 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 1,096 blocks among 6,236 readable files. October’s was 930 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 66 |
| Stopped blocking | 168 |
| Blocked in both | 816 |
| Blocked in neither | 4,839 |
| Net change | -102 |
How to block or allow it
Put these in robots.txt at the site root. Per the robots.txt standard, a group naming Bytespider takes precedence over User-agent: *.
Block the whole site
User-agent: Bytespider
Disallow: /Allow access
User-agent: Bytespider
Allow: /It's the 5th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For Bytespider, our 19 October 2026 collection used Tranco’s top 10,000 sites. An 8-second wait or a file over 262,144 bytes ended a read. Missing, unreachable and oversized files do not contribute to these shares. Reading files again used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) as well. See the full survey method.
Common questions
What does naming this crawler tell me?
The 826 files that mention its name do not all record a whole-site ban. Those without one total 108.
Are leading sites different?
For naming rules, 19.9% of the leading sample mentions it, compared with 13.4% overall. Whole-site blocking is 22.8% at leading ranks and 15.1% across the full sample.
How does the nearby crawler compare?
PerplexityBot is named by 784 files and blocked by 622, or 10.1%. For this crawler, 826 files name it and 930 block it, a 15.1% share.