TikTokSpider
TikTokSpider collects content for model training, and the community reference lists its operator as ByteDance. Our October 2026 survey found 334 sites blocking it, or 5.4% of the files we could read.
Who runs TikTokSpider?
| Fact | TikTokSpider reference entry |
|---|---|
| Operator | “ByteDance” |
| Purpose | “LLM training.” |
| Does TikTokSpider obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for TikTokSpider | No |
These facts quote the stored community ai.robots.txt reference.
“Downloads data to train LLMS, as per Bytespider.”
Model training is this crawler’s listed use. An owner may want a page included in search and available on request without offering its contents for future model development.
How many sites block it?
The 334 files blocking the whole site account for 5.4% of 6,171 readable robots.txt files. We found its name in 99 User-agent lists, a 1.6% share. Without a whole-site Disallow: / block, 5,837 files remain.
From the highest 1,000 ranks, 552 files could be read. Its naming count is 14, or 2.5%, and the block count is 50. Blocking is more common among the leading sites, at 9.1%. Compare 284 blocks among the 5,619 readable files below those ranks, or 5.1%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| TikTokSpider, all ranks | 99 | 1.6% | 334 | 5.4% |
| TikTokSpider, top 1,000 | 14 | 2.5% | 50 | 9.1% |
| TikTokSpider, remaining ranks | 85 | 1.5% | 284 | 5.1% |
These totals cover both groups naming it and rules that apply to every crawler. Of 334 blocks, 260 come only from that general rule, without naming this crawler. Names and bans overlap in 74 files. We also found 25 files with a name and no whole-site ban, plus 6,072 files omitting the name. Case changes and repeated spellings are merged.
Has that changed since August?
Between August and October, 16 sites that had blocked it stopped and 27 started, so 11 more of the same 5,889 sites block it. A further 275 blocked it in both months, and 5,571 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 312 blocks among 6,236 readable files. October’s was 334 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 27 |
| Stopped blocking | 16 |
| Blocked in both | 275 |
| Blocked in neither | 5,571 |
| Net change | +11 |
How to block or allow it
Edit the root robots.txt file on your site. The robots.txt standard gives a TikTokSpider group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: TikTokSpider
Disallow: /Allow access
User-agent: TikTokSpider
Allow: /It's the 60th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For TikTokSpider, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. A request timed out after 8 seconds, with files capped at 262,144 bytes. Missing, unreachable and oversized files do not contribute to these shares. Reading files again used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) as well. See the full survey method.
Common questions
Can a file name it and still allow it?
A name identifies the crawler addressed by a group. The survey found 25 files with that name and no recorded whole-site ban, out of 99.
How do the shares compare by rank?
At the top of the ranking, 2.5% of readable files name it and 9.1% block it. Across all ranks, those shares are 1.6% and 5.4%.
What other entry is close in the index?
The nearby aiHitBot entry reports 98 files that name it and 94 whole-site blocks, 1.5% of readable files. Here, 99 files mention the name and 334 block access, or 5.4%.