DeepSeekBot
DeepSeekBot has 387 sites asking it to stay out in October 2026, or 6.3% of the files we could read. However, the reference says it does not obey robots.txt, and the community reference lists its operator as DeepSeek.
Who runs DeepSeekBot?
| Fact | DeepSeekBot reference entry |
|---|---|
| Operator | “DeepSeek” |
| Purpose | “Training language models and improving AI products” |
| Does DeepSeekBot obey robots.txt? | “No” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for DeepSeekBot | No |
These facts quote the stored community ai.robots.txt reference.
“DeepSeekBot is a web crawler used by DeepSeek to train its language models and improve its AI products.”
The reference says this crawler ignores robots.txt. Helping people find a page through search is a different use from using its contents to train a model, so owners may allow those visits separately.
How many sites block it?
The 387 files blocking the whole site account for 6.3% of 6,171 readable robots.txt files. We found its name in 180 User-agent lists, a 2.9% share. Without a whole-site Disallow: / block, 5,784 files remain.
Breaking out the leading 1,000 sites gives 552 readable files, with 34 naming the crawler, or 6.2%. Whole-site blocking appears in 65 of them. Blocking is more common among the leading sites, at 11.8%. Beyond those ranks, 5,619 files have a blocking total of 322, 5.7% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| DeepSeekBot, all ranks | 180 | 2.9% | 387 | 6.3% |
| DeepSeekBot, top 1,000 | 34 | 6.2% | 65 | 11.8% |
| DeepSeekBot, remaining ranks | 146 | 2.6% | 322 | 5.7% |
A general User-agent: * ban contributes to this total whenever it applies to this crawler. Of 387 blocks, 230 come only from that general rule, without naming this crawler. The counted overlap is 157 files, with 23 files naming it without banning all paths. Missing names account for 5,991 files. Repeating a name or changing its capitalization does not increase the naming total.
Has that changed since August?
Between August and October, 16 sites that had blocked it stopped and 24 started, so 8 more of the same 5,889 sites block it. A further 332 blocked it in both months, and 5,517 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 371 blocks among 6,236 readable files. October’s was 387 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 24 |
| Stopped blocking | 16 |
| Blocked in both | 332 |
| Blocked in neither | 5,517 |
| Net change | +8 |
How to block or allow it
Edit the root robots.txt file on your site. The robots.txt standard gives a DeepSeekBot group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: DeepSeekBot
Disallow: /Allow access
User-agent: DeepSeekBot
Allow: /It's the 42nd most-named crawler of the 171 we counted in the crawler index.
How we counted
For DeepSeekBot, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. An 8-second wait or a file over 262,144 bytes ended a read. Only available files we could read enter the percentages. Reading files again used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) as well. See the full survey method.
Common questions
Can a file name it and still allow it?
A name identifies the crawler addressed by a group. The survey found 23 files with that name and no recorded whole-site ban, out of 180.
How do the shares compare by rank?
Its name appears in 6.2% of readable leading files and 2.9% overall. Blocking affects 11.8% in the leading group and 6.3% in all ranks.
What other entry is close in the index?
Ai2Bot-Dolma is named by 177 files and blocked by 271, or 4.4%. For this crawler, 180 files name it and 387 block it, a 6.3% share.