Diffbot
Diffbot is blocked by 723 sites in the October 2026 survey, or 11.7% of readable files. The community reference lists its operator as Diffbot.
Who runs Diffbot?
| Fact | Diffbot reference entry |
|---|---|
| Operator | “Diffbot” |
| Purpose | “AI Data Providers” |
| Does Diffbot obey robots.txt? | “At the discretion of Diffbot users.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Diffbot | No |
These facts quote the stored community ai.robots.txt reference.
“Diffbot is a web crawler that extracts and structures website content using AI-powered visual understanding, providing knowledge graph data for applications like market i…”
Its listed role is supplying data for AI tools. Keeping a site available to readers may include allowing search and requested fetches. An owner may still keep the text out of training collections.
How many sites block it?
The 723 files blocking the whole site account for 11.7% of 6,171 readable robots.txt files. We found its name in 553 User-agent lists, a 9.0% share. Without a whole-site Disallow: / block, 5,448 files remain.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 73 files carry the name, or 13.2%, with 102 files blocking the whole site. Blocking is more common among the leading sites, at 18.5%. Farther down, 621 of 5,619 files record a block, 11.1% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Diffbot, all ranks | 553 | 9.0% | 723 | 11.7% |
| Diffbot, top 1,000 | 73 | 13.2% | 102 | 18.5% |
| Diffbot, remaining ranks | 480 | 8.5% | 621 | 11.1% |
These totals cover both groups naming it and rules that apply to every crawler. Of 723 blocks, 213 come only from that general rule, without naming this crawler. The overlap between naming and blocking is 510 files. Naming without a whole-site ban accounts for 43, and 5,618 files never name it. We ignore capitalization and repeated names.
Has that changed since August?
Between August and October, 26 sites that had blocked it stopped and 94 started, so 68 more of the same 5,889 sites block it. A further 583 blocked it in both months, and 5,186 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 635 blocks among 6,236 readable files. October’s was 723 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 94 |
| Stopped blocking | 26 |
| Blocked in both | 583 |
| Blocked in neither | 5,186 |
| Net change | +68 |
How to block or allow it
These lines go in robots.txt at the root. User-agent: * yields to a group naming Diffbot under the robots.txt standard.
Block the whole site
User-agent: Diffbot
Disallow: /Allow access
User-agent: Diffbot
Allow: /It's the 14th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For Diffbot, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. A request timed out after 8 seconds, with files capped at 262,144 bytes. Missing, unreachable and oversized files do not contribute to these shares. Repeat readings are included, using scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
A name identifies the crawler addressed by a group. The survey found 43 files with that name and no recorded whole-site ban, out of 553.
How do the shares compare by rank?
For naming rules, 13.2% of the leading sample mentions it, compared with 9.0% overall. Whole-site blocking is 18.5% at leading ranks and 11.7% across the full sample.
What other entry is close in the index?
The nearby omgili entry reports 519 files that name it and 703 whole-site blocks, 11.4% of readable files. Here, 553 files mention the name and 723 block access, or 11.7%.