Diffbot-User
Diffbot-User fetches pages in response to users, according to the reference, and the community reference lists its operator as Diffbot. In October 2026, 513 sites block it, or 8.3% of readable robots.txt files.
Who runs Diffbot-User?
| Fact | Diffbot-User reference entry |
|---|---|
| Operator | “Diffbot” |
| Purpose | “AI Assistants” |
| Does Diffbot-User obey robots.txt? | “Yes” |
| Frequency | “Only when prompted by a user.” |
| Cloudflare managed list for Diffbot-User | No |
These facts quote the stored community ai.robots.txt reference.
“Diffbot-User is used by requests originating on behalf of a human user browsing a URL using Diffbot software, in response to their input. Documented by Diffbot at https://docs.diffbot.com/docs/does-crawl-respect-robotstxt”
Here, the reference describes visits triggered by a user. An owner may permit an assistant to read a page for someone and deny bulk collection that would use it as training material.
How many sites block it?
Instructions to stay out of the whole site appear in 513 files, or 8.3% of 6,171 readable files. We counted 28 with its name in a User-agent line, a 0.5% share. Another 5,658 lack a recorded Disallow: / ban.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 8 files carry the name, or 1.4%, with 67 files blocking the whole site. Blocking is more common among the leading sites, at 12.1%. Farther down, 446 of 5,619 files record a block, 7.9% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Diffbot-User, all ranks | 28 | 0.5% | 513 | 8.3% |
| Diffbot-User, top 1,000 | 8 | 1.4% | 67 | 12.1% |
| Diffbot-User, remaining ranks | 20 | 0.4% | 446 | 7.9% |
For this entry, we counted whole-site bans attached to its own name, leaving general rules for all crawlers out of the total. Of 513 blocks, 485 do not also appear in our count of files naming it. 28 files have the name plus a whole-site ban. Outside that set, 6,143 files never name it. Repeated user-agent mentions or capitalization variants are counted once.
How to block or allow it
Add these lines to robots.txt at your site root. For Diffbot-User, the robots.txt standard gives a named group priority over User-agent: *.
Block the whole site
User-agent: Diffbot-User
Disallow: /Allow access
User-agent: Diffbot-User
Allow: /It's the 142nd most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For Diffbot-User, our 19 October 2026 collection used Tranco’s top 10,000 sites. Requests could run for 8 seconds, with a 262,144-byte file limit. Shares exclude unreachable, unavailable and oversized files. scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) parsed both initial and repeat readings. See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
Here, all 28 files that name it also contain a whole-site ban.
What do the top-ranked sites show?
Within readable robots.txt files near the top of the ranking, 1.4% name it and 12.1% block it. Overall, naming reaches 0.5% and blocking 8.3%.
Which nearby entry can I compare?
The nearby LAIONDownloader entry reports 28 files that name it and 34 whole-site blocks, 0.6% of readable files. Here, 28 files mention the name and 513 block access, or 8.3%.