Devin
Devin is blocked by 69 sites in the October 2026 survey, or 1.1% of readable files. The community reference lists its operator as Devin AI.
Who runs Devin?
| Fact | Devin reference entry |
|---|---|
| Operator | “Devin AI” |
| Purpose | “AI Coding Agents” |
| Does Devin obey robots.txt? | “Yes” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Devin | No |
These facts quote the stored community ai.robots.txt reference.
“Devin is a software engineering AI assistant that can browse websites and perform web-based tasks, functioning as a collaborative AI teammate for engineering teams.”
The listed coding role concerns tools that help with software work. Reading a page to answer a user has a different purpose from building a model with it. Owners may allow the requested visit.
How many sites block it?
A User-agent line names it in 79 files, 1.3% of 6,171 readable files. Whole-site bans, written Disallow: /, total 69, or 1.1%. That leaves 6,102 without a recorded ban.
At the top 1,000 ranks, we could read 552 files. Naming covers 22, a 4.0% share, and blocking covers 21. Blocking is more common among the leading sites, at 3.8%. Elsewhere, 5,619 readable files give 48 blocks, or 0.9%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Devin, all ranks | 79 | 1.3% | 69 | 1.1% |
| Devin, top 1,000 | 22 | 4.0% | 21 | 3.8% |
| Devin, remaining ranks | 57 | 1.0% | 48 | 0.9% |
This is a count of blocks attached to the named crawler, covering the whole site. Blanket rules for every crawler do not enter it. Names and bans overlap in 69 files. We also found 10 files with a name and no whole-site ban, plus 6,092 files omitting the name. Case changes and repeated spellings are merged.
How to block or allow it
These lines go in robots.txt at the root. User-agent: * yields to a group naming Devin under the robots.txt standard.
Block the whole site
User-agent: Devin
Disallow: /Allow access
User-agent: Devin
Allow: /It's the 74th most-named crawler of the 171 we counted in the crawler index.
How we counted
For Devin, our 19 October 2026 collection used Tranco’s top 10,000 sites. Requests could run for 8 seconds, with a 262,144-byte file limit. Missing, unreachable and oversized files do not contribute to these shares. Collection and repeat readings used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
The 79 files that mention its name do not all record a whole-site ban. Those without one total 10.
How do the shares compare by rank?
At the top of the ranking, 4.0% of readable files name it and 3.8% block it. Across all ranks, those shares are 1.3% and 1.1%.
What other entry is close in the index?
Google-NotebookLM is named by 79 files and blocked by 278, or 4.5%. For this crawler, 79 files name it and 69 block it, a 1.1% share.