TwinAgent
Few readable files block TwinAgent in October 2026, at 53 sites or 0.9%. The community reference lists its operator as Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa….
Who runs TwinAgent?
| Fact | TwinAgent reference entry |
|---|---|
| Operator | “Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa…” |
| Purpose | “AI Agents” |
| Does TwinAgent obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for TwinAgent | No |
These facts quote the stored community ai.robots.txt reference.
“TwinAgent is operated by Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa…”
The reference places this crawler among AI agents. Reading a page to answer a user has a different purpose from building a model with it. Owners may allow the requested visit.
How many sites block it?
A User-agent line names it in 55 files, 0.9% of 6,171 readable files. Whole-site bans, written Disallow: /, total 53, or 0.9%. That leaves 6,118 without a recorded ban.
Breaking out the leading 1,000 sites gives 552 readable files, with 18 naming the crawler, or 3.3%. Whole-site blocking appears in 18 of them. Blocking is more common among the leading sites, at 3.3%. Beyond those ranks, 5,619 files have a blocking total of 35, 0.6% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| TwinAgent, all ranks | 55 | 0.9% | 53 | 0.9% |
| TwinAgent, top 1,000 | 18 | 3.3% | 18 | 3.3% |
| TwinAgent, remaining ranks | 37 | 0.7% | 35 | 0.6% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. Of 53 blocks, 1 does not also appear in our count of files naming it. The overlap between naming and blocking is 52 files. Naming without a whole-site ban accounts for 3, and 6,116 files never name it. We ignore capitalization and repeated names.
How to block or allow it
Edit the root robots.txt file on your site. The robots.txt standard gives a TwinAgent group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: TwinAgent
Disallow: /Allow access
User-agent: TwinAgent
Allow: /It's the 114th most-named crawler of the 171 we counted in the crawler index.
How we counted
For TwinAgent, our 19 October 2026 collection used Tranco’s top 10,000 sites. Files had an 8-second timeout and a 262,144-byte limit. Missing, unreachable and oversized files do not contribute to these shares. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
The 55 files naming it include 3 files without a whole-site ban.
Are leading sites different?
Its name appears in 3.3% of readable leading files and 0.9% overall. Blocking affects 3.3% in the leading group and 0.9% in all ranks.
How does the nearby crawler compare?
Cloudflare-AutoRAG is named by 54 files and blocked by 52, or 0.8%. For this crawler, 55 files name it and 53 block it, a 0.9% share.