Crawl4AI
Crawl4AI is blocked by 71 sites in the October 2026 survey, or 1.2% of readable files. Its operator is unclear in the community reference.
Who runs Crawl4AI?
| Fact | Crawl4AI reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “Undocumented AI Agents” |
| Does Crawl4AI obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Crawl4AI | No |
These facts quote the stored community ai.robots.txt reference.
The reference leaves this agent’s purpose undocumented. An owner may allow a search crawler to list a page or an assistant to retrieve it for someone, yet decline to supply the text for training.
How many sites block it?
Our readable sample has 71 whole-site blocks, 1.2% of 6,171 files, with Disallow: / as the instruction. Of that full sample, 69 name this crawler, or 1.1%. The count without a whole-site block is 6,100.
For the top 1,000 ranks, its name appears in 19 files out of 552, or 3.4%. Whole-site blocks total 19. Blocking is more common among the leading sites, at 3.4%. Lower in the ranking, blocking reaches 52 files out of 5,619, a 0.9% share.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Crawl4AI, all ranks | 69 | 1.1% | 71 | 1.2% |
| Crawl4AI, top 1,000 | 19 | 3.4% | 19 | 3.4% |
| Crawl4AI, remaining ranks | 50 | 0.9% | 52 | 0.9% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. Of 71 blocks, 5 do not also appear in our count of files naming it. Names and bans overlap in 66 files. We also found 3 files with a name and no whole-site ban, plus 6,102 files omitting the name. Case changes and repeated spellings are merged.
How to block or allow it
Add these lines to robots.txt at your site root. For Crawl4AI, the robots.txt standard gives a named group priority over User-agent: *.
Block the whole site
User-agent: Crawl4AI
Disallow: /Allow access
User-agent: Crawl4AI
Allow: /It's the 88th most-named crawler of the 171 we counted in the crawler index.
How we counted
For Crawl4AI, our 19 October 2026 collection used Tranco’s top 10,000 sites. An 8-second wait or a file over 262,144 bytes ended a read. Missing, unreachable and oversized files do not contribute to these shares. The file readings, including repeats, use scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
A name identifies the crawler addressed by a group. The survey found 3 files with that name and no recorded whole-site ban, out of 69.
What do the top-ranked sites show?
Its name appears in 3.4% of readable leading files and 1.1% overall. Blocking affects 3.4% in the leading group and 1.2% in all ranks.
Which nearby entry can I compare?
For comparison, LinkupBot has 69 files naming it and 68 blocks, or 1.1%. Our counts for this entry are 69 files with the name and 71 blocking the whole site, or 1.2%.