FirecrawlAgent
FirecrawlAgent is blocked by 347 sites in the October 2026 survey, or 5.6% of readable files. The community reference lists its operator as Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications.
Who runs FirecrawlAgent?
| Fact | FirecrawlAgent reference entry |
|---|---|
| Operator | “Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications” |
| Purpose | “AI Data Providers” |
| Does FirecrawlAgent obey robots.txt? | “Yes” |
| Frequency | “No information provided.” |
| Cloudflare managed list for FirecrawlAgent | No |
These facts quote the stored community ai.robots.txt reference.
“FirecrawlAgent is a web crawler operated by Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications.”
Its listed role is supplying data for AI tools. An owner may permit an assistant to read a page for someone and deny bulk collection that would use it as training material.
How many sites block it?
Our readable sample has 347 whole-site blocks, 5.6% of 6,171 files, with Disallow: / as the instruction. Of that full sample, 143 name this crawler, or 2.3%. The count without a whole-site block is 5,824.
From the highest 1,000 ranks, 552 files could be read. Its naming count is 18, or 3.3%, and the block count is 53. Blocking is more common among the leading sites, at 9.6%. Compare 294 blocks among the 5,619 readable files below those ranks, or 5.2%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| FirecrawlAgent, all ranks | 143 | 2.3% | 347 | 5.6% |
| FirecrawlAgent, top 1,000 | 18 | 3.3% | 53 | 9.6% |
| FirecrawlAgent, remaining ranks | 125 | 2.2% | 294 | 5.2% |
These totals cover both groups naming it and rules that apply to every crawler. Of 347 blocks, 226 come only from that general rule, without naming this crawler. The counted overlap is 121 files, with 22 files naming it without banning all paths. Missing names account for 6,028 files. Repeating a name or changing its capitalization does not increase the naming total.
Has that changed since August?
Between August and October, 16 sites that had blocked it stopped and 42 started, so 26 more of the same 5,889 sites block it. A further 273 blocked it in both months, and 5,558 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 310 blocks among 6,236 readable files. October’s was 347 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 42 |
| Stopped blocking | 16 |
| Blocked in both | 273 |
| Blocked in neither | 5,558 |
| Net change | +26 |
How to block or allow it
Use your site’s root file, robots.txt. Specific groups for FirecrawlAgent take precedence over User-agent: *, according to the robots.txt standard.
Block the whole site
User-agent: FirecrawlAgent
Disallow: /Allow access
User-agent: FirecrawlAgent
Allow: /It's the 49th most-named crawler of the 171 we counted in the crawler index.
How we counted
For FirecrawlAgent, we surveyed Tranco’s top 10,000 sites on 19 October 2026. Files had an 8-second timeout and a 262,144-byte limit. Only available files we could read enter the percentages. The file readings, including repeats, use scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
We found 22 files without a whole-site ban among 143 that name it.
How do the shares compare by rank?
Its name appears in 3.3% of readable leading files and 2.3% overall. Blocking affects 9.6% in the leading group and 5.6% in all ranks.
What other entry is close in the index?
For comparison, ExaBot has 141 files naming it and 334 blocks, or 5.4%. Our counts for this entry are 143 files with the name and 347 blocking the whole site, or 5.6%.