Meta-ExternalFetcher
Meta-ExternalFetcher has 509 sites asking it to stay out in October 2026, or 8.2% of the files we could read. However, the reference says it does not obey robots.txt, and its operator is unclear in the community reference.
Who runs Meta-ExternalFetcher?
| Fact | Meta-ExternalFetcher reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Assistants” |
| Does Meta-ExternalFetcher obey robots.txt? | “No” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Meta-ExternalFetcher | No |
These facts quote the stored community ai.robots.txt reference.
“Meta-ExternalFetcher is dispatched by Meta AI products in response to user prompts, when they need to fetch an individual links.”
The reference says this crawler ignores robots.txt. Indexing helps readers find pages, and a user’s request can fetch one. Training reuses the material in models, which an owner may choose to block.
How many sites block it?
Our readable sample has 509 whole-site blocks, 8.2% of 6,171 files, with Disallow: / as the instruction. Of that full sample, 298 name this crawler, or 4.8%. The count without a whole-site block is 5,662.
At the top 1,000 ranks, we could read 552 files. Naming covers 47, a 8.5% share, and blocking covers 76. Blocking is more common among the leading sites, at 13.8%. Elsewhere, 5,619 readable files give 433 blocks, or 7.7%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Meta-ExternalFetcher, all ranks | 298 | 4.8% | 509 | 8.2% |
| Meta-ExternalFetcher, top 1,000 | 47 | 8.5% | 76 | 13.8% |
| Meta-ExternalFetcher, remaining ranks | 251 | 4.5% | 433 | 7.7% |
A general User-agent: * ban contributes to this total whenever it applies to this crawler. Of 509 blocks, 266 come only from that general rule, without naming this crawler. The counted overlap is 243 files, with 55 files naming it without banning all paths. Missing names account for 5,873 files. Repeating a name or changing its capitalization does not increase the naming total.
Has that changed since August?
Between August and October, 42 sites that had blocked it stopped and 29 started, so 13 fewer of the same 5,889 sites block it. A further 449 blocked it in both months, and 5,369 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 515 blocks among 6,236 readable files. October’s was 509 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 29 |
| Stopped blocking | 42 |
| Blocked in both | 449 |
| Blocked in neither | 5,369 |
| Net change | -13 |
How to block or allow it
The examples include the spelling variants counted together here. Put these in robots.txt at the site root. Per the robots.txt standard, a group naming Meta-ExternalFetcher takes precedence over User-agent: *.
Block the whole site
User-agent: Meta-ExternalFetcher
User-agent: meta-externalfetcher
Disallow: /Allow access
User-agent: Meta-ExternalFetcher
User-agent: meta-externalfetcher
Allow: /It's the 28th most-named crawler of the 171 we counted in the crawler index.
How we counted
For Meta-ExternalFetcher, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. An 8-second wait or a file over 262,144 bytes ended a read. Missing, unreachable and oversized files do not contribute to these shares. Later readings are included in results from scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
The 298 files naming it include 55 files without a whole-site ban.
What do the top-ranked sites show?
Within readable robots.txt files near the top of the ranking, 8.5% name it and 13.8% block it. Overall, naming reaches 4.8% and blocking 8.2%.
Which nearby entry can I compare?
Scrapy is named by 287 files and blocked by 277, or 4.5%. For this crawler, 298 files name it and 509 block it, a 8.2% share.