facebookexternalhit
facebookexternalhit has 23 sites asking it to stay out in October 2026, or 0.4% of the files we could read. However, the reference says it does not obey robots.txt, and the community reference lists its operator as Meta/Facebook.
Who runs facebookexternalhit?
| Fact | facebookexternalhit reference entry |
|---|---|
| Operator | “Meta/Facebook” |
| Purpose | “Ostensibly only for sharing, but likely used as an AI crawler as well” |
| Does facebookexternalhit obey robots.txt? | “No” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for facebookexternalhit | No |
These facts quote the stored community ai.robots.txt reference.
“Note that excluding FacebookExternalHit will block incorporating OpenGraph data when sharing in social media, including rich links in Apple's Messages app. According to Meta, its purpose is "to crawl the content of an app or website that was shared on one of Meta’s family of apps…". However, see discussions here and here for evidence to the contrary.”
The reference says this crawler ignores robots.txt. Reading a page to answer a user has a different purpose from building a model with it. Owners may allow the requested visit.
How many sites block it?
A User-agent line names it in 435 files, 7.0% of 6,171 readable files. Whole-site bans, written Disallow: /, total 23, or 0.4%. That leaves 6,148 without a recorded ban.
Breaking out the leading 1,000 sites gives 552 readable files, with 56 naming the crawler, or 10.1%. Whole-site blocking appears in 1 of them. The leading sites have a lower block share, 0.2%. Beyond those ranks, 5,619 files have a blocking total of 22, 0.4% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| facebookexternalhit, all ranks | 435 | 7.0% | 23 | 0.4% |
| facebookexternalhit, top 1,000 | 56 | 10.1% | 1 | 0.2% |
| facebookexternalhit, remaining ranks | 379 | 6.7% | 22 | 0.4% |
This total covers whole-site bans recorded for this name. A blanket User-agent: * rule alone is outside this measure. The split among files that name it is 23 with a whole-site block and 412 without one. Its name is absent from 5,736 readable files. We ignore capitalization and repeated names.
How to block or allow it
These lines go in robots.txt at the root. User-agent: * yields to a group naming facebookexternalhit under the robots.txt standard.
Block the whole site
User-agent: facebookexternalhit
Disallow: /Allow access
User-agent: facebookexternalhit
Allow: /It's the 20th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For facebookexternalhit, our 19 October 2026 collection used Tranco’s top 10,000 sites. Requests could run for 8 seconds, with a 262,144-byte file limit. Shares exclude unreachable, unavailable and oversized files. Repeat readings are included, using scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
The 435 files that mention its name do not all record a whole-site ban. Those without one total 412.
How do the shares compare by rank?
For naming rules, 10.1% of the leading sample mentions it, compared with 7.0% overall. Whole-site blocking is 0.2% at leading ranks and 0.4% across the full sample.
What other entry is close in the index?
For comparison, omgilibot has 395 files naming it and 523 blocks, or 8.5%. Our counts for this entry are 435 files with the name and 23 blocking the whole site, or 0.4%.