Meta-ExternalAgent

By Matt Kerr·Published 11 November 2026

Meta-ExternalAgent collects content for model training, and the community reference lists its operator as Meta. Our October 2026 survey found 738 sites blocking it, or 12.0% of the files we could read.

Who runs Meta-ExternalAgent?

These facts quote the stored community ai.robots.txt reference.

“"The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly."”

Model training is this crawler’s listed use. Reading a page to answer a user has a different purpose from building a model with it. Owners may allow the requested visit.

How many sites block it?

Of 6,171 robots.txt files we could read, 738 block the whole site with Disallow: /, or 12.0%. Another way to read the sample is by name, with 676 files naming it in a User-agent line, or 11.0%. The 5,433 remaining files have no recorded whole-site block.

From the highest 1,000 ranks, 552 files could be read. Its naming count is 87, or 15.8%, and the block count is 100. Blocking is more common among the leading sites, at 18.1%. Compare 638 blocks among the 5,619 readable files below those ranks, or 11.4%.

Both named groups and general User-agent: * groups can contribute to blocking the whole site. Of 738 blocks, 202 come only from that general rule, without naming this crawler. Looking just at named files gives 536 blocks and 140 without a whole-site block. For 5,495 readable files, we found no mention by name. We ignore capitalization and repeated names.

Has that changed since August?

Between August and October, 190 sites that had blocked it stopped and 61 started, so 129 fewer of the same 5,889 sites block it. A further 635 blocked it in both months, and 5,003 did so in neither. Keeping the sites the same lets us count changes in their rules.

August’s full sample had 931 blocks among 6,236 readable files. October’s was 738 among 6,171. Those monthly totals also reflect which files we could read on each visit.

How to block or allow it

The examples include the spelling variants counted together here. Place the group in your root robots.txt file. Per the robots.txt standard, a named group for Meta-ExternalAgent overrides User-agent: *.

Block the whole site

User-agent: Meta-ExternalAgent
User-agent: meta-externalagent
Disallow: /

Allow access

User-agent: Meta-ExternalAgent
User-agent: meta-externalagent
Allow: /

It's the 8th most-named crawler of the 171 we counted in the crawler index.

Which other crawlers does this operator run?

• FacebookBot
• facebookexternalhit
• meta-webindexer

How we counted

For Meta-ExternalAgent, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. Files had an 8-second timeout and a 262,144-byte limit. Only available files we could read enter the percentages. Reading files again used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) as well. See the full survey method.