meta-webindexer
meta-webindexer is blocked by 154 sites in the October 2026 survey, or 2.5% of readable files. The community reference lists its operator as Meta.
Who runs meta-webindexer?
| Fact | meta-webindexer reference entry |
|---|---|
| Operator | “Meta” |
| Purpose | “AI Assistants” |
| Does meta-webindexer obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unhinged, more than 1 per second.” |
| Cloudflare managed list for meta-webindexer | No |
These facts quote the stored community ai.robots.txt reference.
“As per their documentation, "The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users. In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI. Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."”
The stated role here is an AI assistant. Allowing people to find and read a page need not include permission to gather it for model training. Owners can express those choices in crawler-specific groups.
How many sites block it?
Within 6,171 readable robots.txt files, the name appears in 174, or 2.8%. Instructions blocking every path, Disallow: /, appear in 154, a 2.5% share. No whole-site ban is recorded for 6,017 files.
At the top 1,000 ranks, we could read 552 files. Naming covers 30, a 5.4% share, and blocking covers 26. Blocking is more common among the leading sites, at 4.7%. Elsewhere, 5,619 readable files give 128 blocks, or 2.3%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| meta-webindexer, all ranks | 174 | 2.8% | 154 | 2.5% |
| meta-webindexer, top 1,000 | 30 | 5.4% | 26 | 4.7% |
| meta-webindexer, remaining ranks | 144 | 2.6% | 128 | 2.3% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. Of 154 blocks, 15 do not also appear in our count of files naming it. The split among files that name it is 139 with a whole-site block and 35 without one. Its name is absent from 5,997 readable files. We ignore capitalization and repeated names.
How to block or allow it
Place the group in your root robots.txt file. Per the robots.txt standard, a named group for meta-webindexer overrides User-agent: *.
Block the whole site
User-agent: meta-webindexer
Disallow: /Allow access
User-agent: meta-webindexer
Allow: /It's the 45th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For meta-webindexer, our 19 October 2026 collection used Tranco’s top 10,000 sites. An 8-second wait or a file over 262,144 bytes ended a read. Missing, unreachable and oversized files do not contribute to these shares. scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) parsed both initial and repeat readings. See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
A name identifies the crawler addressed by a group. The survey found 35 files with that name and no recorded whole-site ban, out of 174.
What do the top-ranked sites show?
For naming rules, 5.4% of the leading sample mentions it, compared with 2.8% overall. Whole-site blocking is 4.7% at leading ranks and 2.5% across the full sample.
Which nearby entry can I compare?
For comparison, Kangaroo Bot has 172 files naming it and 0 blocks, or 0.0%. Our counts for this entry are 174 files with the name and 154 blocking the whole site, or 2.5%.