Perplexity-User
Perplexity-User has 440 sites asking it to stay out in October 2026, or 7.1% of the files we could read. However, the reference says it does not obey robots.txt, and the community reference lists its operator as Perplexity.
Who runs Perplexity-User?
| Fact | Perplexity-User reference entry |
|---|---|
| Operator | “Perplexity” |
| Purpose | “AI Assistants” |
| Does Perplexity-User obey robots.txt? | “No” |
| Frequency | “Only when prompted by a user.” |
| Cloudflare managed list for Perplexity-User | No |
These facts quote the stored community ai.robots.txt reference.
“Perplexity-User supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a …”
The reference says this crawler ignores robots.txt. Search referrals or a user’s question can give an owner a reason to allow access while declining collection of page text for training.
How many sites block it?
Of 6,171 robots.txt files we could read, 440 block the whole site with Disallow: /, or 7.1%. Another way to read the sample is by name, with 383 files naming it in a User-agent line, or 6.2%. The 5,731 remaining files have no recorded whole-site block.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 63 files carry the name, or 11.4%, with 77 files blocking the whole site. Blocking is more common among the leading sites, at 13.9%. Farther down, 363 of 5,619 files record a block, 6.5% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Perplexity-User, all ranks | 383 | 6.2% | 440 | 7.1% |
| Perplexity-User, top 1,000 | 63 | 11.4% | 77 | 13.9% |
| Perplexity-User, remaining ranks | 320 | 5.7% | 363 | 6.5% |
A general User-agent: * ban contributes to this total whenever it applies to this crawler. Of 440 blocks, 205 come only from that general rule, without naming this crawler. We found 235 files naming it and blocking the whole site, along with 148 files naming it without that ban. Another 5,788 files omit the name. We ignore capitalization and repeated names.
Has that changed since August?
Between August and October, 22 sites that had blocked it stopped and 23 started, so 1 more of the same 5,889 sites block it. A further 388 blocked it in both months, and 5,456 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 434 blocks among 6,236 readable files. October’s was 440 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 23 |
| Stopped blocking | 22 |
| Blocked in both | 388 |
| Blocked in neither | 5,456 |
| Net change | +1 |
How to block or allow it
Edit the root robots.txt file on your site. The robots.txt standard gives a Perplexity-User group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: Perplexity-User
Disallow: /Allow access
User-agent: Perplexity-User
Allow: /It's the 24th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For Perplexity-User, we surveyed Tranco’s top 10,000 sites on 19 October 2026. A request timed out after 8 seconds, with files capped at 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
A name identifies the crawler addressed by a group. The survey found 148 files with that name and no recorded whole-site ban, out of 383.
Are leading sites different?
For naming rules, 11.4% of the leading sample mentions it, compared with 6.2% overall. Whole-site blocking is 13.9% at leading ranks and 7.1% across the full sample.
How does the nearby crawler compare?
For comparison, Timpibot has 349 files naming it and 550 blocks, or 8.9%. Our counts for this entry are 383 files with the name and 440 blocking the whole site, or 7.1%.