Panscient
Panscient is blocked by 75 sites in the October 2026 survey, or 1.2% of readable files. The community reference lists its operator as Panscient.
Who runs Panscient?
| Fact | Panscient reference entry |
|---|---|
| Operator | “Panscient” |
| Purpose | “Data collection and analysis using machine learning and AI.” |
| Does Panscient obey robots.txt? | “Yes” |
| Frequency | “The Panscient web crawler will request a page at most once every second from the same domain name or the same IP address.” |
| Cloudflare managed list for Panscient | No |
These facts quote the stored community ai.robots.txt reference.
“Compiles data on businesses and business professionals that is structured using AI and machine learning.”
This entry’s stated use is quoted above. An owner may allow a search crawler to list a page or an assistant to retrieve it for someone, yet decline to supply the text for training.
How many sites block it?
A User-agent line names it in 77 files, 1.2% of 6,171 readable files. Whole-site bans, written Disallow: /, total 75, or 1.2%. That leaves 6,096 without a recorded ban.
For the top 1,000 ranks, its name appears in 24 files out of 552, or 4.3%. Whole-site blocks total 24. Blocking is more common among the leading sites, at 4.3%. Lower in the ranking, blocking reaches 51 files out of 5,619, a 0.9% share.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Panscient, all ranks | 77 | 1.2% | 75 | 1.2% |
| Panscient, top 1,000 | 24 | 4.3% | 24 | 4.3% |
| Panscient, remaining ranks | 53 | 0.9% | 51 | 0.9% |
This total covers whole-site bans recorded for this name. A blanket User-agent: * rule alone is outside this measure. The naming total includes 75 files with a ban on every path and 2 files without that ban. Elsewhere, 6,094 files do not name it. We ignore capitalization and repeated names.
How to block or allow it
The examples include the spelling variants counted together here. Put these in robots.txt at the site root. Per the robots.txt standard, a group naming Panscient takes precedence over User-agent: *.
Block the whole site
User-agent: Panscient
User-agent: panscient.com
Disallow: /Allow access
User-agent: Panscient
User-agent: panscient.com
Allow: /It's the 76th most-named crawler of the 171 we counted in the crawler index.
How we counted
For Panscient, our 19 October 2026 collection used Tranco’s top 10,000 sites. A request timed out after 8 seconds, with files capped at 262,144 bytes. Only available files we could read enter the percentages. Collection and repeat readings used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
The 77 files naming it include 2 files without a whole-site ban.
Are leading sites different?
At the top of the ranking, 4.3% of readable files name it and 4.3% block it. Across all ranks, those shares are 1.2% and 1.2%.
How does the nearby crawler compare?
YaK is named by 77 files and blocked by 71, or 1.2%. For this crawler, 77 files name it and 75 block it, a 1.2% share.