Poseidon Research Crawler
Poseidon Research Crawler is listed as a search crawler, and the community reference lists its operator as Poseidon Research. In October 2026, 0 sites block it, a 0.0% share of readable files.
Who runs Poseidon Research Crawler?
| Fact | Poseidon Research Crawler reference entry |
|---|---|
| Operator | “Poseidon Research” |
| Purpose | “AI research crawler” |
| Does Poseidon Research Crawler obey robots.txt? | “Unclear at this time.” |
| Frequency | “No explicit frequency provided.” |
| Cloudflare managed list for Poseidon Research Crawler | No |
These facts quote the stored community ai.robots.txt reference.
“Lab focused on scaling the interpretability research necessary to make better AI systems possible.”
The listed use here is search. Keeping a site available to readers may include allowing search and requested fetches. An owner may still keep the text out of training collections.
How many sites block it?
No whole-site Disallow: / ban was recorded for it among 6,171 readable files. User-agent lines name it in 42, or 0.7%. The total without a recorded ban is 6,171.
From the top 1,000 sites, 552 files could be read, and 5 name it, or 0.9%. Below them, 37 of 5,619 readable files name it, 0.7% of that group. Recorded blocking is absent from both groups.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Poseidon Research Crawler, all ranks | 42 | 0.7% | 0 | 0.0% |
| Poseidon Research Crawler, top 1,000 | 5 | 0.9% | 0 | 0.0% |
| Poseidon Research Crawler, remaining ranks | 37 | 0.7% | 0 | 0.0% |
For this entry, we counted whole-site bans attached to its own name, leaving general rules for all crawlers out of the total. The overlap between naming and blocking is 0 files. Naming without a whole-site ban accounts for 42, and 6,129 files never name it. We ignore capitalization and repeated names.
How to block or allow it
These lines go in robots.txt at the root. User-agent: * yields to a group naming Poseidon Research Crawler under the robots.txt standard.
Block the whole site
User-agent: Poseidon Research Crawler
Disallow: /Allow access
User-agent: Poseidon Research Crawler
Allow: /It's the 128th most-named crawler of the 171 we counted in the crawler index.
How we counted
For Poseidon Research Crawler, we surveyed Tranco’s top 10,000 sites on 19 October 2026. Each file had 8 seconds to arrive, up to 262,144 bytes. Missing, unreachable and oversized files do not contribute to these shares. scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) parsed both initial and repeat readings. See the full survey method.
Common questions
Can a file name it and still allow it?
The 42 files that mention its name do not all record a whole-site ban. Those without one total 42.
How do the shares compare by rank?
Leading files record 0.9% with the name and 0.0% blocking it. Across all readable files, the corresponding shares are 0.7% and 0.0%.
What other entry is close in the index?
For comparison, Linguee Bot has 41 files naming it and 0 blocks, or 0.0%. Our counts for this entry are 42 files with the name and 0 blocking the whole site, or 0.0%.