FriendlyCrawler
FriendlyCrawler is blocked by 196 sites in the October 2026 survey, or 3.2% of readable files. The community reference lists its operator as Unknown.
Who runs FriendlyCrawler?
| Fact | FriendlyCrawler reference entry |
|---|---|
| Operator | “Unknown” |
| Purpose | “We are using the data from the crawler to build datasets for machine learning experiments.” |
| Does FriendlyCrawler obey robots.txt? | “Yes” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for FriendlyCrawler | No |
These facts quote the stored community ai.robots.txt reference.
“Unclear who the operator is; but data is used for training/machine learning.”
This entry’s stated use is quoted above. An owner may allow a search crawler to list a page or an assistant to retrieve it for someone, yet decline to supply the text for training.
How many sites block it?
A User-agent line names it in 201 files, 3.3% of 6,171 readable files. Whole-site bans, written Disallow: /, total 196, or 3.2%. That leaves 5,975 without a recorded ban.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 41 files carry the name, or 7.4%, with 39 files blocking the whole site. Blocking is more common among the leading sites, at 7.1%. Farther down, 157 of 5,619 files record a block, 2.8% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| FriendlyCrawler, all ranks | 201 | 3.3% | 196 | 3.2% |
| FriendlyCrawler, top 1,000 | 41 | 7.4% | 39 | 7.1% |
| FriendlyCrawler, remaining ranks | 160 | 2.8% | 157 | 2.8% |
Here, the total counts whole-site blocks recorded for this crawler. It leaves out general User-agent: * instructions. Of 196 blocks, 4 do not also appear in our count of files naming it. The naming total includes 192 files with a ban on every path and 9 files without that ban. Elsewhere, 5,970 files do not name it. We ignore capitalization and repeated names.
How to block or allow it
Place the group in your root robots.txt file. Per the robots.txt standard, a named group for FriendlyCrawler overrides User-agent: *.
Block the whole site
User-agent: FriendlyCrawler
Disallow: /Allow access
User-agent: FriendlyCrawler
Allow: /It's the 39th most-named crawler of the 171 we counted in the crawler index.
How we counted
For FriendlyCrawler, our 19 October 2026 collection used Tranco’s top 10,000 sites. A request timed out after 8 seconds, with files capped at 262,144 bytes. Only available files we could read enter the percentages. scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) parsed both initial and repeat readings. See the full survey method.
Common questions
What does naming this crawler tell me?
The 201 files naming it include 9 files without a whole-site ban.
Are leading sites different?
Leading files record 7.4% with the name and 7.1% blocking it. Across all readable files, the corresponding shares are 3.3% and 3.2%.
How does the nearby crawler compare?
The nearby img2dataset entry reports 200 files that name it and 181 whole-site blocks, 2.9% of readable files. Here, 201 files mention the name and 196 block access, or 3.2%.