ApifyWebsiteContentCrawler
ApifyWebsiteContentCrawler has 51 recorded blocks in October 2026, or 0.8%, and 46 files naming it. Its operator is unclear in the community reference.
Who runs ApifyWebsiteContentCrawler?
| Fact | ApifyWebsiteContentCrawler reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Data Providers” |
| Does ApifyWebsiteContentCrawler obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for ApifyWebsiteContentCrawler | No |
These facts quote the stored community ai.robots.txt reference.
“ApifyWebsiteContentCrawler is a web crawler by Apify that extracts and downloads full website content for use in AI, data analysis, and automation workflows.”
Its listed role is supplying data for AI tools. Keeping a site available to readers may include allowing search and requested fetches. An owner may still keep the text out of training collections.
How many sites block it?
Our readable sample has 51 whole-site blocks, 0.8% of 6,171 files, with Disallow: / as the instruction. Of that full sample, 46 name this crawler, or 0.7%. The count without a whole-site block is 6,120.
For the top 1,000 ranks, its name appears in 7 files out of 552, or 1.3%. Whole-site blocks total 9. Blocking is more common among the leading sites, at 1.6%. Lower in the ranking, blocking reaches 42 files out of 5,619, a 0.7% share.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| ApifyWebsiteContentCrawler, all ranks | 46 | 0.7% | 51 | 0.8% |
| ApifyWebsiteContentCrawler, top 1,000 | 7 | 1.3% | 9 | 1.6% |
| ApifyWebsiteContentCrawler, remaining ranks | 39 | 0.7% | 42 | 0.7% |
Here, the total counts whole-site blocks recorded for this crawler. It leaves out general User-agent: * instructions. Of 51 blocks, 7 do not also appear in our count of files naming it. The counted overlap is 44 files, with 2 files naming it without banning all paths. Missing names account for 6,125 files. Repeating a name or changing its capitalization does not increase the naming total.
How to block or allow it
Use your site’s root file, robots.txt. Specific groups for ApifyWebsiteContentCrawler take precedence over User-agent: *, according to the robots.txt standard.
Block the whole site
User-agent: ApifyWebsiteContentCrawler
Disallow: /Allow access
User-agent: ApifyWebsiteContentCrawler
Allow: /It's the 121st most-named crawler of the 171 we counted in the crawler index.
How we counted
For ApifyWebsiteContentCrawler, we surveyed Tranco’s top 10,000 sites on 19 October 2026. Requests could run for 8 seconds, with a 262,144-byte file limit. Only available files we could read enter the percentages. Collection and repeat readings used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
We found 2 files without a whole-site ban among 46 that name it.
How do the shares compare by rank?
At the top of the ranking, 1.3% of readable files name it and 1.6% block it. Across all ranks, those shares are 0.7% and 0.8%.
What other entry is close in the index?
The nearby xAI-Grok entry reports 46 files that name it and 64 whole-site blocks, 1.0% of readable files. Here, 46 files mention the name and 51 block access, or 0.8%.