ApifyBot
Few readable files block ApifyBot in October 2026, at 46 sites or 0.7%. Its operator is unclear in the community reference.
Who runs ApifyBot?
| Fact | ApifyBot reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Data Providers” |
| Does ApifyBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for ApifyBot | No |
These facts quote the stored community ai.robots.txt reference.
“ApifyBot is a web scraping and data extraction crawler by Apify that collects website content for use in AI, LLMs, RAG, and automation workflows.”
Its listed role is supplying data for AI tools. An owner who wants search traffic may allow indexing and visits requested by users, yet decline to have the same content gathered for training datasets.
How many sites block it?
A User-agent line names it in 49 files, 0.8% of 6,171 readable files. Whole-site bans, written Disallow: /, total 46, or 0.7%. That leaves 6,125 without a recorded ban.
Among 552 readable files from the top 1,000 sites, we found 8 files naming it, a 1.4% share, with 8 files blocking the whole site. Blocking is more common among the leading sites, at 1.4%. Outside this group, 5,619 files contribute 38 blocks, or 0.7%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| ApifyBot, all ranks | 49 | 0.8% | 46 | 0.7% |
| ApifyBot, top 1,000 | 8 | 1.4% | 8 | 1.4% |
| ApifyBot, remaining ranks | 41 | 0.7% | 38 | 0.7% |
For this entry, we counted whole-site bans attached to its own name, leaving general rules for all crawlers out of the total. Of 46 blocks, 1 does not also appear in our count of files naming it. The counted overlap is 45 files, with 4 files naming it without banning all paths. Missing names account for 6,122 files. Repeating a name or changing its capitalization does not increase the naming total.
How to block or allow it
Put these in robots.txt at the site root. Per the robots.txt standard, a group naming ApifyBot takes precedence over User-agent: *.
Block the whole site
User-agent: ApifyBot
Disallow: /Allow access
User-agent: ApifyBot
Allow: /It's the 119th most-named crawler of the 171 we counted in the crawler index.
How we counted
For ApifyBot, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. Files had an 8-second timeout and a 262,144-byte limit. Only available files we could read enter the percentages. Repeat readings are included, using scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
A name identifies the crawler addressed by a group. The survey found 4 files with that name and no recorded whole-site ban, out of 49.
What do the top-ranked sites show?
Its name appears in 1.4% of readable leading files and 0.8% overall. Blocking affects 1.4% in the leading group and 0.7% in all ranks.
Which nearby entry can I compare?
The nearby LCC entry reports 48 files that name it and 46 whole-site blocks, 0.7% of readable files. Here, 49 files mention the name and 46 block access, or 0.7%.