Does Cloudflare block AI bots by default?

By Matt Kerr·Published 12 November 2026

Cloudflare access policies, Cloudflare traffic announcement and Cloudflare changelog say: Cloudflare gives two answers for new domains. Its access policy docs say Search, Agents and Training start at Allow. Its blog and changelog say Training and Agent bots start blocked on pages with ads.

Check your domain in Security > Settings > AI Crawler and Agent access policies. Read the action for each behavior before changing robots.txt. The policy docs offer both server blocks and a Training request in robots.txt.

Rules for each bot

GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot

Cloudflare access policies, Cloudflare traffic announcement and Cloudflare changelog say: Cloudflare's docs name none of these six bots, so none has a bot-specific default. The access policy docs say new domains start at Allow for Search, Agents and Training. The blog and changelog say Training and Agent bots start blocked on pages with ads. Check the bot behavior and saved action in Security > Settings > AI Crawler and Agent access policies.

Why the published answers differ

Cloudflare access policies and Cloudflare traffic announcement say: The policy page was updated on 2 November 2026. It says all three behaviors use Allow for new domains. The blog, dated 1 July 2026, announces a different start from 15 September 2026. It says Training and Agent traffic will be blocked on ad pages and Search will stay allowed. Both pages were read on 12 November 2026. Neither reading explains why those answers differ.

Cloudflare changelog says: The changelog repeats the ad-page policy and its September start date. The blog also says a crawler with both Search and Training uses follows the most restrictive rule. It names Googlebot, Applebot and BingBot in that discussion. Those names do not give a separate setting for Google-Extended or Applebot-Extended. The saved policy page describes behaviors without assigning each of our six names to them.

Choose an action in your own domain

Cloudflare access policies says: In the dashboard, open Security, then Settings. Filter by Bot traffic and select AI Crawler and Agent access policies. Read the current action under Search, Agent and Training. Save a copy of those choices with the date before you edit them. This lets you compare the control you changed with the result you see on your public site.

Block refuses requests across the zone. Block on pages with ads applies where Cloudflare detects ads. Allow does not apply that block. Disallow is offered only for Training and puts a request in robots.txt. With Bot Preference Sync enabled, Disallow also blocks AI training crawlers at the edge, except mixed-purpose Search and Training crawlers. Check that option when a file edit and a server response seem to disagree.

Read your public file and compare responses

Cloudflare access policies says: Use your own address in the commands below. Fetch robots.txt first, then compare the same public URL with a browser header and each named bot header. Save the response headers. These are checks of the names you send. Typing a bot name in curl does not show how Cloudflare will handle a verified visit from its real operator. The policy covers Verified bots and matching unverified bots.

Look for both a named group and the group for all crawlers in robots.txt. A Disallow request can exist alongside an Allow server policy. A successful page request can exist alongside a robots.txt request to stay out. Review the file and the dashboard together before deciding that one change fixed access. Repeat the same commands after saving your policy.

curl -sS -L https://example.com/robots.txt
curl -sS -L -D - -o /dev/null -A 'Mozilla/5.0' https://example.com/
curl -sS -L -D - -o /dev/null -A 'GPTBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'ClaudeBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'PerplexityBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'Google-Extended' https://example.com/
curl -sS -L -D - -o /dev/null -A 'Applebot-Extended' https://example.com/
curl -sS -L -D - -o /dev/null -A 'CCBot' https://example.com/