Does Webflow block AI bots by default?

By Matt Kerr·Published 12 November 2026

Webflow bot controls says: Webflow’s bot-control article does not state whether these six bots are blocked by default. It says its Cloudflare work brings no immediate changes to Webflow sites. Do not apply Cloudflare’s new-domain policy to a Webflow site from that announcement alone.

Webflow puts a custom robots.txt under Site Settings > SEO. Its article gives a whole-site GPTBot example and a Google-Extended example for /drafts/. Read your published file and choose the paths you mean to restrict.

Rules for each bot

GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot

Webflow bot controls says: The docs do not say. The Webflow announcement says there are no immediate changes to hosted sites. Inspect custom robots.txt under Site Settings > SEO and edit the intended user-agent instructions.

What the Cloudflare announcement says for Webflow

Webflow bot controls says: Webflow discusses working with Cloudflare on future control over AI access. It explicitly says there are no immediate changes for Webflow sites. The post then points readers back to existing SEO controls. That wording does not announce an automatic firewall block for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended or CCBot on Webflow hosting. The table records the missing starting-state information for each name.

The custom robots.txt control accepts instructions by user-agent name. Webflow uses GPTBot with Disallow for the whole site as one example. It uses Google-Extended with Disallow for /drafts/ as another. Those are instructions an owner can choose to add. They do not show that every published Webflow file already contains either exclusion. Read the address that serves your content.

Use the right SEO control

Webflow bot controls says: Site Settings > SEO holds the custom robots.txt text. Before changing it, save the current public file and identify the group you want to edit. For a GPTBot whole-site exclusion, Webflow gives the two lines below. If reopening access is your aim, review the exclusion for that name and any broader instruction before editing. Read the live file again after the change.

User-agent: GPTBot
Disallow: /

Keep subdomain and page indexing choices in view

Webflow bot controls says: The post also describes Disable Webflow subdomain indexing in Site Settings and Index this page in Page Settings. It presents these as indexing controls. Pay attention to their scope. A setting named for the Webflow subdomain does not describe every custom-domain address. A page setting concerns that page. The bot examples are in custom robots.txt and use named instructions.

For Google-Extended, the post uses /drafts/ to show a limited exclusion. The separate guide linked below walks through checking that path while preserving your intended access elsewhere. It also keeps Googlebot distinct, as Webflow’s example does. The four other names have no separate recipe in this post and use the shared inspection steps here.

Compare the public file with a page fetch

Webflow bot controls says: Replace example.com with the exact domain you want to check. Read /robots.txt at its root. Then send the browser and six named headers below to a public page. Treat these as tests of strings you supplied, including the two Extended names. A successful response does not prove the operator uses that string for its real visits or follows the file.

Webflow says the rules are respected by most well-behaved crawlers and do not immediately remove content already indexed. Check the text served now when deciding whether your edit took effect. If your question concerns earlier copies or search listings, the current HTTP response cannot answer when those outside systems will update. Keep the URL, request header and file text together for a later comparison.

curl -sS -L https://example.com/robots.txt
curl -sS -L -D - -o /dev/null -A 'Mozilla/5.0' https://example.com/
curl -sS -L -D - -o /dev/null -A 'GPTBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'ClaudeBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'PerplexityBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'Google-Extended' https://example.com/
curl -sS -L -D - -o /dev/null -A 'Applebot-Extended' https://example.com/
curl -sS -L -D - -o /dev/null -A 'CCBot' https://example.com/

More help for this host

• Google-Extended on Webflow