Google-Extended on Webflow

By Matt Kerr·Published 12 November 2026

Webflow bot controls says: Webflow gives Google-Extended a specific robots.txt example that excludes /drafts/. Its post does not say Google-Extended is blocked by default. It places the custom file control under Site Settings > SEO.

Choose the folder you want to exclude and inspect your live robots.txt before editing it. Webflow distinguishes this example from traditional Search and Ads products using Googlebot. Keep that distinction when you check the published instructions.

Use the folder scope shown in Webflow’s example

Webflow bot controls says: The post uses a named Google-Extended group followed by Disallow for /drafts/. That path limits the stated request to the folder in the example. It is different in scope from the same post’s GPTBot example, which uses the root path for a whole-site exclusion. Decide which content you mean before copying either instruction into your settings.

The two lines below reproduce the method in Webflow’s post. Treat /drafts/ as an example path and use the address structure your own site publishes. Save the current file before changing it. After the edit, read the public robots.txt and confirm that the intended folder text appears under the intended user-agent name, with the rest of your instructions preserved.

User-agent: Google-Extended
Disallow: /drafts/

Keep Googlebot choices separate

Webflow bot controls says: Webflow describes its Google-Extended example as related to Gemini and AI model training services. It explicitly distinguishes traditional Search and Ads products using Googlebot. This is why a named folder exclusion should be reviewed alongside the existing Googlebot and all-crawler instructions. Do not infer the Googlebot result from the presence of a Google-Extended line alone.

The post also discusses page indexing and the Webflow subdomain indexing checkbox. Those settings have their own scope. For this task, inspect the custom robots.txt under Site Settings > SEO and the published hostname-root file. If you change a page setting as well, record that separately so the effect of a folder instruction can be checked against the file you actually served.

Check the instruction and compare a folder URL

Webflow bot controls says: Read robots.txt from your public domain, then inspect the group for Google-Extended. Compare a URL inside the chosen folder with one outside it. The commands below send a synthetic Google-Extended header to the folder address. They show how your server responds to that string. They do not establish which header a real Google visit uses or whether it applies the instruction.

A successful response from the folder can coexist with a file requesting exclusion. Webflow describes these rules as signals respected by well-behaved crawlers and says they do not immediately remove previously indexed content. Evaluate the edit by reading the instruction at the root of the correct hostname. Keep the page response as a separate observation about that request.

curl -sS -L https://example.com/robots.txt
curl -sS -L -D - -o /dev/null -A 'Mozilla/5.0' https://example.com/drafts/
curl -sS -L -D - -o /dev/null -A 'Google-Extended' https://example.com/drafts/

Reopen a folder deliberately

Webflow bot controls says: If you later want to withdraw this exclusion, review the named group and remove or narrow the folder instruction you added. Keep other bot instructions intact. Re-read the live file and compare both folder and non-folder addresses again. Webflow’s custom file control lets owners choose the user-agent request. It does not promise when an outside service will revisit the content.