Does Netlify block AI bots by default?

By Matt Kerr·Published 12 November 2026

Netlify blocking guide says: Netlify explains how to install and activate User Agent Blocker. The docs do not say whether each of these six bots is blocked by default. They say the extension starts working after the selected project is deployed successfully.

Netlify Edge Function guide says: Check Extensions > User Agent Blocker in the project dashboard. Read the selected traffic under Block User Agents. Netlify also supports your own robots.txt and Edge Function rules, so inspect those if the extension does not explain a refusal.

Rules for each bot

GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot

Netlify blocking guide says: The docs do not say. They describe installing and activating a blocker, without a named-bot starting state. Review Block User Agents in the project extension, custom robots.txt and Edge Functions. Redeploy changes.

Installation and activation are separate steps

Netlify blocking guide says: The blocking guide says to find User Agent Blocker under Extensions and select Install. This makes it available to the team. You then go to a project to activate it. Activation happens one project at a time. A team installation alone does not describe which traffic a particular published site is blocking. Inspect the site whose address you will test.

Under Block User Agents, choose the traffic you intend to refuse. The guide says to select the AI crawler options when targeting AI crawlers and bots, and warns that the list does not include every possible one. Select Redeploy site, confirm with Save, then select Enable analytics and deploy site. Wait for a successful deployment before testing the public address.

Categories tell you what a request looks like

Netlify request categories says: Netlify assigns a Netlify-Agent-Category header to deployed requests. Functions and Edge Functions can read it. GPTBot is explicitly a Crawler, while ChatGPT-User and Claude-User appear as AI Agent examples. A rule that targets only ai-agent does not express the same choice as one targeting crawler. The docs give GPTBot enough separate treatment for the linked guide below.

Netlify sets and overrides this category header on deployed sites. To test a category-based rule on a deployed project, change the User-Agent header. Sending your own Netlify-Agent-Category value to the public site does not choose the category. Local development behaves differently, so keep your public test distinct from a local function check.

Check files and functions alongside the extension

Netlify Edge Function guide says: Netlify also gives an example that writes a robots.txt exclusion for each name in a list. Its Edge Function reads the same list and returns HTTP 401 for matching user-agent text. These are two places to look when the extension is absent or a project has older custom code. The guide says bots can ignore robots.txt or misreport their names.

Netlify blocking guide says: Fetch robots.txt, then use the six request headers below on the same published URL. Compare status codes and the returned location if there is a redirect. If a name is disallowed in the file but the page responds, record both results. To reopen access, review that name in the extension selection, any custom exclusion and any Edge Function before redeploying the project.

curl -sS -L https://example.com/robots.txt
curl -sS -L -D - -o /dev/null -A 'Mozilla/5.0' https://example.com/
curl -sS -L -D - -o /dev/null -A 'GPTBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'ClaudeBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'PerplexityBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'Google-Extended' https://example.com/
curl -sS -L -D - -o /dev/null -A 'Applebot-Extended' https://example.com/
curl -sS -L -D - -o /dev/null -A 'CCBot' https://example.com/

More help for this host

• GPTBot on Netlify