GPTBot on Netlify

By Matt Kerr·Published 12 November 2026

Netlify request categories says: Netlify puts GPTBot in the Crawler category. ChatGPT-User is an AI Agent example. A category rule aimed only at ai-agent therefore does not target GPTBot as described in the category guide.

Netlify blocking guide says: The blocking guide does not state a GPTBot default. Check User Agent Blocker in the project and inspect any Edge Function using Netlify-Agent-Category. Test the deployed site by changing User-Agent, because Netlify overrides a category header sent by a client.

Find which category your function uses

Netlify request categories says: Netlify describes Crawler as bulk indexing and scraping by bots. In its AI Agent section, it explicitly says high-volume GPTBot traffic belongs to Crawler. That distinction matters if your function handles a short chat fetch differently from bulk requests. Read the actual header condition used by the function instead of assuming that the word AI in its purpose means ai-agent in Netlify’s header.

The header has one category and may have one subcategory. The guide shows a condition that runs an Edge Function when the value contains ai-agent or crawler. That example logs the raw User-Agent. It does not block a request itself. A project can add its own action, so inspect what the function returns as well as the condition that makes it run.

Keep deployed and local category tests distinct

Netlify request categories says: On a deployed project, Netlify sets Netlify-Agent-Category and overrides any value a client supplies. Its test instructions say to set User-Agent instead. For GPTBot, send a header containing the documented name and inspect your function’s behavior on the published project. Sending Netlify-Agent-Category directly from curl would test a value Netlify then replaces.

Local development does not set or override that category header. Netlify says to provide it directly when testing a local Function or Edge Function. A successful local test therefore uses a different input method. Record which environment you tested so an apparently working local rule is not treated as a reading of the public project’s current category and action.

Review the blocker before redeploying

Netlify blocking guide and Netlify request categories say: The User Agent Blocker extension has its own project activation steps. Check the selected traffic under Block User Agents. If GPTBot access is unwanted, review the available options and any custom Edge Function condition. Netlify warns that the preset list does not cover every AI bot. The category guide’s GPTBot example confirms its classification, without promising the extension’s current preset membership.

Netlify Edge Function guide says: Netlify’s older blocking guide shows a custom Edge Function returning HTTP 401 for names found in a list. It uses the same list to write robots.txt exclusions. Search that list when an older project refuses GPTBot even though its extension settings do not explain it. To reopen access, inspect both the server action and the published file request before the next deployment.

curl -sS -L https://example.com/robots.txt
curl -sS -L -D - -o /dev/null -A 'Mozilla/5.0' https://example.com/
curl -sS -L -D - -o /dev/null -A 'GPTBot/1.0' https://example.com/