AI crawlers / Hosts / Vercel

Does Vercel block AI bots by default?

By Matt Kerr·Published

Vercel managed rules says: Vercel says its AI Bots Managed Ruleset is inactive by default. The dashboard calls this Allow. Matching traffic reaches your app without being checked by that ruleset. The docs do not list all six bot names in its categories.

Open Firewall > Rules in your project and inspect Bot Management. AI Crawlers and AI Assistants have separate actions. Also review custom rules, because Vercel runs those before the managed rules.

Rules for each bot

GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot

Vercel managed rules says: AI Bots Managed Ruleset starts at Allow and is inactive. The docs do not state a separate named-bot default. Configure AI Crawlers or AI Assistants in Firewall > Rules > Bot Management. Review custom rules too.

What we measured

On the 3 sites read on 12 November 2026, one site's file had no group for any of the six bots, the other two used the * group and the home page was allowed for all six bots.

Two of the three sites may be Vercel's own products, so read the rows before generalising.

Only the sites read on the recorded dates. This is not a host-wide policy.

Source: our own research read, 12 November 2026.

What the starting setting covers

Vercel managed rules says: Vercel describes AI Crawlers as bots that collect content for training. AI Assistants retrieve pages for user actions or search queries. The guide points to a separate catalog for membership. It does not identify ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended or CCBot by name. Keep that limit in mind when reading the Allow setting. A category description alone does not tell you the full list of requests it covers.

Bot Protection is another managed ruleset on the same screen. Its inactive label is Off. When enabled in challenge mode, it serves a JavaScript check to traffic that appears unlikely to be a browser. Check both controls if your AI category is allowed but a command-line request still receives a challenge.

Change the two AI categories

Vercel managed rules says: From Firewall, open Rules and select the Configure control in Bot Management. Set an action for each AI category. Log records the traffic without blocking it. Deny blocks traffic identified in that category. Review Changes shows the pending edits. Select Publish to apply them to the production deployment. Until that step, compare your public result with the published configuration.

A custom bypass rule can target a user agent blocked by a managed ruleset. When a custom blocking rule is responsible, a bypass must run ahead of that blocking rule. Review rule order along with the action itself. Vercel lists project custom rules first and managed rules second, so changing only a category can leave an earlier denial in place.

Check the deployed address

Vercel GPTBot guide says: Replace example.com with the public domain attached to this project. Read its robots.txt and request one public page using the browser header and all six names below. These requests show how the server responds to those strings from your machine. They do not verify that the sender is a real crawler. Use your Firewall traffic view to investigate a refusal.

For GPTBot, Vercel gives a separate recipe that uses User Agent contains GPTBot in a custom rule. If that rule exists, review its Deny, Challenge or Rate Limit action as well as the category setting. Compare the page response before and after your published change, keeping the requested URL and request header the same.

curl -sS -L https://example.com/robots.txt
curl -sS -L -D - -o /dev/null -A 'Mozilla/5.0' https://example.com/
curl -sS -L -D - -o /dev/null -A 'GPTBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'ClaudeBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'PerplexityBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'Google-Extended' https://example.com/
curl -sS -L -D - -o /dev/null -A 'Applebot-Extended' https://example.com/
curl -sS -L -D - -o /dev/null -A 'CCBot' https://example.com/

More help for this host

Common questions

Why does Vercel show both Off and Allow?

The saved docs use Off for inactive Bot Protection and Allow for the inactive AI Bots Managed Ruleset. These are two controls. Check which ruleset you are reading in Bot Management.

Can I watch AI traffic before blocking it?

Yes. Choose Log for the AI category you want to inspect. Vercel says this records the traffic without blocking it. Publish the change and review your project traffic.

Who can apply a Vercel firewall change?

The ruleset guide lists Project administrator, Team member or Security access for configuring, saving and applying rules. It lists separate viewing roles. Check your role if the settings are visible but cannot be saved.

Where is the bot-specific GPTBot fix?

The guide linked below walks through the GPTBot user-agent rule, the three available actions and how to compare it with the broader AI category. Use it when your request is caught by an explicit GPTBot rule.

Check your site

Start with the free AI crawler checker to read the rules for your address. Then download Check Engine for an audit on your computer.

Sources