Does your host block AI crawlers?
Your host may offer a crawler switch, a robots.txt editor or a firewall rule. The guides below show what each host says about six bot names and where its docs leave the answer open.
Cloudflare access policies and Cloudflare traffic announcement say: Start with your host’s guide, then read your own public file and compare page responses. Cloudflare’s published accounts disagree on new-domain settings, so its guide shows both and tells you where to inspect your domain.
Find your host
Cloudflare access policies, Vercel managed rules, Netlify blocking guide and Netlify request categories say: Cloudflare describes actions for Search, Agent and Training. Vercel has separate AI Crawlers and AI Assistants controls. Netlify documents a project extension and request categories. Those controls can refuse requests or decide how a function handles them. Inspect the saved action as well as any file asking crawlers to stay away.
Wix crawler guide, Webflow bot controls, Squarespace crawler settings, WordPress.com sharing settings, Shopify Help Center and GitHub Pages setup guide say: Wix and Webflow show custom robots.txt examples. Squarespace offers one checkbox with a published bot list. WordPress.com combines a partner-sharing choice with robots.txt requests. Shopify separates its file instructions from network bot management and Catalog sharing. GitHub’s setup guide describes static publishing without naming an AI bot access policy. Choose the host link that matches where your public content is served.
WordPress.com crawler settings
Choose the bot before changing access
The six names covered here are GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and CCBot. Each host table links to this site’s existing bot reference. Those references retain the stored crawler descriptions and October robots.txt measurements. Use them when deciding which service you mean, then return to the host instructions to inspect the available control.
Netlify request categories and Webflow bot controls say: Some hosts give all of these names the same editing steps. Those names share a host guide. A separate guide is linked only where there is a useful bot-specific question, such as Netlify putting GPTBot in Crawler while ChatGPT-User is an AI Agent, or Webflow showing a Google-Extended folder exclusion. This keeps the instructions for one setting together.
Read the source date with the answer
Cloudflare access policies, Cloudflare traffic announcement and Cloudflare changelog say: The host source links show the exact time they were read on 12 November 2026. A starting setting can change, and two published pages can give different answers. Cloudflare’s policy page says new domains use Allow for all three behaviors, while its blog and changelog describe ad-page blocks for Training and Agent. Its guide preserves both claims with their dates.
When a host does not name a bot or state its starting access, the guide says the docs do not say. An example of how to create a block is not proof that the block already exists. A list of supported controls is not a reading of your domain. Use the dashboard and file checks to answer that next question for your own site.
Keep the file request and the HTTP result
Read robots.txt from the root of the hostname that serves your content. The repository’s robots.txt reference was read on 9 November 2026. It describes instructions for crawlers, while the host guides explain the controls available on that platform. Each guide provides curl commands you can run on your own computer with a browser header and the six named headers. Google robots.txt introduction. Read 9 November 2026
Netlify request categories and Squarespace crawler settings say: Keep the URL and request header unchanged when comparing a before and after result. Typing a bot name in a request is a synthetic check of that name. It does not prove that the real operator visited or honored a request. Netlify specifically tells owners to change User-Agent for deployed category tests, while Squarespace warns that robots.txt compliance is voluntary.
Common questions
Which guide should I use if Cloudflare sits in front of another host?
Start with the host serving your content, then inspect the Cloudflare zone as well. Cloudflare describes its own behavior actions, and Shopify’s guide specifically warns that it does not support placing a proxy in front of a Shopify store.
Why do some bots share a host guide?
When the host gives the same control and checking steps for several names, a single guide covers them together. Squarespace’s listed names share one site-wide checkbox, for example.
Can an AI block leave partner product sharing active?
Shopify says its Catalog sends data to activated agentic storefronts independently of robots.txt. WordPress.com documents a sharing checkbox with both partner-network and crawler-request effects. Read the host-specific control before changing it.
Where should I start checking my own address?
Use the free AI crawler checker below to inspect the rules for your address, then follow your host guide’s file and page-response checks. The app download is the second step for a local audit.
Check your site
Start with the free AI crawler checker to read the rules for your address. Then download Check Engine for an audit on your computer.
Sources
- Cloudflare access policies, read 12 November 2026 at 01:45 UTC
- Cloudflare traffic announcement, read 12 November 2026 at 01:45 UTC
- Vercel managed rules, read 12 November 2026 at 01:45 UTC
- Netlify blocking guide, read 12 November 2026 at 01:45 UTC
- Netlify request categories, read 12 November 2026 at 01:45 UTC
- Wix crawler guide, read 12 November 2026 at 01:45 UTC
- Webflow bot controls, read 12 November 2026 at 01:46 UTC
- Squarespace crawler settings, read 12 November 2026 at 01:45 UTC
- WordPress.com sharing settings, read 12 November 2026 at 01:45 UTC
- Shopify Help Center, read 12 November 2026 at 01:45 UTC
- GitHub Pages setup guide, read 12 November 2026 at 01:45 UTC
- Cloudflare changelog, read 12 November 2026 at 01:45 UTC