Can AI crawlers read your site?
Type your address. The checker reads your robots.txt the way the crawlers do and shows, for each of the 24 crawlers below, whether it may fetch your page. The ones that decide whether ChatGPT, Claude, Perplexity, Google's AI Overviews and Copilot can find and quote you come first, and the ones that only collect training data come second.
Free, no sign-up required. Nothing to install, and no account to make.
- Type your website's address, or any page on it.
- Press Check my site.
- Read which crawlers may fetch it, the ones that let assistants quote you at the top.
You will see:
- Each of the 24 crawlers, allowed or blocked, with the ones that let an assistant quote you first
- The robots.txt line behind each answer
- A link to share the result, and a badge for your site that links to it
Your rules, read once. Check Engine checks robots.txt rules for 22 AI crawlers across your site. A separate check requests your homepage as 9 AI crawlers plus Googlebot to find server refusals the file cannot show, among its 99 checks.
That was one file, read for one page. Check Engine runs 99 checks across your whole site, ranked by what to fix first, each with where it is and the change to make. The free download shows your score and the top problem in full.
$59once
For Macs with Apple silicon or Intel, macOS 12 or later, and 64-bit Windows
30-day refund, no conditions.
Intel Mac? Get the Intel version
Crawlers that decide whether AI can cite you
Block one of these and that assistant cannot read your pages when it answers a question, so it cannot quote or link you. The exception is a fetch a user starts: some operators say those may not follow robots.txt, and each row says which. For most businesses these should be open.
| Crawler | Who | What a block costs you | Your site |
|---|---|---|---|
Googlebot | Google Search, including AI Overviews and AI Mode, which follow your Googlebot rules rather than any AI-specific token. | ||
Bingbot | Microsoft | Bing search. Microsoft's Copilot assistant crawls as Bingbot, so this is the rule it reads. | |
Google-CloudVertexBot | Blocks Vertex AI grounding on your content for Google Cloud customers who ask it to index your site. | ||
ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks about it. OpenAI says that because a user starts these requests, robots.txt rules may not apply, so a block here may not stop ChatGPT reading your page on a user's behalf. | |
OAI-SearchBot | OpenAI | Blocks your pages from ChatGPT Search results entirely. | |
Claude-SearchBot | Anthropic | Blocks your pages from Claude's search results. | |
Claude-User | Anthropic | Stops Claude fetching your page on a user's behalf. | |
Perplexity-User | Perplexity | Fetches a page when a Perplexity user asks about it. Perplexity says that because a user requested the fetch, it generally ignores robots.txt rules, so a block here does not reliably stop it. | |
PerplexityBot | Perplexity | Removes you from Perplexity's index and citations. | |
Amazonbot | Amazon | Blocks Alexa and Amazon's answer surfaces. | |
Meta-ExternalFetcher | Meta | Fetches a page when a Meta AI user asks for it. Meta says that because the user requested the fetch, it may bypass robots.txt, so a block here may not stop it. | |
Meta-WebIndexer | Meta | Indexes pages to improve Meta AI's search results. Meta says allowing it helps Meta AI cite and link to your content in its responses. | |
MistralAI-Index | Mistral | Removes you from Le Chat's search index and citations. | |
Shap-User | Parallel Web Systems | Fetches a page on a user's behalf. Parallel says it is meant to make those requests visible rather than to be managed by robots.txt, so a block here may not stop it. | |
ShapBot | Parallel Web Systems | Removes you from Parallel's search index. Parallel says restricting it may reduce your visibility through its web APIs and search features, and Anthropic lists Parallel as a web search provider for Claude. |
Crawlers that collect training data
These gather text to train future models. Blocking them is a reasonable choice for many publishers, and it does not stop an assistant from finding and citing you through the crawlers above.
| Crawler | Who | What a block costs you | Your site |
|---|---|---|---|
Google-Extended | Blocks Gemini model training and grounding in Gemini Apps and Vertex AI. Does NOT affect AI Overviews, AI Mode or classic Google Search. Those follow your Googlebot rules. | ||
GoogleOther | Blocks Google's non-Search crawls. Does not affect Search. | ||
GPTBot | OpenAI | Blocks OpenAI from training on your content. Does not affect whether ChatGPT can cite you live. | |
ClaudeBot | Anthropic | Blocks Anthropic from training on your content. | |
Applebot-Extended | Apple | Blocks Apple Intelligence training. Applebot (search) is separate. | |
Bytespider | ByteDance | Blocks ByteDance/TikTok AI crawling. | |
CCBot | Common Crawl | Removes you from Common Crawl, an input to many open models. | |
meta-externalagent | Meta | Blocks Meta AI training and grounding. | |
MistralAI-Training | Mistral | Blocks Mistral from training on your content. |
What this checker cannot see
It reads one file, and robots.txt is a request, not a lock. Four things it cannot tell you:
- Whether your server or CDN lets the crawler in. A firewall, a bot-protection setting or a CDN rule can refuse a crawler that robots.txt welcomes. In Check Engine's August 2026 survey of the Tranco top 10,000, 643 hosts (6.43%) refused a self-identifying bot outright before any robots rule applied.
- Tags on the page itself. A
noindexornosnippetin the page or its headers limits what a search engine may show, and robots.txt says nothing about it. - Pages that need JavaScript to show their text. Most AI crawlers do not run it, so a page can be open to them and still look empty.
- What a crawler actually does. The file asks; well-behaved crawlers obey. It is not enforcement.
It checks the path you type, or your homepage if you type none. Rules for other sections of your site can differ.
How it reads the file
The rules follow RFC 9309, the robots.txt standard, and match the parser inside Check Engine, so the
two give the same answer for the same file. A crawler obeys every group that names it; only if none
does, it obeys the * groups. The longest matching rule wins, and Allow
wins a tie. If your server answers robots.txt with a server error, crawlers stay out entirely; if
it answers "not found", they may crawl everything.
The checker fetches only your robots.txt, names itself in the request, keeps a copy for a few minutes so repeated checks do not hit your server, and stores nothing about you or your site.
Sharing a result
After a check, the page gives you a link to it, such as
/tools/ai-crawler-checker/?d=yourbusiness.com. Opening the link runs the check again, so the
person you send it to sees your robots.txt as it is when they look, not a saved copy. The link carries only
the address, and nothing is kept on our side to make it work.
It also gives you a small badge to paste on your own site. The badge says your AI crawler access was checked with Check Engine and links to the live result. It does not say what the check found, so it cannot go out of date.
Checking your rules and homepage access
Check Engine checks robots.txt rules for 22 AI crawlers across your site. A separate check requests your homepage as 9 AI crawlers plus Googlebot from your computer and reports server refusals the file cannot show. It reads tags across the pages it crawls and flags text that only appears after JavaScript runs. It then ranks what to fix first, next to five optional outside checks and the rest of the audit.
$59once
For Macs with Apple silicon or Intel, macOS 12 or later, and 64-bit Windows
Priced in US dollars. Checkout shows your currency and any local tax.
Launch price: $29.50 once, through 30 November. From 1 December: $59. Buy applies the code FOUNDINGDOCKET for you; if the checkout shows $59, enter it under ‘Have a discount code?’.
Screaming Frog is $279 a year. Ahrefs plans with Site Audit run $129–$449 a month. Check Engine is $59 once. Screaming Frog’s price read 4 November 2026, Ahrefs’ 9 November 2026.
30 days, no conditions, no questions asked. Refund policy
Related
- Which AI crawlers to block and which to keep
- When your CDN blocks AI crawlers robots.txt allows
- Does noindex stop AI crawlers?
- The Check Engine Index: who blocks AI search crawlers
Common questions
Is ChatGPT allowed to crawl my website?
Check OAI-SearchBot and ChatGPT-User in the table above. OAI-SearchBot builds the index ChatGPT Search cites from, and ChatGPT-User fetches a page when someone asks about it. GPTBot is OpenAI's training crawler; blocking it does not remove you from ChatGPT's answers.
Does blocking Google-Extended keep me out of AI Overviews?
No. Google-Extended governs Gemini training and grounding in Gemini Apps and Vertex AI. AI Overviews and AI Mode follow your Googlebot rules.
My robots.txt allows every AI crawler. Am I visible?
Not necessarily. A CDN, firewall or bot-protection setting can refuse a crawler that robots.txt allows, and this checker cannot see that. Check Engine checks robots.txt rules for 22 AI crawlers. A separate check requests your homepage as 9 AI crawlers plus Googlebot from your computer. A firewall can treat the real, verified crawler differently, so your server's own logs settle whether it was allowed through.