Can AI search crawlers read your site?
It separates crawler restrictions you chose from restrictions inherited from a general robots rule.
What Check Engine measures
The check reads the saved AI crawler access decisions. An empty access table produces no finding. A robots response with HTTP status 500 or above, or a failed response with a requested robots address, prevents a confident access claim. It separates citation crawlers from training crawlers and distinguishes a named block, an allowlist omission and a wildcard block. The citation group includes OAI-SearchBot, PerplexityBot, Claude-SearchBot and Meta-WebIndexer. For example, a robots response with HTTP status 500 leaves access uncertain even if an earlier copy allowed the crawler.
This is a site policy check, based on the saved robots file and its access table. It does not need a set of crawled article pages.
How to read the result
Accidental citation blocks can be critical. Named choices, allowlist omissions and training restrictions are notices. The report explains which rule applies. Review it against the access you intended. A robots decision alone cannot prove that a server serves the request.
Why it matters
Search access and training permission are separate choices in this code. Keeping them distinct lets you review the rule that caused a finding without being told to remove a restriction you meant to keep.
How to fix it
Decide which crawlers should have access, then inspect the named group and the wildcard group that apply to each. Change only the rules that conflict with your policy. Use the linked guide to check the crawler names and retest the file after saving it. Review server access separately when robots allows a request that still fails.
What we found in field audits
Problems in AI search visibility were found on 14 of the 14 sites we audited. These are area totals. Read the field-audit method.
Method note
backend/seo_engine/checks/ai_visibility.py, crawler_access, revision 0bb8e8d3.
Common questions
Which AI search crawlers does this robots check name?
Among the citation crawlers are OAI-SearchBot, PerplexityBot, Meta-WebIndexer and Claude-SearchBot. Training access is considered separately.
Can a failed robots request prove that AI crawlers are allowed?
No. HTTP status 500 or higher leaves the saved policy uncertain. A failed fetch with a requested robots address also prevents a confident result.
Is an intentional AI training block treated like an accidental search block?
No. An accidental citation restriction can be critical. Training restrictions are notices, preserving the distinction between search access and training choices.