Does your server allow AI crawlers through?
It compares saved server responses with the crawler access your robots file allows.
What Check Engine measures
The connector must have run and returned data. Its ordinary request must have HTTP status from 200 up to but below HTTP status 400 before crawler refusals can be compared. Payment responses with HTTP status 402 are separated from refusals. Challenges and verifying network edges can make a result untestable. Responses with HTTP status 429 or HTTP status 503 are separated as throttling. Other refusals conflict with policy only when the saved robots decision does not block that crawler. For example, a payment response with HTTP status 402 is handled separately from a crawler access refusal.
Saved connector input includes the baseline, search-control request and crawler requests. This is a test of the requested page from the audit's network location, not a complete server log.
How to read the result
Untestable, unverifiable and payment results are notices. Throttling is medium severity. A conflicting refusal is critical when a citation crawler is affected and high otherwise. Agreement can produce a pass. The report also records crawlers not asked before the connector's budget expired.
Why it matters
A robots permission does not override a server refusal. Comparing the policy with a real response helps locate the part of the request path that needs attention, while challenges leave room for a real verified crawler to behave differently.
How to fix it
Inspect the saved response and the matching firewall or bot rule. Check server logs before relaxing protection. If the server asked for a wait, respect that wait and test again. For a verified-crawler challenge, review the network verification rule rather than assuming that changing the user-agent text proves real crawler access.
When your CDN blocks AI crawlers your robots.txt allows
What we found in field audits
Problems in AI search visibility were found on 14 of the 14 sites we audited. These are area totals. Read the field-audit method.
Method note
backend/seo_engine/checks/ai_visibility.py, edge_access, revision 0bb8e8d3.
Common questions
Can my server block an AI crawler that robots.txt allows?
Yes. A saved server refusal can conflict with the saved robots decision. The check compares policy with the connector's responses.
How does the AI server-access check handle rate limits?
Throttling covers HTTP status 503 and HTTP status 429. It gets a different explanation from an ordinary access refusal.
Are payment responses counted as AI crawler refusals?
No. Payment responses at HTTP status 402 are kept separate. They produce notices rather than an ordinary conflicting-refusal finding.