Skip to content
Scority
Start free audit

Free · no signup · per-bot verdicts

AI Crawler Access Checker

A site can be technically perfect and still invisible in AI answers: if robots.txt blocks the crawlers that feed ChatGPT, Claude, Perplexity and Google AI Overviews, the content can't be cited. Enter your site — we read robots.txt once and show a per-bot allow/block verdict.

We read robots.txt for the site's origin once (plus a llms.txtpresence check). The URL isn't sent to marketing analytics.

What it checks

We fetch your site's robots.txt (one request, from our servers) and evaluate it with the standard robots-exclusion parser — the same user-agent group precedence real crawlers apply:

  • Retrieval (answer-engine) crawlers— OAI-SearchBot and ChatGPT-User (ChatGPT Search & browsing), ClaudeBot (Claude), PerplexityBot (Perplexity), Google-Extended (Gemini / AI Overviews). These fetch pages to answer live queries: blocking them removes the site from those AI answers.
  • Training / corpus crawlers — GPTBot, anthropic-ai, CCBot (Common Crawl), Applebot-Extended. Blocking them is a legitimate training opt-out and does not reduce live AI-answer visibility.
  • Explicit vs wildcard — whether each bot is named directly in robots.txt or only covered by the * group.
  • llms.txt — whether the site publishes the emerging AI-curation file (informational only).

How to read the verdict

A blocked retrieval crawler is the costly case: those bots fetch pages at question-time, so a block means your site cannot be cited on that surface no matter how good the content is. A blocked training bot is shown as an opt-out, not a problem — many sites block training deliberately. No robots.txt at all means everything is allowed by convention.

Methodology

Verdicts are deterministic: the robots.txt body is parsed with the standard robots-exclusion protocol parser and each bot's token is evaluated against your site root — the same input always produces the same result, and no AI model decides anything. The check never fetches your pages themselves, only robots.txt and llms.txt.

Limitations

  • One snapshot of robots.txt — a CDN or geo-specific config may serve crawlers a different file.
  • robots.txt is voluntary: allowing a crawler doesn't force it to visit, and a few crawlers ignore robots entirely.
  • Firewalls, bot-management layers and X-Robots-Tag can still block AI crawlers even when robots.txt allows them — this tool checks robots.txt only.
  • llms.txt is an emerging convention; its absence is normal today.

Official sources

Bot documentation: OpenAI crawlers, Anthropic ClaudeBot, PerplexityBot, Google crawlers (incl. Google-Extended), RFC 9309 (Robots Exclusion Protocol).

Want the full picture — technical health plus AI-search readiness? Run the Page SEO Checker or start a free site audit.