AI crawler access

mediumseo-ai-crawler-access

Named AI crawlers are often blocked by accident in robots.txt. Whether that's a problem depends on whether it was a deliberate choice.

What it is

This reads robots.txt and checks whether it blocks any of the named AI crawlers — OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot and Claude-SearchBot, PerplexityBot, Google-Extended, Common Crawl's CCBot and others — distinct from the general-purpose search crawlers (Googlebot, Applebot) that index for classic search.

It reports whether these crawlers can reach the site at all; it doesn't evaluate whether blocking a specific one is the right call for this site — a robots.txt disallow for an AI crawler is treated as a fact to surface, not a mistake to fix.

Why it matters

Blocking these agents is a legitimate, deliberate choice — it keeps content out of model training data and out of answer engines' output, and plenty of sites intend exactly that. The finding exists because it's also easy to block them by accident: a robots.txt template copied from another project, or a blanket 'block all bots' rule that predates ChatGPT existing, ends up excluding AI crawlers as a side effect nobody chose.

If the site should be readable, cited and answered from by ChatGPT, Claude, Perplexity or Gemini's grounding, the fix is a specific Allow rule for the agents that matter, not a wholesale rewrite of robots.txt.

How to fix it

with a reference doc

robots.txt

# Answer-engine fetchers: retrieve a specific page a person linked
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Bulk training crawlers: allow or keep blocking independently of the above
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

Reference documentation

Related checks

mediumStructured data (JSON-LD)Structured data is a <script type="application/ld+json"> block declaring what a page is, using schema.org vocabulary — Article, Product, Organization, FAQPage and similar types. This check looks for at least one such block that parses as valid JSON and declares a recognizable @type; it isn't a full schema.org validator checking that a given type's required fields are present.highMeta robots noindexThe meta robots tag — <meta name="robots" content="..."> — gives per-page instructions to crawlers, most commonly noindex (don't show this page in search results) and nofollow (don't follow links from this page). This check fails when either is set.highContent that requires JavaScriptThis compares the DOM node count and text length in the HTML a server actually sent against the DOM Google's Lighthouse run produces after rendering the page. When the rendered page has several times more nodes than the raw HTML did, and the raw HTML had very little text to begin with, that's the signature of a client-rendered shell rather than a page with real content.

This is one of the 75 checks the scanner runs. See what we check for the full list, every severity weight, and how the score is computed from them.

What it is

Why it matters

How to fix it

Related checks

See whether your own site passes ai crawler access.