Skip to Content
OpportunitiesTechnicalIssue ReferenceRobots.txtAI crawlers blocked in robots.txt

AI crawlers blocked in robots.txt

What it is

Your robots.txt file tells one or more AI crawlers not to crawl any part of the site. AI companies use three kinds of crawler, each with its own name in robots.txt: search crawlers that index pages so an assistant can cite them, user-request fetchers that open a page when someone asks about it, and training crawlers that collect content for future models. You can allow one kind and block another.

Why it matters

Blocking an AI search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) keeps your pages out of that assistant’s answers and citations. Blocking a training crawler keeps your content out of future models: it can be deliberate, though models that never read the site may know less about your brand. Google-Extended also controls whether Gemini uses your pages to ground its answers . Blocks are often accidental, from a security plugin, CDN or copied file.

How Asky checks it

Asky reads the robots.txt of each host serving your pages and applies each crawler’s rules: the groups naming it, or the * group. A crawler is reported when they block the whole site, as Disallow: / does, even if Allow reopens a few paths. Path blocks are not. When every crawler is blocked, only that is reported.

  • Search and user-request: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User
  • Training: GPTBot, ClaudeBot, Cohere-AI, CCBot
  • Gemini training and grounding: Google-Extended

Retired names (Claude-Web, Anthropic-AI) block nothing and are not reported. Reported as critical with high severity, naming each crawler and its purpose.

How to fix it

  1. Open https://example.com/robots.txt for your own domain (each subdomain has its own file) and find the groups that name the crawlers in the issue, or a User-agent: * group with Disallow: /.
  2. Decide per crawler. To appear in AI answers, allow at least the search crawlers. Blocking training crawlers is your choice. OpenAI and Perplexity say their user-request fetchers may not follow robots.txt, so blocking those is not a reliable opt-out.
  3. Remove the Disallow: / rule for the crawlers you want to allow, or give them their own group with Allow: /. A group that names a crawler replaces the * group for it, so repeat any * rules you still want.
  4. If a plugin, a CDN (such as Cloudflare’s managed robots.txt) or a hosting panel writes the file, change it there. Bot blocking in a firewall or CDN never shows in robots.txt, so check those settings too.
  5. To keep a deliberate block, such as training crawlers you chose to exclude, choose Mark as Resolved. The issue stays resolved through later audits and reopens only when your robots.txt changes and blocks an AI crawler it did not block before.

Example

This robots.txt on https://example.com blocks every AI crawler it names, including the ones that decide whether the site appears in AI answers:

# Before: search and training crawlers all blocked User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Claude-SearchBot Disallow: / # After: training crawlers blocked, AI search allowed User-agent: GPTBot User-agent: ClaudeBot Disallow: / User-agent: OAI-SearchBot User-agent: Claude-SearchBot Allow: / Sitemap: https://example.com/sitemap.xml

← Back to Robots.txt

Last updated on