AI crawlers blocked in robots.txt
What it is
Your robots.txt file tells one or more AI crawlers not to crawl any part of the site. AI companies use three kinds of crawler, each with its own name in robots.txt: search crawlers that index pages so an assistant can cite them, user-request fetchers that open a page when someone asks about it, and training crawlers that collect content for future models. You can allow one kind and block another.
Why it matters
Blocking an AI search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) keeps your pages out of that assistant’s answers and citations. Blocking a training crawler keeps your content out of future models: it can be deliberate, though models that never read the site may know less about your brand. Google-Extended also controls whether Gemini uses your pages to ground its answers . Blocks are often accidental, from a security plugin, CDN or copied file.
How Asky checks it
Asky reads the robots.txt of each host serving your pages and applies each crawler’s rules: the groups naming it, or the * group. A crawler is reported when they block the whole site, as Disallow: / does, even if Allow reopens a few paths. Path blocks are not. When every crawler is blocked, only that is reported.
- Search and user-request: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User
- Training: GPTBot, ClaudeBot, Cohere-AI, CCBot
- Gemini training and grounding: Google-Extended
Retired names (Claude-Web, Anthropic-AI) block nothing and are not reported. Reported as critical with high severity, naming each crawler and its purpose.
How to fix it
- Open
https://example.com/robots.txtfor your own domain (each subdomain has its own file) and find the groups that name the crawlers in the issue, or aUser-agent: *group withDisallow: /. - Decide per crawler. To appear in AI answers, allow at least the search crawlers. Blocking training crawlers is your choice. OpenAI and Perplexity say their user-request fetchers may not follow robots.txt, so blocking those is not a reliable opt-out.
- Remove the
Disallow: /rule for the crawlers you want to allow, or give them their own group withAllow: /. A group that names a crawler replaces the*group for it, so repeat any*rules you still want. - If a plugin, a CDN (such as Cloudflare’s managed robots.txt) or a hosting panel writes the file, change it there. Bot blocking in a firewall or CDN never shows in robots.txt, so check those settings too.
- To keep a deliberate block, such as training crawlers you chose to exclude, choose Mark as Resolved. The issue stays resolved through later audits and reopens only when your robots.txt changes and blocks an AI crawler it did not block before.
Example
This robots.txt on https://example.com blocks every AI crawler it names, including the ones that decide whether the site appears in AI answers:
# Before: search and training crawlers all blocked
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
Disallow: /
# After: training crawlers blocked, AI search allowed
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
Allow: /
Sitemap: https://example.com/sitemap.xml