Skip to Content
OpportunitiesTechnicalIssue ReferenceRobots.txtrobots.txt blocks every crawler from the whole site

robots.txt blocks every crawler from the whole site

What it is

The User-agent: * group in your robots.txt disallows the entire site, and neither Googlebot nor Bingbot has a group of its own that lets it back in. Every crawler without its own rules is told to stay out. This is usually a pre-launch or staging setting that went live with the site, or a file copied from a staging server.

Why it matters

Search engines stop fetching your pages. Google  can still list a blocked URL it finds through links, but without a description, and it only links to pages indexed and eligible for a snippet  in AI Overviews and AI Mode. AI search crawlers without a group of their own fall under the same * rule, so assistants whose crawlers honor robots.txt lose access as well.

How Asky checks it

Asky reads your robots.txt during its daily site discovery and at the start of each technical audit. It is reported when a User-agent: * group exists, and the rules for *, for Googlebot and for Bingbot all disallow two sample paths inside the site, /a and /z9/page. Each crawler obeys its own groups, otherwise *. Disallow: / and Disallow: /* block; Disallow: /$ and path blocks do not. Allow rules that reopen only some sections, even the homepage, do not change the result. The longest matching rule wins, Allow on a tie. The same file also raises Googlebot blocked, Bingbot blocked and usually AI crawlers blocked.

Reported as a critical issue with high severity.

How to fix it

  1. Open https://example.com/robots.txt for your own domain and find the User-agent: * group with Disallow: / or Disallow: /*.
  2. On a live site, delete that line, or replace it with Disallow rules for only the paths crawlers should skip, such as /cart/.
  3. Turn off the setting that wrote it: a search engine visibility or staging option in your CMS, a hosting panel toggle or an SEO plugin. Otherwise the block can return on the next publish.
  4. Giving only Googlebot its own Allow: / group clears this issue but leaves Bingbot and every unnamed crawler blocked, which is reported separately.
  5. After the fix, use the robots.txt report  in Search Console to ask Google to fetch the new file.
  6. To leave it as it is: if this is a staging or private site that should not be crawled, mark the issue resolved.

Example

A pre-launch file left on https://example.com, and the live version that replaces it:

# Before: nothing may be crawled
User-agent: *
Disallow: /

# After: only the checkout flow is skipped
User-agent: *
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

← Back to Robots.txt

Last updated on
robots.txt blocks Bingbot from the whole siterobots.txt blocks Googlebot from the whole site