Skip to Content
OpportunitiesTechnicalIssue ReferenceRobots.txtrobots.txt contains lines crawlers cannot read

robots.txt contains lines crawlers cannot read

What it is

One or more lines in your robots.txt are not valid robots.txt records. Crawlers that follow the standard skip the lines they cannot read, while Google reads a few common slips anyway (Disalow, User agent, a missing colon), so a mistyped rule binds some crawlers and not others. A rule written before any User-agent line applies to no crawler.

Why it matters

The damage depends on the line that is lost. A dropped Disallow leaves sections open that you meant to keep crawlers out of, and a dropped Allow can leave pages blocked that you meant to open. Lines no crawler supports, such as Noindex:, are skipped as well, but they never controlled crawling: Google  ignores them, so they are not reported. AI crawlers that honor robots.txt lose the same rule.

How Asky checks it

Asky reads your robots.txt daily and at the start of each audit, ignoring blank lines, anything after # and a byte order mark. A line is an error when a crawler may skip a rule it was meant to hold: a field one or two letters from User-agent, Allow, Disallow, Sitemap or Crawl-delay (Dissallow, User agent), one of those without its colon, or an Allow or Disallow before the first User-agent. Other fields, such as Noindex, are not errors. The details list up to 10 errors with line numbers, and the total when there are more. The other robots.txt checks also apply the slips Google reads, so a block they make is reported.

Reported as a warning with low severity.

How to fix it

  1. Open the issue’s details for the line numbers, then open https://example.com/robots.txt for your own domain.
  2. Write each line as Field: value, for example Disallow: /checkout/. A line without a colon is skipped.
  3. Correct misspelled field names such as Disalow, Useragent or User agent.
  4. Move rules that sit above the first User-agent line into the group of the crawlers they are meant for.
  5. Remove fields robots.txt does not support, such as Noindex:. To keep a page out of search results, put a noindex robots meta tag on the page and leave it crawlable, so crawlers can see the tag.
  6. Check what the corrected rules now do. A repaired Disallow: / under User-agent: * or User-agent: Googlebot blocks the whole site, which is reported as its own critical issue.

Example

In the first version, Disallow /cart/ has no colon and Disalow is misspelled, so neither rule applies:

# Before User-agent: * Disallow /cart/ Disalow: /checkout/ # After User-agent: * Disallow: /cart/ Disallow: /checkout/ Sitemap: https://example.com/sitemap.xml

← Back to Robots.txt

Last updated on