robots.txt contains lines crawlers cannot read
What it is
One or more lines in your robots.txt are not valid robots.txt records. Crawlers that follow the standard skip the lines they cannot read, while Google reads a few common slips anyway (Disalow, User agent, a missing colon), so a mistyped rule binds some crawlers and not others. A rule written before any User-agent line applies to no crawler.
Why it matters
The damage depends on the line that is lost. A dropped Disallow leaves sections open that you meant to keep crawlers out of, and a dropped Allow can leave pages blocked that you meant to open. Lines no crawler supports, such as Noindex:, are skipped as well, but they never controlled crawling: Google ignores them, so they are not reported. AI crawlers that honor robots.txt lose the same rule.
How Asky checks it
Asky reads your robots.txt daily and at the start of each audit, ignoring blank lines, anything after # and a byte order mark. A line is an error when a crawler may skip a rule it was meant to hold: a field one or two letters from User-agent, Allow, Disallow, Sitemap or Crawl-delay (Dissallow, User agent), one of those without its colon, or an Allow or Disallow before the first User-agent. Other fields, such as Noindex, are not errors. The details list up to 10 errors with line numbers, and the total when there are more. The other robots.txt checks also apply the slips Google reads, so a block they make is reported.
Reported as a warning with low severity.
How to fix it
- Open the issue’s details for the line numbers, then open
https://example.com/robots.txtfor your own domain. - Write each line as
Field: value, for exampleDisallow: /checkout/. A line without a colon is skipped. - Correct misspelled field names such as
Disalow,UseragentorUser agent. - Move rules that sit above the first
User-agentline into the group of the crawlers they are meant for. - Remove fields robots.txt does not support, such as
Noindex:. To keep a page out of search results, put anoindexrobots meta tag on the page and leave it crawlable, so crawlers can see the tag. - Check what the corrected rules now do. A repaired
Disallow: /underUser-agent: *orUser-agent: Googlebotblocks the whole site, which is reported as its own critical issue.
Example
In the first version, Disallow /cart/ has no colon and Disalow is misspelled, so neither rule applies:
# Before
User-agent: *
Disallow /cart/
Disalow: /checkout/
# After
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://example.com/sitemap.xml