robots.txt file is empty
What it is
Your robots.txt answers successfully but holds nothing, or only blank lines and spaces. Crawlers read an empty file as permission to crawl the whole site, so nothing is blocked; the file simply says nothing. Some site builders publish a blank robots.txt when the robots.txt field in their settings is left empty.
Why it matters
The cost is the missing Sitemap line. robots.txt is the first file crawlers request, and both Google and Bing read a Sitemap line there to find your sitemap. Bing says the sitemaps it reads serve Copilot as well as its search results. Without the line, crawlers you never submitted your sitemap to must guess a common sitemap path or find new pages through links.
How Asky checks it
Asky requests /robots.txt on each of your domains, and on subdomains with a sitemap or listed pages such as a blog, daily and at the start of each audit. The issue is reported when the file answers with a success status and its body is empty or only whitespace. A file with nothing but comments is not empty: it is read, and raises the missing Sitemap directive issue instead. An empty file is reported on its own, without that issue on top. A sitemap Asky finds elsewhere is also reported as not listed in robots.txt. A 404 counts as robots.txt missing. It clears on the first run that finds content in the file.
Reported as a warning with low severity.
How to fix it
- Find where the file is written: the robots.txt field in Webflow’s SEO settings, your WordPress SEO plugin or the web root, or your front-end app on a headless site.
- Add one line with the full URL of your sitemap or sitemap index:
Sitemap: https://example.com/sitemap.xml. A file with only a Sitemap line is valid: it raises no syntax error and blocks nothing. - Add a line for each other sitemap you use, or list only the sitemap index that contains them.
- Add crawl rules only if you need them, and never
Disallow: /underUser-agent: *, which blocks the whole site. - Open the URL to confirm the new text is served. A CDN may keep serving the cached blank file for a while.
Example
https://example.com/robots.txt answered 200 with no content. One line fixes it:
Sitemap: https://example.com/sitemap.xml