Sitemap returns a server error
What it is
A sitemap Asky tried to read answered with a server error, an HTTP status from 500 to 599, instead of the file. Sitemaps that a CMS or plugin builds on every request fail this way when the query behind them breaks or runs out of time on a large site.
Why it matters
Search engines cannot read the pages it lists, so new and updated URLs in it lose that route to discovery until it loads again. Google reads 5xx answers as a sign the server is struggling and slows its crawling. When the failing file is a sitemap index or your only sitemap, every page it covers loses that route to discovery, for search engines and for AI crawlers that read sitemaps.
How Asky checks it
During its daily site discovery and at the start of each technical audit, Asky fetches every sitemap it uses: those in robots.txt, those added in Asky, one found at a standard path, and the child sitemaps of an index. After a 429 or 5xx it waits and retries once with a browser user agent. The issue is reported when the final status is 500 or higher, with the status in the details. The last good copy is kept, no other sitemap check runs on it, and its pages are not treated as removed. A timeout, 401, 403 or 429 skips the check. It clears on the first run that loads the file.
Reported as a critical issue with high severity.
How to fix it
- Open the sitemap URL from the issue’s details and check the status it answers with.
- If it fails for everyone, check the server or CDN error logs for that request. For a generated sitemap, check the plugin or route that builds it.
- On a large site, split the sitemap into smaller files under a sitemap index, or have it generated ahead of time as a static file, so no single request has to build the whole list.
- If it works in a browser but fails for crawlers, a firewall or bot protection is answering them with an error. Exempt your sitemap URLs from those rules. The AskyBot page shows how to allow Asky’s crawler.
- Confirm the URL answers 200 with the XML sitemap.
Example
The status check for https://example.com/sitemap.xml before and after the fix:
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/sitemap.xml
# 503 before the fix, 200 after