Page keeps failing with a 5xx server error
What it is
The server answered the page with a 5xx status, such as 500 (Internal Server Error), 502 (Bad Gateway) or 504 (Gateway Timeout), on every attempt. The page may well exist, but the server, or a service in front of it, failed to produce it: a crashing template, a database or plugin error, or a host that is overloaded.
Why it matters
Google slows its crawling of a site that answers 5xx, keeps already indexed URLs for a while, and drops them if the errors persist. Visitors see an error page instead of your content. AI crawlers that fetch the page get the same error, so it is likely to be missing from answers that depend on a fresh fetch. Errors that appear only under load tend to hit crawlers first, because they request many pages in a short time.
How Asky checks it
Asky requests the page and follows redirects; the body of a 5xx response is not read. After a 5xx, Asky tries again once a second later, then on later crawls at least an hour apart, and after five failed attempts in a row marks the page unreachable and reports it here with the last status. A successful fetch in between resets the count. A 503 is treated as throttling instead: it is retried longer and then counts toward the AskyBot blocked issue. Until then, the findings from the page’s last successful fetch are kept. A request with no reply at all is reported as no response.
Reported as an error with high severity.
How to fix it
- Open the URL in a browser and with an HTTP status checker a few times. Note the status and whether it fails every time or only now and then.
- Look up the failing requests in your server, hosting or CDN error logs: the log line usually names the template, plugin or upstream service that broke.
- In WordPress, a recently updated plugin or theme is a common cause; switch it off on a staging copy to confirm. On a hosted platform such as Webflow, check the platform’s status page and send support the URL.
- If the page fails only under load, add caching or server capacity so it keeps answering 200 when many pages are requested at once.
- If the page should no longer exist, make it answer 404 or 410 instead of an error, and take it out of your sitemap and links.
Example
https://example.com/products/compare fails because a plugin update broke its template:
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/products/compare
# 500 before the fix, 200 after