AskyBot blocked by your site’s bot protection
What it is
Your website refused AskyBot, Asky’s crawler, on pages listed in your sitemap: with a 401 or 403 status, with a bot-protection challenge such as a captcha page, or with 429, 408 or 503 answers that did not let up. The message names the protection Asky detected, such as Cloudflare or SiteGround Security, and the share of pages affected.
Why it matters
Asky cannot read the blocked pages, so their technical findings, content changes and AI visibility analysis go stale until they load again. The same protection may also challenge other crawlers it does not recognize, search and AI crawlers included. Google slows its crawling after a 429 and treats a 401 or 403 page like one that does not exist, so check your rules let through the crawlers you want.
How Asky checks it
Asky counts sitemap pages its crawler tried in the last 7 days and could not load after retries: refused with 401 or 403, served a challenge page, or answered 429, 408 or 503 throughout. It is raised when any page got a challenge, when 10 or more pages were blocked, or when 3 or more were and make up 5% of pages tried. A nightly or after-crawl check resolves it below that, unless Asky read none of the site that week. A robots.txt rule keeping AskyBot from your homepage or a language home page is reported apart, naming the rule. Details list up to five blocked URLs.
Reported as a critical issue with high severity when 25% or more of the pages are blocked, and medium below that.
How to fix it
- Open the issue for the protection it names and the sample URLs, and load one in a browser to confirm it works for visitors.
- Allow the user agent
AskyBotin your firewall or bot protection. The AskyBot page has the steps for SiteGround, Cloudflare and other hosts, and a fixed address for firewalls that can only allow by IP. - On Cloudflare, a custom rule with the Skip action lets AskyBot past Super Bot Fight Mode, managed rules and rate limiting rules. The free Bot Fight Mode cannot be skipped this way, so use the IP Access rule on the AskyBot page instead.
- If the message says rate-limited, raise the limit for AskyBot or exempt it from the limit.
- If the blocked pages are a member area or paywall that should stay closed, take them out of your sitemap. Asky only counts pages listed there.
- No new scan is needed: while the issue is open, Asky tries a few of the blocked pages every night. Once they load, the issue clears as the week of blocked attempts falls out of the count, within about a week.
Example
A Cloudflare custom rule for https://example.com that skips bot protection for AskyBot, with the action set to Skip:
Field: User Agent
Operator: contains
Value: AskyBot
Action: Skip (Super Bot Fight Mode, managed rules, rate limiting rules)