Skip to Content
OpportunitiesTechnicalIssue ReferenceCrawlabilityAskyBot blocked by your site's bot protection

AskyBot blocked by your site’s bot protection

What it is

Your website refused AskyBot, Asky’s crawler, on pages listed in your sitemap: with a 401 or 403 status, with a bot-protection challenge such as a captcha page, or with 429, 408 or 503 answers that did not let up. The message names the protection Asky detected, such as Cloudflare or SiteGround Security, and the share of pages affected.

Why it matters

Asky cannot read the blocked pages, so their technical findings, content changes and AI visibility analysis go stale until they load again. The same protection may also challenge other crawlers it does not recognize, search and AI crawlers included. Google  slows its crawling after a 429 and treats a 401 or 403 page like one that does not exist, so check your rules let through the crawlers you want.

How Asky checks it

Asky counts the pages in your sitemap its crawler could not load after its retries: refused with 401 or 403, served a challenge page, or answered 429, 408 or 503 across its whole retry budget. It checks after every crawl: onboarding, the daily crawl and each technical audit. It is raised when any page got a challenge, when 10 or more pages were blocked, or when 3 or more were and they make up at least 5% of pages. It clears itself when a later crawl falls below that. robots.txt rules are not counted here. The details list up to five blocked URLs.

Reported as a critical issue with high severity when 25% or more of the pages are blocked, and medium below that.

How to fix it

  1. Open the issue for the protection it names and the sample URLs, and load one in a browser to confirm it works for visitors.
  2. Allow the user agent AskyBot in your firewall or bot protection. The AskyBot page has the steps for SiteGround, Cloudflare and other hosts, and a fixed address for firewalls that can only allow by IP.
  3. On Cloudflare, a custom rule with the Skip action  lets AskyBot past Super Bot Fight Mode, managed rules and rate limiting rules. The free Bot Fight Mode cannot be skipped  this way, so use the IP Access rule on the AskyBot page instead.
  4. If the message says rate-limited, raise the limit for AskyBot or exempt it from the limit.
  5. If the blocked pages are a member area or paywall that should stay closed, take them out of your sitemap. Asky only counts pages listed there.
  6. No new scan is needed: blocked pages are tried again within about a day, and the issue clears once the share falls below the thresholds.

Example

A Cloudflare custom rule for https://example.com that skips bot protection for AskyBot, with the action set to Skip:

Field: User Agent
Operator: contains
Value: AskyBot
Action: Skip (Super Bot Fight Mode, managed rules, rate limiting rules)

← Back to Crawlability

Last updated on
OverviewOverview