Skip to Content
AskyBot

AskyBot

AskyBot is the web crawler operated by Asky. It visits websites that are monitored on the Asky platform to analyze their content, technical health, and visibility in AI search engines. It only crawls websites whose owners or their agencies use Asky.

How AskyBot identifies itself

Every request from AskyBot carries this user agent string:

Mozilla/5.0 (compatible; AskyBot/1.0; +https://askylabs.com)

How AskyBot behaves

  • It honors robots.txt. Rules for the user agent AskyBot (or a wildcard) are respected automatically.
  • It crawls at a low, adaptive rate per website and backs off when a server responds slowly, returns errors, or asks for a retry delay.
  • It uses conditional requests (ETag and Last-Modified) wherever possible, so repeat visits to unchanged pages transfer almost no data.
  • It crawls from cloud infrastructure without a fixed IP range, so it should be identified by its user agent rather than by IP address.

Allowing AskyBot

Bot protection products sometimes challenge AskyBot with a captcha or block it with a 403 response. When that happens, the affected pages cannot be analyzed and the website owner loses monitoring coverage. Asky flags this in the app as an AskyBot is blocked issue under Technical opportunities.

To allow AskyBot, add an allow rule for the user agent AskyBot in your hosting or firewall settings:

SiteGround

SiteGround Security challenges unknown bots with a captcha page. To allow AskyBot:

  1. Open Site Tools for your website
  2. Go to Security, then Blocked Traffic
  3. Under allowed user agents, add AskyBot
  4. If you use the SG Security WordPress plugin, also check its bot protection settings and allow AskyBot there

Cloudflare

Create a WAF rule that skips bot protection for AskyBot:

  1. Open your Cloudflare dashboard and select the site
  2. Go to Security, then WAF, then Custom rules
  3. Create a rule matching: User Agent contains AskyBot
  4. Set the action to Skip, covering Bot Fight Mode and managed challenges

Other hosting or firewalls

Look for bot protection, WAF, or firewall settings in your hosting panel or CDN and add an allow (whitelist) rule for the user agent AskyBot. AskyBot crawls from cloud infrastructure without a fixed IP range, so allow it by user agent rather than by IP.

Also confirm your robots.txt does not disallow AskyBot. It obeys robots.txt rules automatically.

Once AskyBot is allowed, blocked pages are re-checked automatically within about a day and the issue in Asky clears itself. No re-scan is needed.

Blocking AskyBot

If you do not want AskyBot to visit your website, add this to your robots.txt. AskyBot checks robots.txt before crawling and will respect it:

User-agent: AskyBot
Disallow: /

Questions

If you have questions about AskyBot or believe it is not behaving as described here, contact us at support@askylabs.com.

Last updated on
Reddit