Skip to Content
AskyBot

AskyBot

AskyBot is the web crawler operated by Asky. It visits websites that are monitored on the Asky platform to analyze their content, technical health, and visibility in AI search engines. It only crawls websites whose owners or their agencies use Asky.

How AskyBot identifies itself

AskyBot carries one of these two user agent strings:

Mozilla/5.0 (compatible; AskyBot/1.0; +https://askylabs.com)
AskyBot/1.0 (+https://askylabs.com)

The first is the standard form. The second is used when a firewall refuses the standard form, which some bot protection products do to any unfamiliar crawler in that format. Both contain AskyBot, so an allow rule that matches on AskyBot covers every request.

How AskyBot behaves

  • It honors robots.txt. Rules for the user agent AskyBot (or a wildcard) are respected automatically.
  • It crawls at a low, adaptive rate per website and backs off when a server responds slowly, returns errors, or asks for a retry delay.
  • It uses conditional requests (ETag and Last-Modified) wherever possible, so repeat visits to unchanged pages transfer almost no data.
  • It is identified by its user agent. Most requests come from cloud infrastructure whose addresses change. When a site refuses those, AskyBot retries from a fixed address you can allowlist, listed under IP address below.

Allowing AskyBot

Bot protection products sometimes challenge AskyBot with a captcha or block it with a 403 response. When that happens, the affected pages cannot be analyzed and the website owner loses monitoring coverage. Asky flags this in the app as an AskyBot is blocked issue under Technical opportunities.

To allow AskyBot, add an allow rule for the user agent AskyBot in your hosting or firewall settings:

SiteGround

SiteGround’s Anti-Bot AI challenges automated traffic with a captcha page. There is no setting in Site Tools that exempts a crawler from it, and the Blocked Traffic list in Site Tools is a separate feature that does not affect it. To allow AskyBot:

  1. Open a ticket with SiteGround support
  2. Ask them to exempt the user agent AskyBot from the Anti-Bot AI for your site, not only from the Blocked Traffic list
  3. If support asks for an address instead of a user agent, give them the IP address below
  4. If you use the SG Security WordPress plugin, also check its bot protection settings and allow AskyBot there
  5. If Asky also reads or updates your WordPress pages, ask for the IP address to be exempted as well. See Bot protection blocks reads or updates

Cloudflare

Create a WAF rule that skips bot protection for AskyBot:

  1. Open your Cloudflare dashboard and select the site
  2. Go to Security, then WAF, then Custom rules
  3. Create a rule matching: User Agent contains AskyBot
  4. Set the action to Skip, covering Super Bot Fight Mode and managed challenges

The free Bot Fight Mode cannot be skipped by a custom rule. If it is on, add an IP Access rule for the IP address with the action Allow, under Security, WAF, Tools: Bot Fight Mode does not run on a request that rule has already allowed.

Other hosting or firewalls

Look for bot protection, WAF, or firewall settings in your hosting panel or CDN and add an allow (whitelist) rule for the user agent AskyBot. Allowing by user agent is the better rule because it covers every request. If your firewall can only allow by address, use the IP address below.

Also confirm your robots.txt does not disallow AskyBot. It obeys robots.txt rules automatically.

Once AskyBot is allowed, blocked pages are re-checked automatically within about a day and the issue in Asky clears itself. No re-scan is needed.

IP address

Most AskyBot requests come from cloud infrastructure whose addresses change. When a site refuses those requests, AskyBot retries from this fixed address:

77.83.51.191

Allowing this address lets those retries through. It does not cover the first attempt, so where your firewall supports it, allow the user agent AskyBot as well. The address is stable and this page is updated before it ever changes.

When Asky reads or updates your WordPress site and its bot protection blocks the request, Asky retries from this address too. Those requests do not carry the AskyBot user agent, so allow the address.

Blocking AskyBot

If you do not want AskyBot to visit your website, add this to your robots.txt. AskyBot checks robots.txt before crawling and will respect it:

User-agent: AskyBot
Disallow: /

Questions

If you have questions about AskyBot or believe it is not behaving as described here, contact us at support@askylabs.com.

Last updated on
Reddit