AskyBot

AskyBot is the web crawler operated by Asky. It visits websites that are monitored on the Asky platform to analyze their content, technical health, and visibility in AI search engines. It only crawls websites whose owners or their agencies use Asky. Single pages on other websites that AI engines cite are checked by a separate fetcher, Asky-SourceCheck.

How AskyBot identifies itself

AskyBot carries one of these two user agent strings:


Mozilla/5.0 (compatible; AskyBot/1.0; +https://askylabs.com)
AskyBot/1.0 (+https://askylabs.com)

The first is the standard form. The second is used when a firewall refuses the standard form, which some bot protection products do to any unfamiliar crawler in that format. Both contain AskyBot, so an allow rule that matches on AskyBot covers every request.

How AskyBot behaves

It reads pages when they are needed. Asky lists every page in your sitemaps and reads a page when a feature needs it, and reads it again when its copy is older than that feature allows. It does not crawl your site on a schedule.
At night it reads only robots.txt and your sitemaps, to find new and removed pages. When your site is refusing AskyBot, it also tries a few of the refused pages each night, so it notices when you let it in.
It obeys robots.txt rules that name AskyBot. See robots.txt below.
It crawls at a low, adaptive rate per website and backs off when a server responds slowly, returns errors, or asks for a retry delay. A Crawl-delay in a group for AskyBot sets its pace.
It uses conditional requests (ETag and Last-Modified) wherever possible, so repeat visits to unchanged pages transfer almost no data.
It is identified by its user agent. Most requests come from cloud infrastructure whose addresses change. When a site refuses those, AskyBot retries from a fixed address you can allowlist, listed under IP address below. As a last resort on your own domains only, it can read a page under a standard browser user agent; to switch that off for your site, contact us.

robots.txt

AskyBot reads your robots.txt before it reads a page and obeys every group for the user agent AskyBot: the longest matching rule wins, Allow wins a tie, and * and $ work as in Google’s rules. A Crawl-delay in that group sets how fast it reads.

Generic groups (User-agent: *) are not applied, because you or your agency asked Asky to read your site and such groups usually exist to slow down crawlers you never invited. To have AskyBot obey them on your site as well, contact us. To keep AskyBot out of a section, name it:


User-agent: AskyBot
Disallow: /private/

When your robots.txt cannot be reached (a server error or a timeout), AskyBot keeps using the last copy it read. A robots.txt that does not exist means no rules. Pages your robots.txt keeps AskyBot from are listed in Asky but not read, audited or charged.

Allowing AskyBot

Bot protection products sometimes challenge AskyBot with a captcha or block it with a 403 response. When that happens, the affected pages cannot be analyzed and the website owner loses monitoring coverage. Asky flags this in the app as an AskyBot is blocked issue under Technical opportunities.

To allow AskyBot, add an allow rule for the user agent AskyBot in your hosting or firewall settings:

SiteGround

SiteGround’s Anti-Bot AI challenges automated traffic with a captcha page. There is no setting in Site Tools that exempts a crawler from it, and the Blocked Traffic list in Site Tools is a separate feature that does not affect it. To allow AskyBot:

Open a ticket with SiteGround support
Ask them to exempt the user agent AskyBot from the Anti-Bot AI for your site, not only from the Blocked Traffic list
If support asks for an address instead of a user agent, give them the IP address below
If you use the SG Security WordPress plugin, also check its bot protection settings and allow AskyBot there
If Asky also reads or updates your WordPress pages, ask for the IP address to be exempted as well. See Bot protection blocks reads or updates

Cloudflare

Create a WAF rule that skips bot protection for AskyBot:

Open your Cloudflare dashboard and select the site
Go to Security, then WAF, then Custom rules
Create a rule matching: User Agent contains AskyBot
Set the action to Skip, covering Super Bot Fight Mode, managed rules and rate limiting rules

The free Bot Fight Mode cannot be skipped by a custom rule. If it is on, add an IP Access rule for the IP address with the action Allow, under Security, WAF, Tools: Bot Fight Mode does not run on a request that rule has already allowed.

Vercel

Vercel’s firewall can rate limit AskyBot, answering its requests with a 429 response. Asky then slows down on your site, so audits take longer, and pages still refused at the end of an audit are left out of it. Add a custom rule for AskyBot in your project’s firewall:

Open your project in the Vercel dashboard and go to Firewall
Add a custom rule matching: User Agent contains AskyBot
Choose the action:
- Bypass lets AskyBot past your other custom rules
- Rate Limit keeps AskyBot under a limit of its own, for example 300 requests per 60 seconds. When AskyBot reaches it, it slows down and tries those pages again
Move the rule to the top of your custom rules, so it is checked first
Publish the changes. Vercel keeps firewall changes as a draft until they are published

With a rate limit, also add the condition User Agent does not contain AskyBot to any rate limit rule you already have, so AskyBot is counted against its own limit only.

The same rule from the Vercel CLI, with a rate limit:


vercel firewall rules add "AskyBot" --condition '{"type":"user_agent","op":"sub","value":"AskyBot"}' --action rate_limit --rate-limit-window 60 --rate-limit-requests 300 --yes
vercel firewall rules reorder "AskyBot" --first --yes
vercel firewall publish --yes

Other hosting or firewalls

Look for bot protection, WAF, or firewall settings in your hosting panel or CDN and add an allow (whitelist) rule for the user agent AskyBot. Allowing by user agent is the better rule because it covers every request. If your firewall can only allow by address, use the IP address below.

Also confirm your robots.txt does not disallow AskyBot. It obeys every rule that names it.

Once AskyBot is allowed, blocked pages are re-checked automatically within about a day and the issue in Asky clears itself. No re-scan is needed.

IP address

Most AskyBot requests come from cloud infrastructure whose addresses change. When a site refuses those requests, AskyBot retries from this fixed address:


77.83.51.191

Allowing this address lets those retries through. It does not cover the first attempt, so where your firewall supports it, allow the user agent AskyBot as well. The address is stable and this page is updated before it ever changes.

When Asky reads or updates your WordPress site and its bot protection blocks the request, Asky retries from this address too. Those requests do not carry the AskyBot user agent, so allow the address.

Blocking AskyBot

If you do not want AskyBot to visit your website, add this to your robots.txt. AskyBot checks robots.txt before it reads a page and will respect it:


User-agent: AskyBot
Disallow: /

A rule for every crawler (User-agent: *) alone does not block AskyBot on a site that uses Asky. See robots.txt.

Source checks

When an AI engine cites a page on another website, Asky checks that single page: whether it mentions your brand, whether its links work, and what it says about a claim. Those requests use their own name:


Mozilla/5.0 (compatible; Asky-SourceCheck/1.0; +https://askylabs.com/askybot#source-check)

Asky-SourceCheck fetches one page at a time because an AI answer cited it, never follows links and never crawls a site, in the way ChatGPT-User and Perplexity-User fetch a page a person asked about. Like them it does not apply robots.txt, which governs crawlers. It is not AskyBot.

Questions

If you have questions about AskyBot or believe it is not behaving as described here, contact us at support@askylabs.com.