Skip to Content
APIEndpointsMost-crawled pages

Most-crawled pages

The paths on your site that AI crawlers fetch most, ordered by total hits. In other words: which of your content the answer engines are actually reading.

Read alongside Website pages, which gives the citation rate per page. A page crawled heavily but never cited is a content problem; a page never crawled is a crawlability problem. The two need different fixes.

GET /v1/brands/{brand_id}/crawler/top-paths

Authorization

HeaderTypeRequired
AuthorizationBearer asky_sk_…required

Path parameters

ParameterTypeRequiredDescription
brand_iduuidrequiredFrom List brands

Query parameters

ParameterTypeRequiredDescription
limitintegeroptionalRows per page, 1 to 100. Defaults to 20.
cursorstringoptionalOpaque cursor from the previous response. Treat it as a token, never construct one.
start_dateYYYY-MM-DDoptionalInclusive lower bound, YYYY-MM-DD in UTC. Narrows the population before metrics are computed.
end_dateYYYY-MM-DDoptionalInclusive upper bound, YYYY-MM-DD in UTC. Narrows the population before metrics are computed.
scopestringoptionalWhich requests count. pages (default) keeps the visits to the brand’s own pages plus robots.txt, llms.txt and sitemaps, as Crawler Logs shows them; all adds every other request, old links and scanners included.

Request

curl --request GET \ --url 'https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler/top-paths?limit=50' \ --header 'Authorization: Bearer <api-key>'

Response

{ "data": [ { "path": "/blog/ai-visibility-guide", "total": 214, "by_bot": { "ChatGPT": 121, "Perplexity": 34, "Claude": 12, "Google": 42, "Microsoft": 2, "Other": 3 }, "by_crawler": { "GPTBot": 98, "ChatGPT-User": 23, "PerplexityBot": 34 }, "last_visit": "2026-08-13T06:41:19.220Z" } ], "pagination": { "limit": 20, "has_more": true, "next_cursor": "eyJ2IjoxLCJvIjoyMCwibCI6MjB9" }, "meta": { "period": { "start_date": "2026-07-14", "end_date": "2026-08-13" }, "scope": "pages", "timezone": "UTC", "total_paths": 318, "returned": 50 }, "request_id": "req_1d8f43ba09e747c2b310" }

Response fields

FieldTypeDescription
meta.periodobjectrequired
meta.timezonestringrequired. Always UTC
meta.scopestringrequired. pages or all: which requests the figures count
dataobject[]required
data[].pathstringrequired. Path only, without host
data[].totalintegerrequired
data[].by_botobjectrequired. Counts per operator bucket
data[].by_crawlerobjectrequired. Counts per exact bot name
data[].last_visittimestamptz | nullMost recent fetch of this path
meta.total_pathsintegerrequired. Distinct paths crawled in the window, before limit
meta.returnedintegerrequired
paginationobjectrequired. See pagination
request_idstringrequired. Also returned as the x-request-id header. Quote it when contacting support

Errors

StatusCodeWhen
400invalid_requestA parameter is unknown or malformed. The message names it
401invalid_tokenThe key is missing, malformed, or does not exist
401token_expiredThe key passed its expiry date
401token_revokedThe key was revoked
404not_foundThe brand does not exist, or this key is not allowed to reach it. The two are deliberately indistinguishable
429rate_limitedOver 300 requests per minute. See the Retry-After header

Notes

Your pages by default. The figures count the visits to pages Asky lists for the brand, plus robots.txt, llms.txt and sitemaps, as Crawler Logs does. Requests for anything else (old links, scanners probing for .env files under a crawler’s name) are left out unless you pass scope=all. For a brand with no pages listed, both scopes count every request.

Bot traffic only. Ordinary human visits are excluded before anything is counted, and they are the large majority of raw log rows. A figure here is therefore never comparable to a pageview count from your analytics tool.

Compare total_paths against returned to know whether you are seeing the whole picture or the top slice of a longer tail. If has_more is true, page forward with next_cursor to see the rest — total_paths reports how many exist regardless of which page you are on.

Last updated on