Most-crawled pages

The paths on your site that AI crawlers fetch most, ordered by total hits. In other words: which of your content the answer engines are actually reading.

Read alongside Website pages, which gives the citation rate per page. A page crawled heavily but never cited is a content problem; a page never crawled is a crawlability problem. The two need different fixes.


GET /v1/brands/{brand_id}/crawler/top-paths

Authorization

Header	Type	Required
`Authorization`	`Bearer asky_sk_…`	required

Path parameters

Parameter	Type	Required	Description
`brand_id`	`uuid`	required	From List brands

Query parameters

Parameter	Type	Required	Description
`limit`	`integer`	optional	Rows per page, 1 to 100. Defaults to 20.
`cursor`	`string`	optional	Opaque cursor from the previous response. Treat it as a token, never construct one.
`start_date`	`YYYY-MM-DD`	optional	Inclusive lower bound, YYYY-MM-DD in UTC. Narrows the population before metrics are computed.
`end_date`	`YYYY-MM-DD`	optional	Inclusive upper bound, YYYY-MM-DD in UTC. Narrows the population before metrics are computed.

Request

cURL


curl --request GET \
  --url 'https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler/top-paths?limit=50' \
  --header 'Authorization: Bearer <api-key>'

Python


import requests
 
url = "https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler/top-paths?limit=50"
 
headers = {"Authorization": "Bearer <api-key>"}
 
response = requests.get(url, headers=headers)
 
print(response.text)

JavaScript


const options = {method: 'GET', headers: {Authorization: 'Bearer <api-key>'}};
 
fetch('https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler/top-paths?limit=50', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

PHP


<?php
 
$curl = curl_init();
 
curl_setopt_array($curl, [
  CURLOPT_URL => "https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler/top-paths?limit=50",
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_CUSTOMREQUEST => "GET",
  CURLOPT_HTTPHEADER => [
    "Authorization: Bearer <api-key>"
  ],
]);
 
$response = curl_exec($curl);
$err = curl_error($curl);
 
curl_close($curl);
 
if ($err) {
  echo "cURL Error #:" . $err;
} else {
  echo $response;
}

Go


package main
 
import (
    "fmt"
    "net/http"
    "io"
)
 
func main() {
 
    url := "https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler/top-paths?limit=50"
 
    req, _ := http.NewRequest("GET", url, nil)
 
    req.Header.Add("Authorization", "Bearer <api-key>")
 
    res, _ := http.DefaultClient.Do(req)
 
    defer res.Body.Close()
    body, _ := io.ReadAll(res.Body)
 
    fmt.Println(string(body))
 
}

Ruby


require 'uri'
require 'net/http'
 
url = URI("https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler/top-paths?limit=50")
 
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
 
request = Net::HTTP::Get.new(url)
request["Authorization"] = 'Bearer <api-key>'
 
response = http.request(request)
puts response.read_body

Response

200


{
  "data": [
    {
      "path": "/blog/ai-visibility-guide",
      "total": 214,
      "by_bot": {
        "ChatGPT": 121,
        "Perplexity": 34,
        "Claude": 12,
        "Google": 42,
        "Microsoft": 2,
        "Other": 3
      },
      "by_crawler": {
        "GPTBot": 98,
        "ChatGPT-User": 23,
        "PerplexityBot": 34
      },
      "last_visit": "2026-08-13T06:41:19.220Z"
    }
  ],
  "pagination": {
    "limit": 20,
    "has_more": true,
    "next_cursor": "eyJ2IjoxLCJvIjoyMCwibCI6MjB9"
  },
  "meta": {
    "period": {
      "start_date": "2026-07-14",
      "end_date": "2026-08-13"
    },
    "timezone": "UTC",
    "total_paths": 318,
    "returned": 50
  },
  "request_id": "req_1d8f43ba09e747c2b310"
}

401


{
  "error": {
    "code": "token_expired",
    "message": "This API key has expired. Roll it in Settings to get a replacement.",
    "request_id": "req_8f21c0aa47b1490d2e63"
  }
}

404


{
  "error": {
    "code": "not_found",
    "message": "Not found.",
    "request_id": "req_5b0e77d213a94f6a8ce1"
  }
}

429


{
  "error": {
    "code": "rate_limited",
    "message": "Too many requests. Retry after 24 seconds.",
    "request_id": "req_31c9ba7f0e5d4a118cc2"
  }
}

Response fields

Field	Type	Description
`data.period`	`object`	required
`data.timezone`	`string`	required. Always `UTC`
`data.paths`	`object[]`	required
`data.paths[].path`	`string`	required. Path only, without host
`data.paths[].total`	`integer`	required
`data.paths[].by_bot`	`object`	required. Counts per operator bucket
`data.paths[].by_crawler`	`object`	required. Counts per exact bot name
`data.paths[].last_visit`	`timestamptz \| null`	Most recent fetch of this path
`data.total_paths`	`integer`	required. Distinct paths crawled in the window, before `limit`
`data.returned`	`integer`	required
`request_id`	`string`	required. Also returned as the `x-request-id` header. Quote it when contacting support

Errors

Status	Code	When
400	`invalid_request`	A parameter is unknown or malformed. The message names it
401	`invalid_token`	The key is missing, malformed, or does not exist
401	`token_expired`	The key passed its expiry date
401	`token_revoked`	The key was revoked
404	`not_found`	The brand does not exist, or this key is not allowed to reach it. The two are deliberately indistinguishable
429	`rate_limited`	Over 300 requests per minute. See the `Retry-After` header

Notes

Bot traffic only. Ordinary human visits are excluded before anything is counted, and they are the large majority of raw log rows. A figure here is therefore never comparable to a pageview count from your analytics tool.

Compare total_paths against returned to know whether you are seeing the whole picture or the top slice of a longer tail.