Import crawler logs

Bulk-imports your own AI-crawler server log rows into Asky’s crawler tracking, in Asky’s fixed schema — for when you already have raw server logs (for example an export from your own hosting) rather than relying solely on the live Cloudflare Worker capture.

Provide either rows (a JSON array of row objects) or csv_text (the same columns as CSV, header row required) — not both. Each row needs path, user_agent and timestamp; ip, referrer and query are optional. bot_name and visit_type are never accepted as input — they are always computed server-side from user_agent, the same classification the live Worker capture uses, so an imported visit and a Worker-captured one are indistinguishable afterward.

This is not a raw-file upload. Both rows and csv_text are plain text in the JSON body.

Requires the write:crawler_logs scope, and the workspace must be enabled for API writes.


POST /v1/brands/{brand_id}/crawler-logs/import

Authorization

Header	Type	Required
`Authorization`	`Bearer asky_sk_…`	required
`Idempotency-Key`	`string`	required

Path parameters

Parameter	Type	Required	Description
`brand_id`	`uuid`	required	From List brands

Body

application/json

Field	Type	Required	Description
`host`	`string`	required	One of the brand’s active domains. See List website hosts
`rows`	`object[]`, 1-20,000	one of `rows`/`csv_text`	Row objects: `{ path, user_agent, timestamp, ip?, referrer?, query? }`
`csv_text`	`string`	one of `rows`/`csv_text`	CSV with header row `path,user_agent,timestamp,ip,referrer,query`

Row schema

Field	Type	Required	Description
`path`	`string`	required	Must start with `/`
`user_agent`	`string`	required	The raw User-Agent header. Classified server-side into `bot_name`/`visit_type`
`timestamp`	`string` (ISO 8601)	required	When the request happened
`ip`	`string`	optional	IPv4 or IPv6. Dropped (not the row) if it doesn’t parse as one
`referrer`	`string`	optional	Truncated at 2048 characters
`query`	`string`	optional	The request’s query string. Truncated at 2048 characters

Request

cURL


curl --request POST \
  --url 'https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler-logs/import' \
  --header 'Authorization: Bearer <api-key>' \
  --header 'Idempotency-Key: 5b0e77d2-13a9-4f6a-8ce1-9a3f5e21c0aa' \
  --header 'Content-Type: application/json' \
  --data '{
    "host": "www.askylabs.com",
    "rows": [
      {
        "path": "/pricing",
        "user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)",
        "timestamp": "2026-08-01T10:15:00Z",
        "ip": "203.0.113.4",
        "referrer": null,
        "query": null
      }
    ]
  }'

Python


import requests
 
url = "https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler-logs/import"
 
headers = {
    "Authorization": "Bearer <api-key>",
    "Idempotency-Key": "5b0e77d2-13a9-4f6a-8ce1-9a3f5e21c0aa",
    "Content-Type": "application/json",
}
payload = {
    "host": "www.askylabs.com",
    "rows": [
        {
            "path": "/pricing",
            "user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)",
            "timestamp": "2026-08-01T10:15:00Z",
            "ip": "203.0.113.4",
        }
    ],
}
 
response = requests.post(url, json=payload, headers=headers)
print(response.text)

JavaScript


const options = {
  method: 'POST',
  headers: {
    Authorization: 'Bearer <api-key>',
    'Idempotency-Key': '5b0e77d2-13a9-4f6a-8ce1-9a3f5e21c0aa',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    host: 'www.askylabs.com',
    rows: [
      {
        path: '/pricing',
        user_agent: 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)',
        timestamp: '2026-08-01T10:15:00Z',
        ip: '203.0.113.4',
      },
    ],
  }),
};
 
fetch('https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler-logs/import', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

PHP


<?php
 
$curl = curl_init();
 
curl_setopt_array($curl, [
  CURLOPT_URL => "https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler-logs/import",
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_CUSTOMREQUEST => "POST",
  CURLOPT_POSTFIELDS => json_encode([
    "host" => "www.askylabs.com",
    "rows" => [
      [
        "path" => "/pricing",
        "user_agent" => "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)",
        "timestamp" => "2026-08-01T10:15:00Z",
        "ip" => "203.0.113.4",
      ],
    ],
  ]),
  CURLOPT_HTTPHEADER => [
    "Authorization: Bearer <api-key>",
    "Idempotency-Key: 5b0e77d2-13a9-4f6a-8ce1-9a3f5e21c0aa",
    "Content-Type: application/json"
  ],
]);
 
$response = curl_exec($curl);
$err = curl_error($curl);
 
curl_close($curl);
 
if ($err) {
  echo "cURL Error #:" . $err;
} else {
  echo $response;
}

Go


package main
 
import (
    "fmt"
    "net/http"
    "strings"
    "io"
)
 
func main() {
 
    url := "https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler-logs/import"
 
    payload := strings.NewReader(`{"host": "www.askylabs.com", "rows": [{"path": "/pricing", "user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)", "timestamp": "2026-08-01T10:15:00Z", "ip": "203.0.113.4"}]}`)
 
    req, _ := http.NewRequest("POST", url, payload)
 
    req.Header.Add("Authorization", "Bearer <api-key>")
    req.Header.Add("Idempotency-Key", "5b0e77d2-13a9-4f6a-8ce1-9a3f5e21c0aa")
    req.Header.Add("Content-Type", "application/json")
 
    res, _ := http.DefaultClient.Do(req)
 
    defer res.Body.Close()
    body, _ := io.ReadAll(res.Body)
 
    fmt.Println(string(body))
 
}

Ruby


require 'uri'
require 'net/http'
require 'json'
 
url = URI("https://api.askylabs.com/v1/brands/242546de-4cee-4a7d-952e-f6aa60c63ef8/crawler-logs/import")
 
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
 
request = Net::HTTP::Post.new(url)
request["Authorization"] = 'Bearer <api-key>'
request["Idempotency-Key"] = '5b0e77d2-13a9-4f6a-8ce1-9a3f5e21c0aa'
request["Content-Type"] = 'application/json'
request.body = {
  host: "www.askylabs.com",
  rows: [
    {
      path: "/pricing",
      user_agent: "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)",
      timestamp: "2026-08-01T10:15:00Z",
      ip: "203.0.113.4",
    },
  ],
}.to_json
 
response = http.request(request)
puts response.read_body

Response

200


{
  "data": {
    "import": {
      "id": "9c3e1a7b-4d2f-4e6a-8c1b-2f7a9d3e5b60",
      "host": "www.askylabs.com",
      "format": "json",
      "status": "queued",
      "total_rows": 1,
      "inserted_rows": 0,
      "skipped_rows": 0,
      "error_message": null,
      "created_at": "2026-08-31T14:20:00.000Z",
      "updated_at": "2026-08-31T14:20:00.000Z",
      "completed_at": null
    }
  },
  "request_id": "req_6a1c8f0b3e9d47a5b6c2"
}

400


{
  "error": {
    "code": "invalid_request",
    "message": "Payload has 24,118 rows, over the 20,000-row limit per call. Split into multiple import calls.",
    "request_id": "req_2d61f0ac48b34e59c7bd"
  }
}

403 plan


{
  "error": {
    "code": "plan_required",
    "message": "This workspace is not enabled for API writes. Contact support to have it turned on.",
    "request_id": "req_9d4a7c1e3b5f48d0ac2e"
  }
}

404


{
  "error": {
    "code": "not_found",
    "message": "Not found.",
    "request_id": "req_5b0e77d213a94f6a8ce1"
  }
}

429


{
  "error": {
    "code": "rate_limited",
    "message": "Too many requests. Retry after 24 seconds.",
    "request_id": "req_9a3f5e21c0aa47b190d2"
  }
}

Response fields

Field	Type	Description
`data.import.id`	`uuid`	required. Our own internal reference for this import; there is no endpoint to look it up later
`data.import.host`	`string`	required. The resolved domain the import targeted
`data.import.format`	`string`	required. `json` or `csv`, whichever you sent
`data.import.status`	`string`	required. Always `queued` at creation
`data.import.total_rows`	`integer`	required. Rows found in the payload
`data.import.inserted_rows`	`integer`	required. Always `0` at creation — populated once processing completes
`data.import.skipped_rows`	`integer`	required. Always `0` at creation
`data.import.error_message`	`string \| null`	required. Always `null` at creation
`data.import.created_at` / `updated_at`	`timestamptz`	required
`data.import.completed_at`	`timestamptz \| null`	required. `null` until processing finishes
`request_id`	`string`	required. Also returned as the `x-request-id` header. Quote it when contacting support

Errors

Status	Code	When
400	`invalid_request`	Unknown field, missing `host`, neither or both of `rows`/`csv_text`, malformed JSON/CSV, zero rows, over 20,000 rows, or the payload exceeds 15MB
401	`invalid_token`	The key is missing, malformed, or does not exist
401	`token_expired`	The key passed its expiry date
401	`token_revoked`	The key was revoked
403	`insufficient_scope`	The key does not carry `write:crawler_logs`
403	`plan_required`	This workspace is not enabled for API writes
404	`not_found`	The brand does not exist, this key is not allowed to reach it, or `host` does not match any of the brand’s active domains
429	`rate_limited`	Over 60 requests per minute

Notes

Processing is asynchronous, and there is no status endpoint. This call parses your payload just enough to fail fast (malformed JSON/CSV, too many rows, too large) and returns status: "queued" immediately; the rest of processing happens in the background. There is no endpoint to check progress or final row counts afterward — a 200 response here means the payload was well-formed and queued, not that every row was inserted. If you need to confirm data landed, check the effect downstream (e.g. via the crawler-analytics endpoints) rather than polling this import.

bot_name and visit_type are never accepted from you. They are always computed server-side from user_agent, using the same classification the live Cloudflare Worker capture applies. This keeps an imported visit and a Worker-captured one indistinguishable in every crawler-analytics tool afterward.

A larger dataset is multiple calls, not one bigger call. The 20,000-row cap per call is a deliberate ceiling, not a temporary limitation — split a large historical export into sequential import calls instead of raising it.

Row-level failures do not fail the whole import. A row missing path/user_agent/timestamp, or with an unparseable timestamp, is skipped and counted in skipped_rows — the rest of the import still proceeds. A malformed ip is dropped from just that row (stored as null) rather than skipping the row entirely.

Idempotency-Key is required. Reuse it when retrying an attempt and the original response is replayed rather than a second import being created. See Idempotency.