Web web Crawl Status API

Check a background crawl and read the finished pages

curl --request POST \
  --url https://api.stophy.dev/v1/web/crawl/status \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "id": "<string>",
  "cursor": "<string>"
}
'

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({id: '<string>', cursor: '<string>'})
};

fetch('https://api.stophy.dev/v1/web/crawl/status', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

import requests

url = "https://api.stophy.dev/v1/web/crawl/status"

payload = {
    "id": "<string>",
    "cursor": "<string>"
}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

{
  "success": true,
  "data": {
    "status": "queued",
    "total": 4503599627370495,
    "completed": 4503599627370495,
    "failed": 4503599627370495,
    "creditsUsed": 4503599627370495,
    "errors": [
      {
        "url": "<string>",
        "code": "<string>",
        "message": "<string>"
      }
    ],
    "results": [
      {
        "depth": 0,
        "url": "<string>",
        "title": "<string>",
        "markdown": "<string>"
      }
    ],
    "id": "<string>",
    "message": "<string>",
    "cursor": "<string>"
  },
  "creditsUsed": 123,
  "requestId": "<string>"
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

POST

web

crawl

status

Check a background crawl and read the finished pages

curl --request POST \
  --url https://api.stophy.dev/v1/web/crawl/status \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "id": "<string>",
  "cursor": "<string>"
}
'

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({id: '<string>', cursor: '<string>'})
};

fetch('https://api.stophy.dev/v1/web/crawl/status', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

import requests

url = "https://api.stophy.dev/v1/web/crawl/status"

payload = {
    "id": "<string>",
    "cursor": "<string>"
}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

{
  "success": true,
  "data": {
    "status": "queued",
    "total": 4503599627370495,
    "completed": 4503599627370495,
    "failed": 4503599627370495,
    "creditsUsed": 4503599627370495,
    "errors": [
      {
        "url": "<string>",
        "code": "<string>",
        "message": "<string>"
      }
    ],
    "results": [
      {
        "depth": 0,
        "url": "<string>",
        "title": "<string>",
        "markdown": "<string>"
      }
    ],
    "id": "<string>",
    "message": "<string>",
    "cursor": "<string>"
  },
  "creditsUsed": 123,
  "requestId": "<string>"
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

Check a background crawl and read the finished pages.

Request

id is required.

import { Stophy } from "stophy";

const stophy = new Stophy({ apiKey: process.env.STOPHY_API_KEY });
const result = await stophy.web.crawl.status({ id: "job_Ab3dE5gH7jK9mN1pQ3sT5v" });
console.log(result.data.results);

from stophy import Stophy

stophy = Stophy()  # reads STOPHY_API_KEY
result = stophy.web.crawl.status(id="job_Ab3dE5gH7jK9mN1pQ3sT5v")
print(result["data"]["results"])

curl -X POST https://api.stophy.dev/v1/web/crawl/status \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "content-type: application/json" \
  -d '{"id":"job_Ab3dE5gH7jK9mN1pQ3sT5v"}'

stophy web crawl status job_Ab3dE5gH7jK9mN1pQ3sT5v

Response

If successful, the response body contains the standard envelope. data contains id, status, total, completed, failed, creditsUsed, message, errors, results and cursor. The following example is a real response, truncated: each list shows one item and long strings are cut.

{
  "success": true,
  "data": {
    "id": "job_Qg9km6Ss5DAJktZgPYOUkA",
    "status": "completed",
    "total": 4,
    "completed": 4,
    "failed": 0,
    "creditsUsed": 4,
    "results": [
      {
        "url": "https://bun.com/docs",
        "title": "Welcome to Bun | Bun Docs",
        "depth": 0,
        "markdown": "[Bun RuntimeA fast JavaScript runtime designed as a drop-in replacement for Node.jsGet started with `bun…"
      }
    ]
  },
  "creditsUsed": 0,
  "requestId": "2fe9dbad-fc48-4890-9751-0b24b596035b"
}

Field	Type	Description
`total`	integer	The total number of matching results, across all pages.
`depth`	integer	The nesting level of the comment in the thread. Top-level comments are 0.

For every field, see the response schema on this page.

Send the cursor from the last answer to get only the pages finished since. total is the number of pages found so far and grows while the crawl runs. A crawl and its pages are kept for 24 hours.

Optional parameters

Parameter	Type	Description
`cursor`	string	The pagination token from the previous response.

Billing

Each successful request consumes 0 credits, regardless of the number of items returned. Requests that fail, or that return no items, are not billed.

Pagination

Results are paginated. To retrieve the next page, pass the value of data.cursor as cursor in a subsequent request. A response that omits cursor is the last page.

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Query Parameters

maxTextLength

integer

Cut every text value in the response to this many characters. A value that was cut ends with …, and a page whose text was cut has isTruncated set to true. A screenshot is never cut. Leave it out to get the full text.

Required range: x >= 1

Body

application/json

string

required

The id web.crawl.start returned.

Pattern: ^job_[A-Za-z0-9_-]{22}$

cursor

string

The cursor from the previous response. Send it as is.

Minimum string length: 1

Response

The data, and the credits this call used.

success

any

required

data

object

required

Show child attributes

creditsUsed

integer

required

requestId

string

required

Using the API

Catalog

Account

Search and AI answers

Video

Social

Places and travel

Jobs

Shopping

Apps

Real estate

Ads

Web web Crawl Status API

Request

Response

Optional parameters

Billing

Authorizations

Query Parameters

Body

Response

​Request

​Response

​Optional parameters

​Billing

​Pagination

Authorizations

Query Parameters

Body

Response

Request

Response

Optional parameters

Billing

Pagination