Skip to main content
A crawl runs in the background. POST /v1/crawl answers at once with an id, and these calls let you follow it: read its status and finished pages, read its errors, cancel it, and list your crawls. All of them are free and need the same API key. A crawl and its pages are kept for 24 hours.

Poll a crawl and read every page

Poll GET /v1/crawl/{id} until status is completed, failed or cancelled. Then follow next until the answer has none. This reads every page of a crawl.
A status answer holds up to 100 finished pages or 2 MB, whichever comes first. That is separate from the limit you sent to the crawl, which is how many pages it reads in all. A crawl with limit 30 is usually read in one call once it is done, and one with limit 250 in three. next appears only when more finished pages are waiting, so it never leads to an empty data. It is a full URL on api.stophy.dev. Call it with the same API key.

Read the status

While a crawl runs, status is scraping. This real answer came right after the crawl started: one page is known and none is done.

Read page by page

GET /v1/crawl/{id}/results returns the finished pages with more detail on each, 20 at a time. Send limit up to 100, and send cursor from the answer to read the next part. Keep going until cursor is missing.
This real answer is cut. The Markdown and the metadata of each page are shortened.
Each result has the url the crawl ended on, the requestedUrl it followed, the page’s depth (links from the start page), and its own costUsd. Sending cursor=eqfj7g9apage_2 with the same limit returned the last page, with no cursor.

Read the errors

GET /v1/crawl/{id}/errors lists the pages the crawl could not read, and why. This crawl had none.
Each item of errors has the page’s url, a stable code, a message, whether trying again can help (retryable) and when it failed (at). The list holds up to 1,000 errors, and isTruncated says when there were more. A failed page costs nothing. Pages that the site’s robots rules disallow are skipped, not counted as failures, so robotsBlocked stays empty. A page that answers with a 404 or 410 status is not an error. It is a finished page with statusCode 404 and metadata.error, and it costs nothing.

Cancel a crawl

DELETE /v1/crawl/{id} stops a crawl and keeps the pages it already read. You are charged for those pages and no more.
A crawl that already finished or failed cannot be cancelled. The call returns 409 jobFinished.

List your crawls

GET /v1/crawl lists your crawls, newest first, 20 at a time. Send status to keep one status, limit up to 100, and cursor from the answer for the next part. A page can hold fewer than limit, so keep going until cursor is missing.
GET /v1/crawl/active lists only the crawls that are waiting or running, each with its id, url, status (queued or scraping), createdAt and the options it was started with.

Get told with a webhook

Send webhook when you start the crawl, instead of polling. It is a URL, or an object. Stophy sends a JSON POST when the crawl starts, for each finished page, and when it ends.
Each call is signed in a Stophy-Signature header, t=<unix seconds>,v1=<hex>, where v1 is the HMAC-SHA256 of <t>.<body> with your webhook secret. A call that gets no 2xx answer is sent again, up to 3 times. If your endpoint answers slowly, a page event can carry only the page’s url. Read that page with the status call. See Webhooks.

Options

These fields go in the body of POST /v1/crawl.

Batches work the same way

A batch scrape is read like a crawl, at https://api.stophy.dev/v1/batch/scrape/<id>, with /results and /errors under it. Its pages are kept for 24 hours as well.

Limits and errors

  • A status answer holds up to 100 pages or 2 MB. /results holds up to 100 pages, 20 by default.
  • An id that does not exist, belongs to another account, or has expired returns 404 notFound.
  • A cursor is only good for the crawl and the limit it came from. Send it unchanged.
  • A crawl that has failed shows status failed and the reason in error.
  • Without an API key, every call returns 401 apiKeyRequired.

Price

Free. Reading, listing, cancelling and reading errors cost nothing. The pages a crawl reads are charged when they finish, at the price in Crawl.