POST /v1/crawl answers at once with an id, and these calls let you follow it: read its status and finished pages, read its errors, cancel it, and list your crawls. All of them are free and need the same API key. A crawl and its pages are kept for 24 hours.
Poll a crawl and read every page
PollGET /v1/crawl/{id} until status is completed, failed or cancelled. Then follow next until the answer has none. This reads every page of a crawl.
limit you sent to the crawl, which is how many pages it reads in all. A crawl with limit 30 is usually read in one call once it is done, and one with limit 250 in three.
next appears only when more finished pages are waiting, so it never leads to an empty data. It is a full URL on api.stophy.dev. Call it with the same API key.
Read the status
While a crawl runs,status is scraping. This real answer came right after the crawl started: one page is known and none is done.
Read page by page
GET /v1/crawl/{id}/results returns the finished pages with more detail on each, 20 at a time. Send limit up to 100, and send cursor from the answer to read the next part. Keep going until cursor is missing.
url the crawl ended on, the requestedUrl it followed, the page’s depth (links from the start page), and its own costUsd. Sending cursor=eqfj7g9apage_2 with the same limit returned the last page, with no cursor.
Read the errors
GET /v1/crawl/{id}/errors lists the pages the crawl could not read, and why. This crawl had none.
errors has the page’s url, a stable code, a message, whether trying again can help (retryable) and when it failed (at). The list holds up to 1,000 errors, and isTruncated says when there were more. A failed page costs nothing. Pages that the site’s robots rules disallow are skipped, not counted as failures, so robotsBlocked stays empty.
A page that answers with a 404 or 410 status is not an error. It is a finished page with statusCode 404 and metadata.error, and it costs nothing.
Cancel a crawl
DELETE /v1/crawl/{id} stops a crawl and keeps the pages it already read. You are charged for those pages and no more.
409 jobFinished.
List your crawls
GET /v1/crawl lists your crawls, newest first, 20 at a time. Send status to keep one status, limit up to 100, and cursor from the answer for the next part. A page can hold fewer than limit, so keep going until cursor is missing.
GET /v1/crawl/active lists only the crawls that are waiting or running, each with its id, url, status (queued or scraping), createdAt and the options it was started with.
Get told with a webhook
Sendwebhook when you start the crawl, instead of polling. It is a URL, or an object. Stophy sends a JSON POST when the crawl starts, for each finished page, and when it ends.
Each call is signed in a
Stophy-Signature header, t=<unix seconds>,v1=<hex>, where v1 is the HMAC-SHA256 of <t>.<body> with your webhook secret. A call that gets no 2xx answer is sent again, up to 3 times. If your endpoint answers slowly, a page event can carry only the page’s url. Read that page with the status call. See Webhooks.
Options
These fields go in the body ofPOST /v1/crawl.
Batches work the same way
A batch scrape is read like a crawl, athttps://api.stophy.dev/v1/batch/scrape/<id>, with /results and /errors under it. Its pages are kept for 24 hours as well.
Limits and errors
- A status answer holds up to 100 pages or 2 MB.
/resultsholds up to 100 pages, 20 by default. - An
idthat does not exist, belongs to another account, or has expired returns404 notFound. - A
cursoris only good for the crawl and thelimitit came from. Send it unchanged. - A crawl that has failed shows
statusfailedand the reason inerror. - Without an API key, every call returns
401 apiKeyRequired.
Price
Free. Reading, listing, cancelling and reading errors cost nothing. The pages a crawl reads are charged when they finish, at the price in Crawl.Related
- Crawl overview starts a crawl.
- Scope and limits chooses the pages.
- Page content chooses what each page returns.
- Webhooks explains signatures and retries.