status is final, then read its pages, its errors, or both. All of these calls are free.
Check status
Get an API key in the Quickstart.GET /v1/batch/scrape/{id} returns the counts and the finished pages. Poll it every few seconds while status is scraping.
A page that answered
404 or 410 is not an error. It counts as completed, its metadata.error says what happened, and it costs nothing.
Read the results in parts
GET /v1/batch/scrape/{id}/results returns the same pages in a shape made for paging. Send limit and cursor.
limit 2. The page text is cut. The cURL tab reads one part, and the TypeScript and Python tabs read every part, 100 results at a time.
limit is 20 by default and at most 100. Send cursor from the answer, with the same limit, to read the next part. A part can hold fewer results than limit, so keep going until there is no cursor. Each result has its own url, creditsUsed and costUsd.
Pages that failed show up in the errors of the part where they landed. This real second part of a two-page batch had no results and one error.
Read the errors
GET /v1/batch/scrape/{id}/errors lists only the pages that could not be read, with the reason for each.
code is one of the error codes. retryable: true means a new batch with that URL can work. robotsBlocked lists pages the site’s robots rules kept Stophy from reading. At most 1,000 errors are listed, and isTruncated is true when there were more. See Limits and errors.
Delete the results
DELETE /v1/batch/scrape/{id}/results removes the stored pages at once. It works once the batch is completed, failed or cancelled. The batch keeps its counts.
results, and message set to The stored results were deleted. Deleting again returns deletedPages of 0. A batch that is still running returns 400 invalidRequest: This batch scrape has not finished, so its results cannot be deleted yet. Wait until its status is completed, failed or cancelled, then send the call again. To stop a batch first, see Limits and errors.
List your batches
GET /v1/batch/scrape lists your batches, newest first, and is free.
limit, so keep sending cursor until it is missing. A batch that stopped early has its reason in message. Run this to confirm that no batch you started is still reading.
Options
Limits and errors
- A batch, its pages and its errors are kept for 24 hours after it starts. After that, reading it returns
404 notFound:No batch scrape with that id. Results are kept for 24 hours.Copy what you need before then. - A status read returns at most 100 pages or 2 MB in one answer.
- An id that is not yours returns
404 notFoundtoo.
Related
- Batches explains the statuses and the price.
- Submit a batch sends the list and the options.
- Limits and errors covers cancelling and failed pages.
- Results reference and status reference list every field.