Skip to main content
A batch can fail in three places: the call that starts it, a page inside it, and the batch as a whole. This page covers each, and how to stop a batch you no longer need.

Cancel a batch

DELETE /v1/batch/scrape/{id} stops a batch from reading more pages. It answers with status cancelled at once, and the pages already read are kept.
This real batch had 3 pages and was cancelled right after it started. Reading it showed status cancelled, total 3, completed 0 and costUsd 0. no page was read, so nothing was charged. A batch cancelled later is charged for the pages it finished. Reading the batch then gives its final counts, and data holds the pages read. Cancelling a batch that already ended returns 409 jobFinished.
To remove a finished batch’s pages, see Delete the results. A batch also deletes itself 24 hours after it starts.

Jobs at once

Batches share three limits with crawls, extracts and agent runs. They depend on your tier, which GET /v1/limits returns. A batch that starts while your running jobs are at the limit waits. Its answer has status scraping and a queuePosition, and it starts when a place frees up. A batch over the waiting limit returns 429 rateLimited: Too many jobs: you already have 6 jobs waiting or running. Send the call again after the number of seconds in retryAfterSeconds. On the free tier that is 1 running and 5 waiting. retryAfterSeconds is 30. See Rate limits. A batch holds 1 to 10,000 URLs, and is kept for 24 hours. Pages are read at most 4 at once whatever maxConcurrency you send. Read the results as pages finish: a status read already shows the pages that are done.

Pages that fail

A failed page never makes the call fail. The batch goes on, the page is listed in errors with a code, and a failed page costs nothing.
Every code is in API errors. Read a batch’s errors with GET /v1/batch/scrape/{id}/errors or in the errors of a status read. See Results. A batch ends failed when the batch itself stops, not when some pages fail. It says why in error, with one of these messages.
  • Your balance ran out. The pages already read are kept. Add funds, then start a new job for the pages left.
  • No page could be read. See errors for the reason on each page.
  • We could not finish this job. The pages already read are kept. Start a new job for the pages left.

Retry the failed pages

Collect the url of every retryable error, then submit them as a new batch.
Send the same formats and options as the first batch. When the first batch is still running, add the URLs to it instead with appendToId. See Submit. Wait a few seconds before you retry sourceUnavailable and sourceRefused, and do not retry a code marked No.

Errors from the calls

Failed calls cost nothing. Troubleshooting has a retry loop.