Cancel a batch
DELETE /v1/batch/scrape/{id} stops a batch from reading more pages. It answers with status cancelled at once, and the pages already read are kept.
status cancelled, total 3, completed 0 and costUsd 0. no page was read, so nothing was charged. A batch cancelled later is charged for the pages it finished. Reading the batch then gives its final counts, and data holds the pages read.
Cancelling a batch that already ended returns 409 jobFinished.
Jobs at once
Batches share three limits with crawls, extracts and agent runs. They depend on your tier, whichGET /v1/limits returns.
A batch that starts while your running jobs are at the limit waits. Its answer has
status scraping and a queuePosition, and it starts when a place frees up. A batch over the waiting limit returns 429 rateLimited: Too many jobs: you already have 6 jobs waiting or running. Send the call again after the number of seconds in retryAfterSeconds. On the free tier that is 1 running and 5 waiting. retryAfterSeconds is 30. See Rate limits.
A batch holds 1 to 10,000 URLs, and is kept for 24 hours. Pages are read at most 4 at once whatever maxConcurrency you send. Read the results as pages finish: a status read already shows the pages that are done.
Pages that fail
A failed page never makes the call fail. The batch goes on, the page is listed inerrors with a code, and a failed page costs nothing.
Every code is in API errors. Read a batch’s errors with
GET /v1/batch/scrape/{id}/errors or in the errors of a status read. See Results.
A batch ends failed when the batch itself stops, not when some pages fail. It says why in error, with one of these messages.
Your balance ran out. The pages already read are kept. Add funds, then start a new job for the pages left.No page could be read. See errors for the reason on each page.We could not finish this job. The pages already read are kept. Start a new job for the pages left.
Retry the failed pages
Collect theurl of every retryable error, then submit them as a new batch.
formats and options as the first batch. When the first batch is still running, add the URLs to it instead with appendToId. See Submit. Wait a few seconds before you retry sourceUnavailable and sourceRefused, and do not retry a code marked No.
Errors from the calls
Failed calls cost nothing. Troubleshooting has a retry loop.
Related
- Batches explains the statuses and the price.
- Submit a batch covers webhooks,
appendToIdandIdempotency-Key. - Results reads the pages and deletes them.
- Rate limits lists every limit by tier.
- Cancel reference lists every field.