Skip to main content
A batch reads a list of URLs in the background. You send the list once, get an id at once, and read the pages when they finish.

Make a request

Get an API key in the Quickstart. Batches need an API key. This request reads four addresses as markdown. One is a page that does not exist, and one is not a website address.
maxAge: 0 reads every page live. See Submit a batch for every option.

Read the response

The call answers at once, before any page is read. Starting a batch costs nothing.
Keep id. url is where to read the batch. invalidURLs lists the addresses that are not public website URLs. They are skipped, and the rest of the batch runs. An address sent twice is read once. A few seconds later, GET /v1/batch/scrape/{id} returns the batch. This real answer is cut to a few lines of each page.
Each page in data has the fields of a scrape with scrapeId set to the batch id. total is the pages in the batch, completed the pages that came back, and failed the pages that could not be read, which are listed in errors. The 404 page came back, so it counts as completed, with its error in metadata, and it cost nothing. costUsd is what the batch has cost so far. expiresAt is when the batch and its pages are deleted. See Results.

Status

completed, cancelled and failed are final. Poll the batch every few seconds, or send a webhook to be told as each page finishes and when the batch ends. See Webhooks.

What a batch takes

You send the list and the options once, and every page is read the way Scrape would read it.

Price

A page costs what a scrape costs: $1 per 1,000 pages. A page that comes back with a 404 or 410 status, or cannot be read, costs nothing. `json` with a schema or a prompt, `summary`, `highlights`, `question`, `product`, `menu` and `changeTracking` in `json` mode add $3 per 1,000 pages each when they return something. Each page is charged when it finishes, and starting the batch is free. A batch that reads 10,000 pages costs $10. See Pricing.

Choose another API

  • Scrape reads one page and answers in the same call.
  • Crawl finds the pages of a site for you and reads them in the background.
  • Map lists the pages of a site, to put in a batch.
  • Monitors read a page on a schedule and tell you when it changes.
  • Browser sessions give you a browser to drive yourself.