Skip to main content
POST /v1/batch/scrape takes a list of urls and the options every page shares, and answers with an id. This page covers what you can send. Results covers how to read the pages.

Send a list

Get an API key in the Quickstart. The options are the ones Scrape takes, sent at the top level next to urls. This request reads two pages as markdown and links, keeping only the main content.
The answer is the one in the overview: id, url, invalidURLs and costUsd. A batch that waits for a free place also has status set to scraping and a queuePosition.

Options

Every scrape option works on a batch, and applies to every page. Those include onlyMainContent, includeTags, excludeTags, maxAge, waitFor, mobile, timeout, actions, location, profile, maxChars, query, checkPromptInjection, redactPII and zeroDataRetention. timeout is how long each page may take, in milliseconds. Browse them in Scrape. A batch takes urls in place of the single url, and has no offset. See the API reference for the full list.

Get told when pages finish

Send webhook as an address, or as an object, and Stophy sends a JSON POST when the batch starts, for each finished page and when it ends.
Each call is signed, and a call that gets no 2xx is sent again, up to 3 times. Webhooks explains how to verify the signature. If your address answers slowly, a page event can carry only the page’s url. Read that page from the batch. An address that is not a public URL returns 400 invalidRequest.

Add URLs to a running batch

Send appendToId with the new urls while the batch is still running. The batch keeps its own options and webhook, and the new URLs are read with them. A URL the batch already has is skipped.
The answer has the same id. This real batch started with 4 URLs, and its status then showed total 5 and all five pages in data. When the batch is over, the call returns 409 jobFinished: This batch scrape already finished, so pages cannot be added. Start a new batch, or read the pages with GET /v1/batch/scrape/{id}. The most pages in one batch is still 10,000, new ones included.

Send the same batch twice safely

Add an Idempotency-Key header, any text you choose. If the same key and the same body arrive again within 24 hours, Stophy returns the first answer instead of starting a second batch.
Sent twice, both answers had the same id, and the second had the header idempotent-replayed: true. Only one batch ran. A different body with the same key starts a new batch, so keep the key and the body together when you retry. Use it for every submit you might send again after a timeout.

Price

A page costs what a scrape costs: $1 per 1,000 pages, plus the extras of the options you send. `checkPromptInjection` adds $2.50 per 1,000 pages when the scan runs. json with a schema or a prompt, summary, highlights, question, product, menu and changeTracking in json mode add $3 per 1,000 pages each when they return something. A PDF adds $0.50 per 1,000 pages of text from its text layer and $2.50 per 1,000 pages of text read from its picture, that come back. Browser actions and CSS rules add nothing. A page that comes back with a 404 or 410 status, or cannot be read, costs nothing. Each page is charged when it finishes, and starting the job is free. See Pricing.

Limits and errors

  • A batch holds at most 10,000 URLs. More returns 400 invalidRequest: A batch holds at most 10000 pages. Send fewer.
  • An empty urls returns 400 invalidRequest: urls: Send at least 1 item. A list with no public website URL returns 400 invalidRequest: urls: None of these is a public website URL.
  • With ignoreInvalidURLs: false, one bad address returns 400 invalidRequest: urls: Expected public website URLs. Not valid: ftp://example.com/file.txt.
  • Too many jobs returns 429 rateLimited. See Limits and errors.
  • Your balance is checked as pages are read, not when you submit. A batch that runs out of balance ends as failed with Your balance ran out. The pages already read are kept. Add funds, then start a new job for the pages left. Adding URLs with appendToId returns 402 insufficientCredits when your balance cannot cover them.
  • Batches explains what a batch is and what it costs.
  • Results reads the pages, the errors and the list of batches.
  • Limits and errors covers cancelling and jobs at once.
  • Scrape has every format and option.