Batch scrape
Batch scrape
Scrape many pages in the background. Send one request and get JSON back. Examples in cURL, TypeScript and Python. A failed call costs nothing.
POST
Best for a list of addresses you already have. It answers at once with an id; ask batch.scrape.status for the pages, or send webhook to be told as each one finishes.
Price
1 credit a page, however it is reached. json with a schema or a prompt, summary and highlights add 4 credits each when they return something. CSS rules and attributes add nothing. A PDF adds 1 credit for each page read. Each perform action adds 2 credits. An answer from a stored copy within maxAge is free. A page that comes back with a 404 or 410 status is charged like any page. A page that cannot be read costs nothing. Each page is charged when it finishes. Starting the job is free. See Pricing for what credits cost.Limits
An account runs up to 5 crawls and batch scrapes at once. A sixth start answers 400invalidRequest: jobs: you already have 5 jobs running. Wait for one of them to finish, then try again. Wait until one is completed, then start it again.
Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Query Parameters
Cut every text value in the response to this many characters. A value that was cut ends with …, and a page whose text was cut has isTruncated set to true. A screenshot is never cut. Leave it out to get the full text.
Required range:
x >= 1Body
application/json
The pages or PDFs to read. An address sent twice is read once.
Minimum array length:
1Maximum string length:
2048Where to be told about the job: { url } gets a JSON POST when the job starts, for each finished page, and when the job ends. If your address answers slowly, a page event can carry only the page url. Read that page with the status call.
Skip addresses that are not public website URLs and list them in invalidURLs. Send false to refuse the whole batch instead.
What to return: markdown (clean page text), summary (a short summary of the page), html (the cleaned page), rawHtml (the page as sent), links (every link on the page), images (every image address), screenshot (a link to a JPEG of the visible page, kept for 7 days), changeTracking (what changed in the markdown since your key last read this page with the same options, kept 30 days) network (the JSON the page loaded while it rendered, up to 30 responses) and branding (the page's colors, fonts, typography, spacing, button and input styles, logo and brand images, read from how the page looks, with a field left out when the page does not show it). Send { "type": "screenshot", "fullPage": true } for the whole page. Send { "type": "highlights", "query": "refund policy" } for the passages that best match a query. Send { "type": "json", "schema": { "type": "object", "properties": { "price": { "type": "number" } } } } or { "type": "json", "prompt": "the price and the stock" } for fields filled from the page, each checked against it. Send { "type": "json", "parser": "amazon-product" } to get a site's own data as json instead of reading the page; GET /v1/parsers lists the parsers. Send { "type": "json", "rules": { "title": "h1", "links": { "selector": "a", "type": "list", "output": "@href" } } } to read the page with CSS rules and no model, at the page's own price. Send { "type": "attributes", "selectors": [{ "selector": "a", "attribute": "href" }] } for attribute values, returned in attributes. Without a parser or rules, json needs a schema or a prompt and is filled from the page. json with a schema or a prompt, summary and highlights cost 4 credits more each when they return something. json with rules or a parser, and attributes, cost nothing more. A screenshot, network or branding read costs 1 credit, the same as any page. A PDF comes back as markdown only, and costs 1 credit more for each page read.
Required array length:
1 - 20 elementsAvailable options:
markdown, summary, html, rawHtml, links, images, network, branding, screenshot, changeTracking Keep only the page's main content, without menus, headers, footers and ads.
Keep only the parts of the page that match these CSS selectors, such as article or .price. When set, onlyMainContent is not applied.
Minimum array length:
1Minimum string length:
1Remove the parts of the page that match these CSS selectors, such as .comments.
Minimum array length:
1Minimum string length:
1Accept a stored answer up to this many milliseconds old, at most 30 days. Without it, 2 days, or less for a site whose pages change often, and a copy up to twice that age (at most 7 days) comes back at once as stale while a fresh read runs. 0 always reads the page live. metadata.cacheState says which one you got.
Required range:
0 <= x <= 9007199254740991Answer only from a stored copy that is at least this many milliseconds old, and never read the page live. 0 accepts any stored copy. A page with no such copy is notFound.
Required range:
0 <= x <= 9007199254740991Extra milliseconds to wait before reading, up to 60000, for pages that load their content late. To wait for an element, add a wait action with a selector. A read that waits costs 1 credit, the same as any page.
Required range:
0 <= x <= 60000How long to keep trying, in milliseconds, between 10 seconds and 2 minutes. Pages behind a bot check can need 10 to 30 seconds the first time.
Required range:
1000 <= x <= 300000Only ["pdf"] is supported: PDFs are read as markdown.
Minimum array length:
1Allowed value:
"pdf"Up to 50 steps to run on the page, in order, before reading it: { "type": "wait", "milliseconds": 500 } (up to 30 seconds), { "type": "wait", "selector": ".results" }, { "type": "click", "selector": "button.more" }, { "type": "write", "text": "tides" }, { "type": "press", "key": "Enter" }, { "type": "scroll", "direction": "down" } (also up, left or right; amount is pixels, "viewport" or "max"; maxScrolls 1 to 50; selector scrolls an inner container; "max" keeps scrolling until the page stops growing, for feeds that load as you go), { "type": "screenshot", "fullPage": true } (links come back in actions.screenshots), { "type": "scrape" } (the page at that point, in actions.scrapes), { "type": "executeJavascript", "script": "document.title" } (results in actions.javascriptReturns), { "type": "pdf", "format": "A4" } (links in actions.pdfs) or { "type": "perform", "action": "Click the Load more button" } (a step in plain words, up to 5 browser steps on this page, never leaving the site; at most 5 per call). A read with actions costs 1 credit, the same as any page, and each perform action costs 2 credits more. If an action fails, you still get the page as it stood, with a warning that names the action and says why.
Required array length:
1 - 50 elements- Option 1
- Option 2
- Option 3
- Option 4
- Option 5
- Option 6
- Option 7
- Option 8
- Option 9
- Option 10
- Option 11
Where to read the page from. The default is the United States.
Allowed value:
"auto"Send false to neither read nor keep a stored copy of this page.
Read the page with the cookies and local storage that earlier calls saved under this name, kept for your account for 30 days after the last use. A call with a profile always loads the page in a browser, costs 1 credit, the same as any page, and is never answered from a stored copy. Only one call at a time can save to a profile.
Send true to keep nothing from this call: no stored copy is read or made, and nothing from it is logged. Costs 1 credit more per page. Not with a screenshot.
How links appear in the markdown. inline keeps text [blocked]. numbered writes Pricing⟨3⟩ and lists each address once at the end. none keeps only the link text, which makes link-heavy pages 10 to 70% shorter; ask for the links format to get the addresses.
Available options:
inline, numbered, none Cut the markdown at this many characters; isTruncated says when that happened.
Required range:
500 <= x <= 1000000