scrapeOptions sets how every page of a crawl is read. It takes the same fields as Scrape, so a crawl can return Markdown, links, HTML, a screenshot, JSON and the rest. Without it, each page comes back as markdown.
Make a request
Get an API key in the Quickstart. This request crawls a site for 3 pages and returns each as Markdown plus its links, keeping only the main content and dropping the footer.Read the response
Each page indata has the formats you asked for and its own metadata, as a Scrape response has. The first page of this crawl was the start URL. This real answer is cut. The Markdown and the links are shortened, and only two of the three pages are shown.
metadata has the same fields as in Scrape. See Page metadata. The results endpoint also gives each page its depth, the number of links from the start page.
Options
scrapeOptions is an object. Every field below is a field of Scrape, and means the same here.
These Scrape fields are not allowed in
scrapeOptions: url, linkStyle, maxChars, offset and query. A crawl takes its URLs from url and the links it follows, and it returns whole pages.
Limits and errors
- Each page gets about 20 seconds plus your
waitFor. AtimeoutinsidescrapeOptionsis replaced by that value. - With a profile that saves changes, a crawl reads one page at a time, so it is slower. Send
"saveChanges": falseinprofileto read at full speed. - A page that fails does not stop the crawl, and it is listed in the crawl’s errors. See Read a crawl.
- Options that need an API key in Scrape, such as
actionsandwaitFor, work here, because Crawl always needs an API key.
Price
$1 per 1,000 pages, plus what the formats and options you choose add in Scrape: for example `summary`, `json` with a schema and `question` add $3 per 1,000 pages each, andcheckPromptInjection adds $2.50. See the Scrape price. A page that comes back with a 404 or 410 status, or cannot be read, costs nothing.
Related
- Scrape overview lists every format and option.
- Scope and limits chooses which pages are read.
- Read a crawl polls, pages and cancels a crawl.
- Crawl overview shows the whole flow.