Skip to main content
scrapeOptions sets how every page of a crawl is read. It takes the same fields as Scrape, so a crawl can return Markdown, links, HTML, a screenshot, JSON and the rest. Without it, each page comes back as markdown.

Make a request

Get an API key in the Quickstart. This request crawls a site for 3 pages and returns each as Markdown plus its links, keeping only the main content and dropping the footer.

Read the response

Each page in data has the formats you asked for and its own metadata, as a Scrape response has. The first page of this crawl was the start URL. This real answer is cut. The Markdown and the links are shortened, and only two of the three pages are shown.
metadata has the same fields as in Scrape. See Page metadata. The results endpoint also gives each page its depth, the number of links from the start page.

Options

scrapeOptions is an object. Every field below is a field of Scrape, and means the same here. These Scrape fields are not allowed in scrapeOptions: url, linkStyle, maxChars, offset and query. A crawl takes its URLs from url and the links it follows, and it returns whole pages.

Limits and errors

  • Each page gets about 20 seconds plus your waitFor. A timeout inside scrapeOptions is replaced by that value.
  • With a profile that saves changes, a crawl reads one page at a time, so it is slower. Send "saveChanges": false in profile to read at full speed.
  • A page that fails does not stop the crawl, and it is listed in the crawl’s errors. See Read a crawl.
  • Options that need an API key in Scrape, such as actions and waitFor, work here, because Crawl always needs an API key.

Price

$1 per 1,000 pages, plus what the formats and options you choose add in Scrape: for example `summary`, `json` with a schema and `question` add $3 per 1,000 pages each, and checkPromptInjection adds $2.50. See the Scrape price. A page that comes back with a 404 or 410 status, or cannot be read, costs nothing.