Skip to main content
Crawl starts at one page, follows its links, and reads every page it finds. It runs in the background. The call answers at once with an id, and you read the pages as they finish.

Make a request

Get an API key in the Quickstart. Crawl needs one. This request crawls a docs section and stops after 3 pages.

Read the response

The call answers at once with the crawl’s id and the URL to read it from.
Read the crawl with GET /v1/crawl/{id} until status is completed. This real answer came 3 seconds later. The Markdown of each page is cut.
While the crawl runs, status is scraping, total is the number of pages found so far, and data holds the pages that are done. See Read a crawl for polling, paging, cancelling and webhooks.

Options

To turn a plain-language request into these fields, see Describe a crawl.

Limits and errors

  • A crawl reads at most 10,000 pages. A crawl and its pages are kept for 24 hours.
  • A start page that cannot be read, or that the site’s robots rules do not allow, ends the crawl with status failed and the reason in error. Later pages that those rules disallow are skipped.
  • Crawl needs an API key. Without one, the call returns 401 apiKeyRequired.
  • A crawl can wait before it starts. While it waits, the answer has a queuePosition.

Price

$1 per 1,000 pages. Starting a crawl is free, and each page is charged when it finishes. A page that answers 404 or 410, or cannot be read, costs nothing. The formats and options you send in `scrapeOptions` add what they add in [Scrape](/scrape/overview#price). This crawl read 3 pages and cost $0.003. See Pricing.

Choose another API

  • Map lists the pages of a site without reading them.
  • Scrape reads one page.
  • Batches read a list of URLs in the background.
  • Search finds pages from a query and can read each result.
  • Extract pulls the fields you name from one or more pages.