id, and you read the pages as they finish.
Make a request
Get an API key in the Quickstart. Crawl needs one. This request crawls a docs section and stops after 3 pages.Read the response
The call answers at once with the crawl’sid and the URL to read it from.
GET /v1/crawl/{id} until status is completed. This real answer came 3 seconds later. The Markdown of each page is cut.
status is scraping, total is the number of pages found so far, and data holds the pages that are done. See Read a crawl for polling, paging, cancelling and webhooks.
Options
To turn a plain-language request into these fields, see Describe a crawl.
Limits and errors
- A crawl reads at most 10,000 pages. A crawl and its pages are kept for 24 hours.
- A start page that cannot be read, or that the site’s robots rules do not allow, ends the crawl with
statusfailedand the reason inerror. Later pages that those rules disallow are skipped. - Crawl needs an API key. Without one, the call returns
401 apiKeyRequired. - A crawl can wait before it starts. While it waits, the answer has a
queuePosition.