Skip to main content
POST /v1/crawl/params-preview turns what you want into crawl options. Send a site and a sentence, read the options it picks, and pass them to Crawl yourself. Nothing is crawled and no page is read.

Make a request

Get an API key in the Quickstart. This request asks for the blog posts of a site, two links deep.

Read the response

data holds the crawl options the request maps to. A field the sentence does not mention is left out.
Check the options, change what you want, and send them to POST /v1/crawl:

Options

The answer can hold these fields, each described in Scope and limits: url, includePaths, excludePaths, maxDiscoveryDepth, crawlEntireDomain, allowSubdomains, allowExternalLinks, sitemap, ignoreQueryParameters and limit. It never sets scrapeOptions, so choose the page formats and filters yourself.

Limits and errors

  • The options are a suggestion made from your sentence. Read them before you start a crawl.
  • Each pattern in includePaths and excludePaths is checked as a valid regular expression, and a pattern that is not valid is dropped. At most 20 patterns of each are returned.
  • maxDiscoveryDepth is at most 10, and limit is at most 10,000.
  • A request that finds nothing to set returns only url and costs nothing.
  • The call needs an API key. Without one, it returns 401 apiKeyRequired.

Price

$3 per 1,000 calls. A call that finds nothing to set is free. See Pricing.