POST /v1/crawl/params-preview turns what you want into crawl options. Send a site and a sentence, read the options it picks, and pass them to Crawl yourself. Nothing is crawled and no page is read.
Make a request
Get an API key in the Quickstart. This request asks for the blog posts of a site, two links deep.Read the response
data holds the crawl options the request maps to. A field the sentence does not mention is left out.
POST /v1/crawl:
Options
The answer can hold these fields, each described in Scope and limits:
url, includePaths, excludePaths, maxDiscoveryDepth, crawlEntireDomain, allowSubdomains, allowExternalLinks, sitemap, ignoreQueryParameters and limit. It never sets scrapeOptions, so choose the page formats and filters yourself.
Limits and errors
- The options are a suggestion made from your sentence. Read them before you start a crawl.
- Each pattern in
includePathsandexcludePathsis checked as a valid regular expression, and a pattern that is not valid is dropped. At most 20 patterns of each are returned. maxDiscoveryDepthis at most 10, andlimitis at most 10,000.- A request that finds nothing to set returns only
urland costs nothing. - The call needs an API key. Without one, it returns
401 apiKeyRequired.
Price
$3 per 1,000 calls. A call that finds nothing to set is free. See Pricing.Related
- Crawl overview starts a crawl with these options.
- Scope and limits describes every field in the answer.
- Page content chooses what each crawled page returns.
- Map lists a site’s pages before you crawl.