url and follows the links that are in scope. These fields decide what is in scope and when the crawl stops. By default a crawl stays on the start page’s host and under the start page’s path.
Make a request
Get an API key in the Quickstart. This request reads only pages under/docs/runtime/ whose name starts with s, skips the sql page, goes one link away from the start, and stops after 3 pages.
Read the response
The call answers with anid. Read it with GET /v1/crawl/{id} as Read a crawl shows. When this crawl finished, its pages were these three. The response is cut to the fields that matter.
https://bun.com/docs/runtime, is not in the list, because it has to match includePaths like any other page.
Options
How paths match
includePathsandexcludePathsare JavaScript regular expressions, up to 200 characters each. They are not anchored, soblogmatches/company/blog/post. Start with^to anchor one.- A pattern is tested against the path, such as
/docs/runtime/s3. WithregexOnFullURL, it is tested against the whole URL. - A path that matches
excludePathsis skipped even when it also matchesincludePaths. - A group cannot be repeated with
*,+or{}, such as(a|b)+. The call returns400 invalidRequestwith the messageA group cannot be repeated with *, + or {}. - Pages are read in order of fit: first those a
promptpoints to, then those closest to the start page.
Limits and errors
- A crawl reads at most
limitpages. It can read fewer: a site may expose fewer pages in scope. - A crawl and its pages are kept for 24 hours.
- Pages that the site’s robots rules disallow are skipped. They are not counted as failures.
- A pattern that is not a valid regular expression returns
400 invalidRequestwith the messageExpected a regular expression such as blog/.*. - A page after the start page that cannot be read does not stop the crawl. It is listed in the crawl’s errors.
Price
$1 per 1,000 pages read. A page counts towardlimit when it is read. Scope fields add nothing. See Pricing.
Related
- Crawl overview shows the request and the answer.
- Page content chooses how each page is read.
- Describe a crawl turns a plain-language request into these fields.
- Map lists a site’s pages before you decide what to crawl.