Skip to main content
A crawl starts at url and follows the links that are in scope. These fields decide what is in scope and when the crawl stops. By default a crawl stays on the start page’s host and under the start page’s path.

Make a request

Get an API key in the Quickstart. This request reads only pages under /docs/runtime/ whose name starts with s, skips the sql page, goes one link away from the start, and stops after 3 pages.

Read the response

The call answers with an id. Read it with GET /v1/crawl/{id} as Read a crawl shows. When this crawl finished, its pages were these three. The response is cut to the fields that matter.
The start page, https://bun.com/docs/runtime, is not in the list, because it has to match includePaths like any other page.

Options

How paths match

  • includePaths and excludePaths are JavaScript regular expressions, up to 200 characters each. They are not anchored, so blog matches /company/blog/post. Start with ^ to anchor one.
  • A pattern is tested against the path, such as /docs/runtime/s3. With regexOnFullURL, it is tested against the whole URL.
  • A path that matches excludePaths is skipped even when it also matches includePaths.
  • A group cannot be repeated with *, + or {}, such as (a|b)+. The call returns 400 invalidRequest with the message A group cannot be repeated with *, + or {}.
  • Pages are read in order of fit: first those a prompt points to, then those closest to the start page.

Limits and errors

  • A crawl reads at most limit pages. It can read fewer: a site may expose fewer pages in scope.
  • A crawl and its pages are kept for 24 hours.
  • Pages that the site’s robots rules disallow are skipped. They are not counted as failures.
  • A pattern that is not a valid regular expression returns 400 invalidRequest with the message Expected a regular expression such as blog/.*.
  • A page after the start page that cannot be read does not stop the crawl. It is listed in the crawl’s errors.

Price

$1 per 1,000 pages read. A page counts toward limit when it is read. Scope fields add nothing. See Pricing.