Crawl
Crawl a website in the background. Send one request and get JSON back. Examples in cURL, TypeScript and Python. A failed call costs nothing.
Price
1 credit a page, however it is reached. json with a schema or a prompt, summary and highlights add 4 credits each when they return something. CSS rules and attributes add nothing. A PDF adds 1 credit for each page read. Each perform action adds 2 credits. An answer from a stored copy within maxAge is free. A page that comes back with a 404 or 410 status is charged like any page. A page that cannot be read costs nothing. Each page is charged when it finishes. Starting the job is free. See Pricing for what credits cost.Limits
An account runs up to 5 crawls and batch scrapes at once. A sixth start answers 400invalidRequest: jobs: you already have 5 jobs running. Wait for one of them to finish, then try again. Wait until one is completed, then start it again.
Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Query Parameters
Cut every text value in the response to this many characters. A value that was cut ends with …, and a page whose text was cut has isTruncated set to true. A screenshot is never cut. Leave it out to get the full text.
x >= 1Body
The page to start from, such as https://bun.com/docs.
What you are looking for, such as sqlite performance. Pages about it are read first, and the crawl stops once new pages stop mentioning it.
1 - 1000Read only pages whose path matches one of these regular expressions, such as blog/.*.
1001 - 200Skip pages whose path matches one of these regular expressions, such as tag/.*.
1001 - 200How many links away from the start page to go, up to 10.
0 <= x <= 10include also uses the site's sitemaps. only reads just the pages the sitemaps name. skip ignores them.
skip, include, only Treat addresses that differ only in their ?query part as one page.
Match includePaths and excludePaths against the whole address, not only the path.
Stop after reading this many pages in total, across every call.
1 <= x <= 10000Read the whole site, not only pages under the start page's path.
Also read pages on subdomains, such as docs.bun.com for bun.com.
How to read each page, the same options and formats as scrape. Without formats, each page comes back as markdown.
Send true to keep nothing from this crawl: no stored copy is read or made, page results are held for one hour at most, and nothing is logged. Costs 1 credit more per page.
Where to be told about the job: { url } gets a JSON POST when the job starts, for each finished page, and when the job ends. If your address answers slowly, a page event can carry only the page url. Read that page with the status call.