Skip to main content
POST
Best for reading a whole section of a site without holding a call open. It answers at once with an id; ask crawl.status for the pages, or send webhook to be told as each one finishes.

Price

1 credit a page, however it is reached. json with a schema or a prompt, summary and highlights add 4 credits each when they return something. CSS rules and attributes add nothing. A PDF adds 1 credit for each page read. Each perform action adds 2 credits. An answer from a stored copy within maxAge is free. A page that comes back with a 404 or 410 status is charged like any page. A page that cannot be read costs nothing. Each page is charged when it finishes. Starting the job is free. See Pricing for what credits cost.

Limits

An account runs up to 5 crawls and batch scrapes at once. A sixth start answers 400 invalidRequest: jobs: you already have 5 jobs running. Wait for one of them to finish, then try again. Wait until one is completed, then start it again.

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Query Parameters

maxTextLength
integer

Cut every text value in the response to this many characters. A value that was cut ends with …, and a page whose text was cut has isTruncated set to true. A screenshot is never cut. Leave it out to get the full text.

Required range: x >= 1

Body

application/json
url
string
required

The page to start from, such as https://bun.com/docs.

prompt
string

What you are looking for, such as sqlite performance. Pages about it are read first, and the crawl stops once new pages stop mentioning it.

Required string length: 1 - 1000
includePaths
string[]

Read only pages whose path matches one of these regular expressions, such as blog/.*.

Maximum array length: 100
Required string length: 1 - 200
excludePaths
string[]

Skip pages whose path matches one of these regular expressions, such as tag/.*.

Maximum array length: 100
Required string length: 1 - 200
maxDiscoveryDepth
integer
default:3

How many links away from the start page to go, up to 10.

Required range: 0 <= x <= 10
sitemap
enum<string>
default:include

include also uses the site's sitemaps. only reads just the pages the sitemaps name. skip ignores them.

Available options:
skip,
include,
only
ignoreQueryParameters
boolean
default:false

Treat addresses that differ only in their ?query part as one page.

regexOnFullURL
boolean
default:false

Match includePaths and excludePaths against the whole address, not only the path.

limit
integer
default:10000

Stop after reading this many pages in total, across every call.

Required range: 1 <= x <= 10000
crawlEntireDomain
boolean
default:false

Read the whole site, not only pages under the start page's path.

allowSubdomains
boolean
default:false

Also read pages on subdomains, such as docs.bun.com for bun.com.

ignoreRobotsTxt
boolean
robotsUserAgent
any
delay
any
maxConcurrency
any
scrapeOptions
object

How to read each page, the same options and formats as scrape. Without formats, each page comes back as markdown.

zeroDataRetention
boolean
default:false

Send true to keep nothing from this crawl: no stored copy is read or made, page results are held for one hour at most, and nothing is logged. Costs 1 credit more per page.

webhook
object

Where to be told about the job: { url } gets a JSON POST when the job starts, for each finished page, and when the job ends. If your address answers slowly, a page event can carry only the page url. Read that page with the status call.

Response

The data, and the credits this call used.

success
boolean
required
id
string
url
string