Skip to main content
timeout sets how long Scrape keeps trying to read a page. When a read fails, you get an error with a stable code and nothing is charged. This page lists every code Scrape returns and what to do about it.

Make a request

Get an API key in the Quickstart. This request reads a page that answers after 10 seconds, with a timeout of 3 seconds.

Read the response

The read ran out of time, so the call failed with HTTP 503. Nothing was charged.
Every error has this flat shape. code is stable, so branch on it. error says what went wrong and what to do. retryable says whether sending the same call again can help, and retryAfterSeconds, with the Retry-After header, says how long to wait. Keep requestId for support. See Errors.

Options

A page that needs a browser needs at least 5 seconds. A slow page can need 10 to 30 seconds the first time. Waits and actions use up the same time. Set your own client timeout longer than timeout, with a few seconds to spare.

Error codes

A page that answers 404 or 410 comes back as a page, with that metadata.statusCode and metadata.error set to The page answered with status 404. It costs nothing. The same holds for each page of a crawl.

Retry a failed read

Retry only when retryable is true. Wait retryAfterSeconds, or back off 0.5, 1 and 2 seconds. After three tries, stop and log the requestId. A retry costs nothing extra, because a failed read is free. Production troubleshooting has a retry loop.

Many pages

Scrape reads one URL. A body with urls returns 400 invalidRequest with the message urls: Send url. To read many pages in one call, use POST /v1/batch/scrape. To read a list of URLs, send them to a batch. Each page of a batch or a crawl succeeds or fails on its own, so one bad page does not fail the rest. The failed pages are listed with their own code, message and retryable.

Warnings

Some problems do not fail the call. The page still comes back, with a warning that says what happened:
  • A browser action failed, so the page is as it stood before it.
  • The text was cut by maxChars, and the warning gives the offset to read on from.
  • onlyCleanContent was skipped, or checkPromptInjection could not scan the whole page.

Limits and errors

  • timeout outside 1,000 to 300,000 returns 400 invalidRequest.
  • The most a single call can run is 5 minutes plus a few seconds.
  • A page with more than 8 MB of text, or a file larger than 20 MB, returns 400 invalidRequest.