timeout sets how long Scrape keeps trying to read a page. When a read fails, you get an error with a stable code and nothing is charged. This page lists every code Scrape returns and what to do about it.
Make a request
Get an API key in the Quickstart. This request reads a page that answers after 10 seconds, with a timeout of 3 seconds.Read the response
The read ran out of time, so the call failed with HTTP503. Nothing was charged.
code is stable, so branch on it. error says what went wrong and what to do. retryable says whether sending the same call again can help, and retryAfterSeconds, with the Retry-After header, says how long to wait. Keep requestId for support. See Errors.
Options
A page that needs a browser needs at least 5 seconds. A slow page can need 10 to 30 seconds the first time. Waits and actions use up the same time. Set your own client timeout longer than
timeout, with a few seconds to spare.
Error codes
A page that answers 404 or 410 comes back as a page, with that
metadata.statusCode and metadata.error set to The page answered with status 404. It costs nothing. The same holds for each page of a crawl.
Retry a failed read
Retry only whenretryable is true. Wait retryAfterSeconds, or back off 0.5, 1 and 2 seconds. After three tries, stop and log the requestId. A retry costs nothing extra, because a failed read is free. Production troubleshooting has a retry loop.
Many pages
Scrape reads one URL. A body withurls returns 400 invalidRequest with the message urls: Send url. To read many pages in one call, use POST /v1/batch/scrape. To read a list of URLs, send them to a batch. Each page of a batch or a crawl succeeds or fails on its own, so one bad page does not fail the rest. The failed pages are listed with their own code, message and retryable.
Warnings
Some problems do not fail the call. The page still comes back, with awarning that says what happened:
- A browser action failed, so the page is as it stood before it.
- The text was cut by
maxChars, and the warning gives theoffsetto read on from. onlyCleanContentwas skipped, orcheckPromptInjectioncould not scan the whole page.
Limits and errors
timeoutoutside 1,000 to 300,000 returns400 invalidRequest.- The most a single call can run is 5 minutes plus a few seconds.
- A page with more than 8 MB of text, or a file larger than 20 MB, returns
400 invalidRequest.
Related
- Production timeouts shows how to set client timeouts.
- Production troubleshooting covers retries and support.
- Errors lists every Stophy error code.
- Batches read many URLs in the background.