scrapeOptions to a search to read each web and news result, or set highlights to get the passages that match your query. A result stays in the answer even when its page cannot be read.
Read each result
Get an API key in the Quickstart. Reading result pages needs an API key. This request reads two results asmarkdown.
Read the response
Each web row getsmarkdown and metadata, the same as a scrape of that page. metadata.costUsd is what reading that page cost. This real response is cut to one row. The Markdown was 38,988 characters long.
costUsd is $0.003 for two results: $0.001 for the search and $0.001 for each page. A format other than markdown, such as summary, html or links, appears as its own field on each row. See Scrape for every format. With scrapeOptions.formats, the description of a web row is filled from the page.
Get highlights
Sendhighlights as true to get the passages of each page that best match query. They replace the description, in Markdown, up to about 2,000 characters. A result whose page Stophy has already read gets passages. The others keep their usual description. Highlights add nothing to the price.
description to show. In the second, the … between passages marks where text was skipped.
highlights format with a query to scrapeOptions.formats. See Highlights.
Read each outcome
A page that cannot be read does not fail the search. The row keeps itsurl, title and description, has no markdown, and its metadata.error says why. A row with an error is not charged for the page. This real call asked for a site that Stophy does not read as a page.
metadata.error before you use markdown. The request was {"query": "site:linkedin.com bun runtime", "limit": 2, "scrapeOptions": {"formats": ["markdown"]}}, and the response is cut to one of its two rows. These are the other messages you can meet:
Options
These fields live insidescrapeOptions. It takes the options of Scrape, so every Scrape page applies here.
Limits and errors
- Only web and news results are read.
scrapeOptions.formatswithimagesinsourcesreturns400 invalidRequest. The error saysPage text is only added to web and news results. Drop images. - Without an API key,
scrapeOptionsreturns400 invalidRequestwithscrapeOptions: Reading result pages needs an API key. Get one at https://stophy.dev/signup, or leave scrapeOptions out. - Up to 10 pages are read at once. The search
timeoutis 60,000 ms by default when pages are read.
Price
$1 per 1,000 results whose page is read, on top of the search price. `summary` and `json` with a schema or a prompt add $3 per 1,000 more each when they return something. A result with ametadata.error is not charged for its page. See Pricing.
Related
- Scrape lists every format a result can return.
- Highlights returns passages as their own list.
- Timeouts and errors covers page read failures.
- Search overview lists every search field.