> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Page content

> Read the text of every search result, or only the passages that match your query. A result stays useful when its page cannot be read, and costs nothing for it.

Add `scrapeOptions` to a [search](https://docs.stophy.dev/search/overview) to read each web and news result, or set `highlights` to get the passages that match your query. A result stays in the answer even when its page cannot be read.

## Read each result

Get an API key in the [Quickstart](https://docs.stophy.dev/quickstart). Reading result pages needs an API key. This request reads two results as `markdown`.

```bash cURL theme={null}
curl -X POST https://api.stophy.dev/v1/search \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "bun sqlite",
    "limit": 2,
    "scrapeOptions": {
      "formats": ["markdown"],
      "onlyMainContent": true
    }
  }'
```

```typescript TypeScript theme={null}
const response = await fetch("https://api.stophy.dev/v1/search", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.STOPHY_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    query: "bun sqlite",
    limit: 2,
    scrapeOptions: {
      formats: ["markdown"],
      onlyMainContent: true,
    },
  }),
});
const { data } = await response.json();
for (const result of data.web) {
  console.log(result.url, result.markdown?.length ?? result.metadata.error);
}
```

```python Python theme={null}
import os

import requests

response = requests.post(
    "https://api.stophy.dev/v1/search",
    headers={"Authorization": f"Bearer {os.environ['STOPHY_API_KEY']}"},
    json={
        "query": "bun sqlite",
        "limit": 2,
        "scrapeOptions": {
            "formats": ["markdown"],
            "onlyMainContent": True,
        },
    },
)
for result in response.json()["data"]["web"]:
    print(result["url"], len(result.get("markdown", "")) or result["metadata"]["error"])
```

## Read the response

Each web row gets `markdown` and `metadata`, the same as a [scrape](https://docs.stophy.dev/scrape/page-metadata.md) of that page. `metadata.costUsd` is what reading that page cost. This real response is cut to one row. The Markdown was 38,988 characters long.

```json theme={null}
{
  "success": true,
  "data": {
    "web": [
      {
        "url": "https://bun.com/docs/runtime/sqlite",
        "title": "SQLite | Bun Docs",
        "description": "Bun natively implements a high-performance SQLite3 driver.",
        "position": 1,
        "markdown": "# SQLite\n\n> Bun natively implements a high-performance SQLite3 driver.\n\n…",
        "metadata": {
          "title": "SQLite",
          "sourceURL": "https://bun.com/docs/runtime/sqlite",
          "url": "https://bun.com/docs/runtime/sqlite",
          "statusCode": 200,
          "contentType": "text/markdown",
          "cacheState": "hit",
          "costUsd": 0.001,
          "scrapeId": "3291abae-3e8d-4b28-b3d4-1f064024090b"
        }
      }
    ]
  },
  "id": "3291abae-3e8d-4b28-b3d4-1f064024090b",
  "costUsd": 0.003,
  "requestId": "3291abae-3e8d-4b28-b3d4-1f064024090b"
}
```

`costUsd` is $0.003 for two results: $0.001 for the search and \$0.001 for each page. A format other than `markdown`, such as `summary`, `html` or `links`, appears as its own field on each row. See [Scrape](https://docs.stophy.dev/scrape/overview) for every format. With `scrapeOptions.formats`, the `description` of a web row is filled from the page.

## Get highlights

Send `highlights` as `true` to get the passages of each page that best match `query`. They replace the `description`, in Markdown, up to about 2,000 characters. A result whose page Stophy has already read gets passages. The others keep their usual description. Highlights add nothing to the price.

```bash cURL theme={null}
curl -X POST https://api.stophy.dev/v1/search \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "bun sqlite transactions",
    "limit": 3,
    "highlights": true
  }'
```

```typescript TypeScript theme={null}
const response = await fetch("https://api.stophy.dev/v1/search", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.STOPHY_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    query: "bun sqlite transactions",
    limit: 3,
    highlights: true,
  }),
});
const { data } = await response.json();
console.log(data.web.map((result) => result.description));
```

```python Python theme={null}
import os

import requests

response = requests.post(
    "https://api.stophy.dev/v1/search",
    headers={"Authorization": f"Bearer {os.environ['STOPHY_API_KEY']}"},
    json={
        "query": "bun sqlite transactions",
        "limit": 3,
        "highlights": True,
    },
)
print([result.get("description") for result in response.json()["data"]["web"]])
```

This real response is cut to two of the three rows. The first row has no `description` to show. In the second, the `…` between passages marks where text was skipped.

```json theme={null}
{
  "success": true,
  "data": {
    "web": [
      {
        "url": "https://www.youtube.com/watch?v=xE5SP2VYMMo",
        "title": "Bun JS SQLite Crash Course (2024) - YouTube",
        "position": 1
      },
      {
        "url": "https://bun.com/docs/runtime/sqlite",
        "title": "SQLite | Bun Docs",
        "description": "Credit to better-sqlite3 and its contributors for inspiring the API of bun:sqlite. Features include: Transactions Parameters (named & positional) Prepared statements …\n\n## Transactions\nCreate a transaction with the `db.transaction()` method:\n…\n## Transactions\n<Note> **Nested transactions** — You can call transaction functions from inside other transaction functions.",
        "position": 2
      }
    ]
  },
  "id": "9480aea1-f5dd-4990-b604-53f9fa7e396b",
  "costUsd": 0.001,
  "requestId": "9480aea1-f5dd-4990-b604-53f9fa7e396b"
}
```

To get the passages as their own list, with your own query and count, add a `highlights` format with a `query` to `scrapeOptions.formats`. See [Highlights](https://docs.stophy.dev/scrape/highlights.md).

## Read each outcome

A page that cannot be read does not fail the search. The row keeps its `url`, `title` and `description`, has no `markdown`, and its `metadata.error` says why. A row with an error is not charged for the page. This real call asked for a site that Stophy does not read as a page.

```json theme={null}
{
  "success": true,
  "data": {
    "web": [
      {
        "url": "https://www.linkedin.com/pulse/revolutionizing-javascript-runtimes-bun-breakthrough",
        "title": "Revolutionizing JavaScript Runtimes: The Bun Breakthrough",
        "description": "As mentioned earlier, Bun serves as an all-encompassing JavaScript runtime, doubling as a Node.js-compatible package manager.",
        "position": 1,
        "metadata": {
          "error": "Stophy does not read pages on linkedin.com. If a parser takes this page, send its URL with {\"type\":\"json\",\"parser\":\"<name>\"}: GET /v1/parsers lists them. Otherwise search the web with POST /v1/search to find what linkedin.com covers.",
          "scrapeId": "59072bce-14c9-4036-a60b-e29bcb9e3735"
        },
        "date": "2023-10-04T00:00:00.000Z"
      }
    ]
  },
  "id": "59072bce-14c9-4036-a60b-e29bcb9e3735",
  "costUsd": 0.001,
  "requestId": "59072bce-14c9-4036-a60b-e29bcb9e3735"
}
```

Check `metadata.error` before you use `markdown`. The request was `{"query": "site:linkedin.com bun runtime", "limit": 2, "scrapeOptions": {"formats": ["markdown"]}}`, and the response is cut to one of its two rows. These are the other messages you can meet:

| `metadata.error` | Meaning |
| - | - |
| `The page took too long to read. Read this address with scrape.` | The page did not answer inside its deadline. |
| `There was no time left to read this page. Read this address with scrape.` | The search `timeout` ran out before the page was reached. |
| `The page was not found or has no readable text.` | The page is gone or has no text to return. |
| `The page could not be read in this call. Read this address with scrape.` | Another failure. Read the address with [Scrape](https://docs.stophy.dev/scrape/overview) to see why. |

## Options

These fields live inside `scrapeOptions`. It takes the options of [Scrape](https://docs.stophy.dev/scrape/overview), so every Scrape page applies here.

| Field | Type | Default | Effect |
| - | - | - | - |
| `scrapeOptions.formats` | `array` | none | What to return for each result. With no `formats`, no page is read and nothing more is charged. Send up to 20, such as `markdown`, `summary`, `html`, `links` or `json`. |
| `scrapeOptions.onlyMainContent` | `boolean` | `true` | Keep only the main content of each page. See [Content filtering](https://docs.stophy.dev/scrape/content-filtering.md). |
| `scrapeOptions.maxAge` | `number` | 2 days | Accept a stored copy of each page up to this many milliseconds old, at most 30 days. `0` reads live. See [Freshness and caching](https://docs.stophy.dev/scrape/freshness-and-caching.md). |
| `scrapeOptions.timeout` | `number` | 30,000 ms | The most milliseconds one page may take, from 1,000 to 300,000. A page never gets more than what is left of the search `timeout`. Pages that run out of time get an error row. |
| `scrapeOptions.zeroDataRetention` | `boolean` | `false` | Keep nothing from the call. Adds \$2 per 1,000 pages. See [Zero data retention](https://docs.stophy.dev/production/zero-data-retention.md). |
| `highlights` | `boolean` | `false` | Fill each web `description` with the passages that best match `query`. |
| `limit` | `number` | 5 | With `scrapeOptions.formats`, how many results are read. Every web and news result returned is read. |
| `maxTextLength` | `number` | none | A query parameter. Cuts every text value in the response to this many characters. A cut value ends with `…`. |

## Limits and errors

* Only web and news results are read. `scrapeOptions.formats` with `images` in `sources` returns `400 invalidRequest`. The error says `Page text is only added to web and news results. Drop images.`
* Without an API key, `scrapeOptions` returns `400 invalidRequest` with `scrapeOptions: Reading result pages needs an API key. Get one at https://stophy.dev/signup, or leave scrapeOptions out.`
* Up to 10 pages are read at once. The search `timeout` is 60,000 ms by default when pages are read.

## Price

$1 per 1,000 results whose page is read, on top of the search price. `summary` and `json` with a schema or a prompt add $3 per 1,000 more each when they return something. A result with a `metadata.error` is not charged for its page. See [Pricing](https://docs.stophy.dev/account/pricing.md).

## Related

* [Scrape](https://docs.stophy.dev/scrape/overview) lists every format a result can return.
* [Highlights](https://docs.stophy.dev/scrape/highlights.md) returns passages as their own list.
* [Timeouts and errors](https://docs.stophy.dev/scrape/timeouts-and-errors.md) covers page read failures.
* [Search overview](https://docs.stophy.dev/search/overview) lists every search field.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.