> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Page content

> Choose what each crawled page returns with scrapeOptions: formats, main content, selectors, waits, actions and more, the same options as Scrape.

`scrapeOptions` sets how every page of a crawl is read. It takes the same fields as [Scrape](https://docs.stophy.dev/scrape/overview), so a crawl can return Markdown, links, HTML, a screenshot, JSON and the rest. Without it, each page comes back as `markdown`.

## Make a request

Get an API key in the [Quickstart](https://docs.stophy.dev/quickstart). This request crawls a site for 3 pages and returns each as Markdown plus its links, keeping only the main content and dropping the footer.

```bash cURL theme={null}
curl -X POST https://api.stophy.dev/v1/crawl \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://quotes.toscrape.com",
    "limit": 3,
    "scrapeOptions": {
      "formats": ["markdown", "links"],
      "onlyMainContent": true,
      "excludeTags": ["footer"],
      "waitFor": 500
    }
  }'
```

```typescript TypeScript theme={null}
const response = await fetch("https://api.stophy.dev/v1/crawl", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.STOPHY_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://quotes.toscrape.com",
    limit: 3,
    scrapeOptions: {
      formats: ["markdown", "links"],
      onlyMainContent: true,
      excludeTags: ["footer"],
      waitFor: 500,
    },
  }),
});
const { id } = await response.json();
console.log(id);
```

```python Python theme={null}
import os

import requests

response = requests.post(
    "https://api.stophy.dev/v1/crawl",
    headers={"Authorization": f"Bearer {os.environ['STOPHY_API_KEY']}"},
    json={
        "url": "https://quotes.toscrape.com",
        "limit": 3,
        "scrapeOptions": {
            "formats": ["markdown", "links"],
            "onlyMainContent": True,
            "excludeTags": ["footer"],
            "waitFor": 500,
        },
    },
)
print(response.json()["id"])
```

## Read the response

Each page in `data` has the formats you asked for and its own `metadata`, as a [Scrape](https://docs.stophy.dev/scrape/overview) response has. The first page of this crawl was the start URL. This real answer is cut. The Markdown and the links are shortened, and only two of the three pages are shown.

```json theme={null}
{
  "success": true,
  "data": [
    {
      "markdown": "# [Quotes to Scrape](https://quotes.toscrape.com/)\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein …",
      "links": [
        "https://quotes.toscrape.com/",
        "https://quotes.toscrape.com/login",
        "https://quotes.toscrape.com/author/Albert-Einstein"
      ],
      "metadata": {
        "title": "Quotes to Scrape",
        "language": "en",
        "sourceURL": "https://quotes.toscrape.com/",
        "url": "https://quotes.toscrape.com/",
        "statusCode": 200,
        "contentType": "text/html",
        "cacheState": "miss",
        "costUsd": 0.001
      }
    },
    {
      "markdown": "# [Quotes to Scrape](https://quotes.toscrape.com/)\n\nIn 1879, Albert Einstein was born in U…",
      "links": ["https://quotes.toscrape.com/", "https://quotes.toscrape.com/login"],
      "metadata": {
        "sourceURL": "https://quotes.toscrape.com/author/Albert-Einstein",
        "url": "https://quotes.toscrape.com/author/Albert-Einstein/",
        "statusCode": 200,
        "costUsd": 0.001
      }
    }
  ],
  "status": "completed",
  "total": 3,
  "completed": 3,
  "failed": 0,
  "costUsd": 0.003
}
```

`metadata` has the same fields as in Scrape. See [Page metadata](https://docs.stophy.dev/scrape/page-metadata.md). The results endpoint also gives each page its `depth`, the number of links from the start page.

## Options

`scrapeOptions` is an object. Every field below is a field of Scrape, and means the same here.

| Field | Type | Default | Effect |
| - | - | - | - |
| `formats` | `array` | `["markdown"]` | What each page returns. See the formats in the [Scrape overview](https://docs.stophy.dev/scrape/overview#formats). |
| `onlyMainContent` | `boolean` | `true` | Keep the main content, without menus, headers and footers. |
| `onlyCleanContent` | `boolean` | `false` | A model removes banners and prompts that are left. See [Content filtering](https://docs.stophy.dev/scrape/content-filtering.md). |
| `includeTags`, `excludeTags` | `array of string` | none | Keep or remove parts of each page by CSS selector. |
| `removeBase64Images` | `boolean` | `true` | Drop images that carry their data in the URL. |
| `waitFor`, `mobile`, `fastMode`, `blockAds` | varies | varies | Waiting and rendering. See [Waiting and rendering](https://docs.stophy.dev/scrape/waiting-and-rendering.md). |
| `actions` | `array of object` | none | Steps to run on each page before it is read. See [Browser actions](https://docs.stophy.dev/scrape/browser-actions.md). |
| `location`, `headers`, `skipTlsVerification`, `proxy` | varies | varies | Where to read from and what to send. See [Location and headers](https://docs.stophy.dev/scrape/location-and-headers.md). |
| `maxAge`, `minAge`, `storeInCache`, `lockdown` | varies | varies | How old a stored copy may be. See [Freshness and caching](https://docs.stophy.dev/scrape/freshness-and-caching.md). |
| `checkPromptInjection`, `redactPII` | varies | `false` | Scan for prompt injection, or hide personal data. See [Safety](https://docs.stophy.dev/scrape/safety.md). |
| `parsers` | `array` | none | PDF options for document pages. See [PDFs and documents](https://docs.stophy.dev/scrape/pdfs-and-documents.md). |
| `profile` | `object` | none | Read with the cookies a browser profile saved. |

These Scrape fields are not allowed in `scrapeOptions`: `url`, `linkStyle`, `maxChars`, `offset` and `query`. A crawl takes its URLs from `url` and the links it follows, and it returns whole pages.

## Limits and errors

* Each page gets about 20 seconds plus your `waitFor`. A `timeout` inside `scrapeOptions` is replaced by that value.
* With a profile that saves changes, a crawl reads one page at a time, so it is slower. Send `"saveChanges": false` in `profile` to read at full speed.
* A page that fails does not stop the crawl, and it is listed in the crawl's errors. See [Read a crawl](https://docs.stophy.dev/crawl/async.md).
* Options that need an API key in Scrape, such as `actions` and `waitFor`, work here, because Crawl always needs an API key.

## Price

$1 per 1,000 pages, plus what the formats and options you choose add in Scrape: for example `summary`, `json` with a schema and `question` add $3 per 1,000 pages each, and `checkPromptInjection` adds \$2.50. See the [Scrape price](https://docs.stophy.dev/scrape/overview#price). A page that comes back with a 404 or 410 status, or cannot be read, costs nothing.

## Related

* [Scrape overview](https://docs.stophy.dev/scrape/overview) lists every format and option.
* [Scope and limits](https://docs.stophy.dev/crawl/scope.md) chooses which pages are read.
* [Read a crawl](https://docs.stophy.dev/crawl/async.md) polls, pages and cancels a crawl.
* [Crawl overview](https://docs.stophy.dev/crawl/overview) shows the whole flow.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.