> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl

> Read a whole section of a site in the background. Start from one URL, follow its links, and collect each page as Markdown. $1 per 1,000 pages.

Crawl starts at one page, follows its links, and reads every page it finds. It runs in the background. The call answers at once with an `id`, and you read the pages as they finish.

## Make a request

Get an API key in the [Quickstart](https://docs.stophy.dev/quickstart). Crawl needs one. This request crawls a docs section and stops after 3 pages.

```bash cURL theme={null}
curl -X POST https://api.stophy.dev/v1/crawl \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://bun.com/docs/runtime",
    "limit": 3
  }'
```

```typescript TypeScript theme={null}
const response = await fetch("https://api.stophy.dev/v1/crawl", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.STOPHY_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://bun.com/docs/runtime",
    limit: 3,
  }),
});
const { id } = await response.json();
console.log(id);
```

```python Python theme={null}
import os

import requests

response = requests.post(
    "https://api.stophy.dev/v1/crawl",
    headers={"Authorization": f"Bearer {os.environ['STOPHY_API_KEY']}"},
    json={
        "url": "https://bun.com/docs/runtime",
        "limit": 3,
    },
)
print(response.json()["id"])
```

## Read the response

The call answers at once with the crawl's `id` and the URL to read it from.

```json theme={null}
{
  "success": true,
  "id": "job_9FAn-0pRXhHk7olrXk0sVg",
  "url": "https://api.stophy.dev/v1/crawl/job_9FAn-0pRXhHk7olrXk0sVg",
  "costUsd": 0,
  "creditsUsed": 0,
  "requestId": "a02dc5f9-a3b8-4ce5-9368-a155b693f74b"
}
```

Read the crawl with `GET /v1/crawl/{id}` until `status` is `completed`. This real answer came 3 seconds later. The Markdown of each page is cut.

```json theme={null}
{
  "success": true,
  "data": [
    {
      "markdown": "# Bun Runtime\n\n> Execute JavaScript/TypeScript files, package.json scripts, and executable packages …",
      "metadata": {
        "title": "Bun Runtime",
        "sourceURL": "https://bun.com/docs/runtime",
        "url": "https://bun.com/docs/runtime",
        "statusCode": 200,
        "contentType": "text/markdown",
        "cacheState": "miss",
        "creditsUsed": 1,
        "costUsd": 0.001
      }
    },
    {
      "markdown": "# Archive\n\n> Create and extract tar archives with Bun's fast native implementation …",
      "metadata": { "title": "Archive", "url": "https://bun.com/docs/runtime/archive", "statusCode": 200, "costUsd": 0.001 }
    },
    {
      "markdown": "# Auto-install\n\n> Bun's automatic package installation feature for standalone script execution …",
      "metadata": { "title": "Auto-install", "url": "https://bun.com/docs/runtime/auto-install", "statusCode": 200, "costUsd": 0.001 }
    }
  ],
  "status": "completed",
  "total": 3,
  "completed": 3,
  "failed": 0,
  "creditsUsed": 2,
  "costUsd": 0.003,
  "createdAt": "2026-11-17T18:03:18.610Z",
  "completedAt": "2026-11-17T18:03:21.668Z",
  "duration": 3.058,
  "expiresAt": "2026-11-18T18:03:18.610Z",
  "errors": [],
  "requestId": "8acd8bf5-72b1-4798-99b6-c86690e64fbb"
}
```

While the crawl runs, `status` is `scraping`, `total` is the number of pages found so far, and `data` holds the pages that are done. See [Read a crawl](https://docs.stophy.dev/crawl/async.md) for polling, paging, cancelling and webhooks.

## Options

| Field | What it does | Guide |
| - | - | - |
| `url` | The page to start from. Required. | [Scope](https://docs.stophy.dev/crawl/scope.md) |
| `limit` | Stop after reading this many pages. 1 to 10,000, default 10,000. | [Scope](https://docs.stophy.dev/crawl/scope.md) |
| `includePaths`, `excludePaths`, `regexOnFullURL` | Read only the paths you match, or skip them. | [Scope](https://docs.stophy.dev/crawl/scope.md) |
| `maxDiscoveryDepth`, `crawlEntireDomain` | How far from the start page to go, and whether to leave its path. | [Scope](https://docs.stophy.dev/crawl/scope.md) |
| `allowSubdomains`, `allowExternalLinks` | Follow links to subdomains, or to other sites. | [Scope](https://docs.stophy.dev/crawl/scope.md) |
| `sitemap`, `ignoreQueryParameters`, `deduplicateSimilarURLs` | How pages are found and counted as one. | [Scope](https://docs.stophy.dev/crawl/scope.md) |
| `prompt` | Read pages about a topic first, and stop when new pages stop mentioning it. | [Scope](https://docs.stophy.dev/crawl/scope.md) |
| `scrapeOptions` | How to read each page: formats, filters, actions and more. | [Page content](https://docs.stophy.dev/crawl/page-content.md) |
| `webhook`, `delay`, `maxConcurrency`, `zeroDataRetention` | Be told as pages finish, pace the crawl, or keep nothing. | [Read a crawl](https://docs.stophy.dev/crawl/async.md) |

To turn a plain-language request into these fields, see [Describe a crawl](https://docs.stophy.dev/crawl/describe.md).

## Limits and errors

* A crawl reads at most 10,000 pages. A crawl and its pages are kept for 24 hours.
* A start page that cannot be read, or that the site's robots rules do not allow, ends the crawl with `status` `failed` and the reason in `error`. Later pages that those rules disallow are skipped.
* Crawl needs an API key. Without one, the call returns `401 apiKeyRequired`.
* A crawl can wait before it starts. While it waits, the answer has a `queuePosition`.

## Price

$1 per 1,000 pages. Starting a crawl is free, and each page is charged when it finishes. A page that answers 404 or 410, or cannot be read, costs nothing. The formats and options you send in `scrapeOptions` add what they add in [Scrape](https://docs.stophy.dev/scrape/overview#price). This crawl read 3 pages and cost $0.003. See [Pricing](https://docs.stophy.dev/account/pricing.md).

## Choose another API

* [Map](https://docs.stophy.dev/map/overview) lists the pages of a site without reading them.
* [Scrape](https://docs.stophy.dev/scrape/overview) reads one page.
* [Batches](https://docs.stophy.dev/batches/overview) read a list of URLs in the background.
* [Search](https://docs.stophy.dev/search/overview) finds pages from a query and can read each result.
* [Extract](https://docs.stophy.dev/extract/overview) pulls the fields you name from one or more pages.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.