> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Scope and limits

> Bound a crawl by page count, path, depth, subdomains and sitemap, and read pages about a topic first. Start from one URL and stay where you mean to.

A crawl starts at `url` and follows the links that are in scope. These fields decide what is in scope and when the crawl stops. By default a crawl stays on the start page's host and under the start page's path.

## Make a request

Get an API key in the [Quickstart](https://docs.stophy.dev/quickstart). This request reads only pages under `/docs/runtime/` whose name starts with `s`, skips the `sql` page, goes one link away from the start, and stops after 3 pages.

```bash cURL theme={null}
curl -X POST https://api.stophy.dev/v1/crawl \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://bun.com/docs/runtime",
    "includePaths": ["/docs/runtime/s.*"],
    "excludePaths": ["/docs/runtime/sql"],
    "maxDiscoveryDepth": 1,
    "limit": 3
  }'
```

```typescript TypeScript theme={null}
const response = await fetch("https://api.stophy.dev/v1/crawl", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.STOPHY_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://bun.com/docs/runtime",
    includePaths: ["/docs/runtime/s.*"],
    excludePaths: ["/docs/runtime/sql"],
    maxDiscoveryDepth: 1,
    limit: 3,
  }),
});
const { id } = await response.json();
console.log(id);
```

```python Python theme={null}
import os

import requests

response = requests.post(
    "https://api.stophy.dev/v1/crawl",
    headers={"Authorization": f"Bearer {os.environ['STOPHY_API_KEY']}"},
    json={
        "url": "https://bun.com/docs/runtime",
        "includePaths": ["/docs/runtime/s.*"],
        "excludePaths": ["/docs/runtime/sql"],
        "maxDiscoveryDepth": 1,
        "limit": 3,
    },
)
print(response.json()["id"])
```

## Read the response

The call answers with an `id`. Read it with `GET /v1/crawl/{id}` as [Read a crawl](https://docs.stophy.dev/crawl/async.md) shows. When this crawl finished, its pages were these three. The response is cut to the fields that matter.

```json theme={null}
{
  "status": "completed",
  "total": 3,
  "completed": 3,
  "failed": 0,
  "costUsd": 0.003,
  "data": [
    { "metadata": { "url": "https://bun.com/docs/runtime/s3" } },
    { "metadata": { "url": "https://bun.com/docs/runtime/secrets" } },
    { "metadata": { "url": "https://bun.com/docs/runtime/semver" } }
  ]
}
```

The start page, `https://bun.com/docs/runtime`, is not in the list, because it has to match `includePaths` like any other page.

## Options

| Field | Type | Default | Effect |
| - | - | - | - |
| `url` | `string` | none | The page to start from. Required. |
| `limit` | `number` | `10000` | Stop after reading this many pages in total, 1 to 10,000. |
| `includePaths` | `array of string` | `[]` | Read only pages whose path matches one of these regular expressions, such as `blog/.*`. Up to 100 patterns. |
| `excludePaths` | `array of string` | `[]` | Skip pages whose path matches one of these regular expressions, such as `tag/.*`. Up to 100 patterns. |
| `regexOnFullURL` | `boolean` | `false` | Match `includePaths` and `excludePaths` against the whole URL, not only the path. |
| `maxDiscoveryDepth` | `number` | none | How many links away from the start page to go. With none, `limit` bounds the crawl. |
| `crawlEntireDomain` | `boolean` | `false` | Read the whole site, not only pages under the start page's path. |
| `allowSubdomains` | `boolean` | `false` | Also read pages on subdomains, such as `docs.bun.com` for `bun.com`. |
| `allowExternalLinks` | `boolean` | `false` | Also follow links to other sites. They count toward `limit` and `maxDiscoveryDepth`. |
| `sitemap` | `string` | `"include"` | `include` also uses the site's sitemaps. `only` reads the pages the sitemaps name. `skip` ignores them. |
| `ignoreQueryParameters` | `boolean` | `false` | Treat URLs that differ only in their `?query` part as one page. |
| `deduplicateSimilarURLs` | `boolean` | `true` | Read one page when URLs differ only by `www`, a trailing slash, a `#fragment`, tracking parameters such as `utm_source`, or the order of the `?query`. Send `false` to read each URL. |
| `prompt` | `string` | none | What you are looking for, such as `sqlite performance`, up to 1,000 characters. Pages about it are read first, and the crawl stops once new pages stop mentioning it. |

## How paths match

* `includePaths` and `excludePaths` are JavaScript regular expressions, up to 200 characters each. They are not anchored, so `blog` matches `/company/blog/post`. Start with `^` to anchor one.
* A pattern is tested against the path, such as `/docs/runtime/s3`. With `regexOnFullURL`, it is tested against the whole URL.
* A path that matches `excludePaths` is skipped even when it also matches `includePaths`.
* A group cannot be repeated with `*`, `+` or `{}`, such as `(a|b)+`. The call returns `400 invalidRequest` with the message `A group cannot be repeated with *, + or {}`.
* Pages are read in order of fit: first those a `prompt` points to, then those closest to the start page.

## Limits and errors

* A crawl reads at most `limit` pages. It can read fewer: a site may expose fewer pages in scope.
* A crawl and its pages are kept for 24 hours.
* Pages that the site's robots rules disallow are skipped. They are not counted as failures.
* A pattern that is not a valid regular expression returns `400 invalidRequest` with the message `Expected a regular expression such as blog/.*`.
* A page after the start page that cannot be read does not stop the crawl. It is listed in [the crawl's errors](https://docs.stophy.dev/crawl/async.md#read-the-errors).

## Price

\$1 per 1,000 pages read. A page counts toward `limit` when it is read. Scope fields add nothing. See [Pricing](https://docs.stophy.dev/account/pricing.md).

## Related

* [Crawl overview](https://docs.stophy.dev/crawl/overview) shows the request and the answer.
* [Page content](https://docs.stophy.dev/crawl/page-content.md) chooses how each page is read.
* [Describe a crawl](https://docs.stophy.dev/crawl/describe.md) turns a plain-language request into these fields.
* [Map](https://docs.stophy.dev/map/overview) lists a site's pages before you decide what to crawl.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.