> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Web web Crawl API

> Crawl a website and scrape its pages as markdown. Call the Web web Crawl API for JSON. From 1 credit per call. Every call returns the same flat shape.

Crawl a website and scrape its pages as markdown.

## Request

`url` is required.

<CodeGroup>
  ```typescript TypeScript icon="https://mintlify.s3.us-west-1.amazonaws.com/sirathic/images/icons/typescript.svg" theme={null} theme={null}
  import { Stophy } from "stophy";

  const stophy = new Stophy({ apiKey: process.env.STOPHY_API_KEY });
  const result = await stophy.web.crawl({ url: "https://bun.com/docs", goal: "sqlite", maxPages: 4 });
  console.log(result.data.results);
  ```

  ```python Python icon="python" theme={null} theme={null}
  from stophy import Stophy

  stophy = Stophy()  # reads STOPHY_API_KEY
  result = stophy.web.crawl(url="https://bun.com/docs", goal="sqlite", max_pages=4)
  print(result["data"]["results"])
  ```

  ```bash cURL icon="terminal" theme={null} theme={null}
  curl -X POST https://api.stophy.dev/v1/web/crawl \
    -H "Authorization: Bearer $STOPHY_API_KEY" \
    -H "content-type: application/json" \
    -d '{"url":"https://bun.com/docs","goal":"sqlite","maxPages":4}'
  ```

  ```bash CLI icon="square-terminal" theme={null} theme={null}
  stophy web crawl https://bun.com/docs --goal sqlite --maxPages 4
  ```
</CodeGroup>

## Response

If successful, the response body contains the standard envelope. `data.results[]` contains the pages read, each with its depth and its text as markdown.

The following example is a real response, truncated: each list shows one item and long strings are cut.

```json theme={null}
{
  "success": true,
  "data": {
    "pagesRead": 4,
    "results": [
      {
        "url": "https://bun.com/docs",
        "title": "Welcome to Bun | Bun Docs",
        "depth": 0,
        "markdown": "[Bun RuntimeA fast JavaScript runtime designed as a drop-in replacement for Node.jsGet started with `bun…"
      }
    ]
  },
  "creditsUsed": 1,
  "requestId": "2fe9dbad-fc48-4890-9751-0b24b596035b"
}
```

| Field | Type | Description |
| - | - | - |
| `depth` | integer | The nesting level of the comment in the thread. Top-level comments are 0. |

For every field, see the response schema on this page.

<Note>Set goal to read the pages about one topic first and stop when the site has no more about it. Use [Web web Map](/api-reference/endpoint/web-map) to only list pages.</Note>

## Optional parameters

| Parameter | Type | Description |
| - | - | - |
| `goal` | string | What you are looking for, such as sqlite performance. Pages about it are read first, and the crawl stops once new pages stop mentioning it. |
| `includePaths` | string\[] | Read only pages whose path matches one of these, such as /blog/\*. Accepts up to 20 values. |
| `excludePaths` | string\[] | Skip pages whose path matches one of these, such as /tag/\*. Accepts up to 20 values. |
| `maxDepth` | integer | The number of links away from the start page to go. Must be between 1 and 10. If unset, defaults to `3`. |
| `maxPages` | integer | Stop after reading this many pages in total, across every call. Must be between 1 and 1000. If unset, defaults to `100`. |
| `entireSite` | boolean | Read the whole site, not only pages under the start page's path. If unset, defaults to `false`. |
| `includeSubdomains` | boolean | Also read pages on subdomains, such as docs.bun.com for bun.com. If unset, defaults to `false`. |
| `onlyMainContent` | boolean | Keep only each page's main content, without menus, headers and footers. If unset, defaults to `true`. |
| `cursor` | string | The pagination token from the previous response. |

## Billing

A successful request consumes 1 credit a page for most pages, 2 or 3 credits when a page takes more work to load, and 15 credits for the hardest pages to reach. A page that cannot be read costs nothing. Each call reads up to 4 pages, up to 60 credits. Requests that fail, or that return no items, are not billed.

## Pagination

Results are paginated. To retrieve the next page, pass the value of `data.cursor` as `cursor` in a subsequent request. A response that omits `cursor` is the last page.


## OpenAPI

````yaml api-reference/openapi.json POST /v1/web/crawl
openapi: 3.1.0
info:
  title: Stophy API
  version: '1'
  description: >-
    Every endpoint is a POST that takes its input as a JSON object body. Unknown
    fields are rejected with 400.


    Responses leave out empty values: null, empty strings, empty objects and
    empty nested lists are not sent.


    Text comes back whole. To get shorter responses, add
    ?maxTextLength=<characters> to the URL. Every text value longer than that is
    cut and ends with …. A screenshot is never cut, and a page whose text was
    cut has isTruncated set to true.


    x-credits is what one call costs. Every call costs the same, whatever it
    returns, unless an operation's description says it costs per 10 results, or
    per second of audio. Then x-credits is the lowest price and x-credits-max
    the most one call can cost. Where an operation takes limit, send it to
    return fewer results and pay for fewer.


    When a response has a cursor, send it back unchanged with the same input to
    get the next page. When it has a page, send the next page number.
servers:
  - url: https://api.stophy.dev
security:
  - apiKey: []
paths:
  /v1/web/crawl:
    post:
      summary: Crawl a website and scrape its pages as markdown
      description: >-
        Costs 1 credit a page for most pages, 2 or 3 credits when a page takes
        more work to load, and 15 credits for the hardest pages to reach. A page
        that cannot be read costs nothing. Each call reads up to 4 pages. Pages
        with cursor.
      operationId: webCrawl
      parameters:
        - $ref: '#/components/parameters/MaxTextLength'
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                url:
                  description: The page to start from, such as https://bun.com/docs.
                  type: string
                goal:
                  description: >-
                    What you are looking for, such as sqlite performance. Pages
                    about it are read first, and the crawl stops once new pages
                    stop mentioning it.
                  type: string
                  minLength: 1
                  maxLength: 300
                includePaths:
                  default: []
                  description: >-
                    Read only pages whose path matches one of these, such as
                    /blog/*.
                  maxItems: 20
                  type: array
                  items:
                    type: string
                    minLength: 1
                    maxLength: 200
                excludePaths:
                  default: []
                  description: Skip pages whose path matches one of these, such as /tag/*.
                  maxItems: 20
                  type: array
                  items:
                    type: string
                    minLength: 1
                    maxLength: 200
                maxDepth:
                  default: 3
                  description: How many links away from the start page to go.
                  type: integer
                  minimum: 1
                  maximum: 10
                maxPages:
                  default: 100
                  description: >-
                    Stop after reading this many pages in total, across every
                    call.
                  type: integer
                  minimum: 1
                  maximum: 1000
                entireSite:
                  default: false
                  description: >-
                    Read the whole site, not only pages under the start page's
                    path.
                  type: boolean
                includeSubdomains:
                  default: false
                  description: >-
                    Also read pages on subdomains, such as docs.bun.com for
                    bun.com.
                  type: boolean
                onlyMainContent:
                  default: true
                  description: >-
                    Keep only each page's main content, without menus, headers
                    and footers.
                  type: boolean
                cursor:
                  type: string
                  minLength: 1
                  description: The cursor from the previous response. Send it as is.
              required:
                - url
              additionalProperties: false
            example:
              url: https://bun.com/docs
              goal: sqlite
              maxPages: 4
      responses:
        '200':
          description: The data, and the credits this call used.
          headers:
            X-Request-ID:
              $ref: '#/components/headers/RequestId'
            x-credits-used:
              $ref: '#/components/headers/CreditsUsed'
            X-RateLimit-Limit:
              $ref: '#/components/headers/RateLimitLimit'
            X-RateLimit-Remaining:
              $ref: '#/components/headers/RateLimitRemaining'
            X-RateLimit-Reset:
              $ref: '#/components/headers/RateLimitReset'
            x-cache:
              $ref: '#/components/headers/Cache'
            x-cache-age:
              $ref: '#/components/headers/CacheAge'
          content:
            application/json:
              schema:
                type: object
                required:
                  - success
                  - data
                  - creditsUsed
                  - requestId
                properties:
                  success:
                    const: true
                  data:
                    type: object
                    additionalProperties: false
                    properties:
                      pagesRead:
                        type: integer
                        minimum: -9007199254740991
                        maximum: 9007199254740991
                      results:
                        type: array
                        items:
                          type: object
                          additionalProperties: false
                          properties:
                            url:
                              type: string
                            title:
                              type: string
                            depth:
                              type: integer
                              minimum: -9007199254740991
                              maximum: 9007199254740991
                            markdown:
                              type: string
                          required:
                            - depth
                      cursor:
                        type: string
                    required:
                      - results
                      - pagesRead
                  creditsUsed:
                    type: integer
                  requestId:
                    type: string
        4XX:
          $ref: '#/components/responses/Error'
        5XX:
          $ref: '#/components/responses/Error'
components:
  parameters:
    MaxTextLength:
      name: maxTextLength
      in: query
      required: false
      description: >-
        Cut every text value in the response to this many characters. A value
        that was cut ends with …, and a page whose text was cut has isTruncated
        set to true. A screenshot is never cut. Leave it out to get the full
        text.
      schema:
        type: integer
        minimum: 1
  headers:
    RequestId:
      description: The request id. Quote it when you report a problem.
      schema:
        type: string
    CreditsUsed:
      description: Credits this call used.
      schema:
        type: integer
    RateLimitLimit:
      description: Requests allowed in the current window.
      schema:
        type: integer
    RateLimitRemaining:
      description: Requests left in the current window.
      schema:
        type: integer
    RateLimitReset:
      description: Unix time in seconds when the current window resets.
      schema:
        type: integer
    Cache:
      description: hit if this result came from the cache, miss if it was fetched live.
      schema:
        type: string
    CacheAge:
      description: Seconds since the returned result was written to the cache.
      schema:
        type: integer
    RetryAfter:
      description: Seconds to wait before retrying.
      schema:
        type: integer
  responses:
    Error:
      description: The request failed. retryable says whether trying again can help.
      headers:
        X-Request-ID:
          $ref: '#/components/headers/RequestId'
        Retry-After:
          $ref: '#/components/headers/RetryAfter'
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
  schemas:
    Error:
      type: object
      required:
        - success
        - error
      properties:
        success:
          const: false
        error:
          type: object
          required:
            - code
            - message
            - retryable
            - requestId
          properties:
            code:
              type: string
              enum:
                - cliSessionCompleted
                - cliSessionExists
                - cliSessionNotFound
                - forbidden
                - insufficientCredits
                - internalError
                - inviteClaimed
                - inviteDisabled
                - inviteExpired
                - inviteUsedUp
                - invalidCliSession
                - invalidCliVerifier
                - invalidRequest
                - notFound
                - rateLimited
                - sourceChanged
                - sourceRefused
                - sourceTimeout
                - sourceUnavailable
                - tooManyInFlight
                - unauthorized
                - unsupportedMediaType
            message:
              type: string
            retryable:
              type: boolean
            retryAfterSeconds:
              type: integer
            requestId:
              type: string
            requiredCredits:
              type: number
            availableCredits:
              type: number
            billingUrl:
              type: string
              const: https://stophy.dev/billing
  securitySchemes:
    apiKey:
      type: http
      scheme: bearer

````

This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.