> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Web web Crawl Start API

> Crawl a website in the background. Call the Web web Crawl Start API and get JSON back in one request. From 0 credits per call. Failed calls cost nothing.

Crawl a website in the background.

## Request

`url` is required.

<CodeGroup>
  ```typescript TypeScript icon="https://mintlify.s3.us-west-1.amazonaws.com/sirathic/images/icons/typescript.svg" theme={null} theme={null}
  import { Stophy } from "stophy";

  const stophy = new Stophy({ apiKey: process.env.STOPHY_API_KEY });
  const result = await stophy.web.crawl.start({ url: "https://bun.com/docs", goal: "sqlite", maxPages: 4 });
  console.log(result.data);
  ```

  ```python Python icon="python" theme={null} theme={null}
  from stophy import Stophy

  stophy = Stophy()  # reads STOPHY_API_KEY
  result = stophy.web.crawl.start(url="https://bun.com/docs", goal="sqlite", max_pages=4)
  print(result["data"])
  ```

  ```bash cURL icon="terminal" theme={null} theme={null}
  curl -X POST https://api.stophy.dev/v1/web/crawl/start \
    -H "Authorization: Bearer $STOPHY_API_KEY" \
    -H "content-type: application/json" \
    -d '{"url":"https://bun.com/docs","goal":"sqlite","maxPages":4}'
  ```

  ```bash CLI icon="square-terminal" theme={null} theme={null}
  stophy web crawl start https://bun.com/docs --goal sqlite --maxPages 4
  ```
</CodeGroup>

## Response

If successful, the response body contains the standard envelope. `data` contains the crawl's id and status, to follow with `web.crawl.status`.

The following example is a real response, truncated: each list shows one item and long strings are cut.

```json theme={null}
{
  "success": true,
  "data": {
    "id": "job_Qg9km6Ss5DAJktZgPYOUkA",
    "status": "queued",
    "total": 1
  },
  "creditsUsed": 0,
  "requestId": "2fe9dbad-fc48-4890-9751-0b24b596035b"
}
```

| Field | Type | Description |
| - | - | - |
| `total` | integer | The total number of matching results, across all pages. |

For every field, see the response schema on this page.

<Note>It answers at once with an id; ask [Web web Crawl Status](/api-reference/endpoint/web-crawl-status) for the pages, or send webhook to be told as each one finishes.</Note>

## Optional parameters

| Parameter | Type | Description |
| - | - | - |
| `goal` | string | What you are looking for, such as sqlite performance. Pages about it are read first, and the crawl stops once new pages stop mentioning it. |
| `includePaths` | string\[] | Read only pages whose path matches one of these, such as /blog/\*. Accepts up to 20 values. |
| `excludePaths` | string\[] | Skip pages whose path matches one of these, such as /tag/\*. Accepts up to 20 values. |
| `maxDepth` | integer | The number of links away from the start page to go. Must be between 1 and 10. If unset, defaults to `3`. |
| `maxPages` | integer | Stop after reading this many pages in total, across every call. Must be between 1 and 1000. If unset, defaults to `100`. |
| `entireSite` | boolean | Read the whole site, not only pages under the start page's path. If unset, defaults to `false`. |
| `includeSubdomains` | boolean | Also read pages on subdomains, such as docs.bun.com for bun.com. If unset, defaults to `false`. |
| `onlyMainContent` | boolean | Keep only each page's main content, without menus, headers and footers. If unset, defaults to `true`. |
| `webhook` | string | A public address that gets a JSON POST when the job starts, for each finished page, and when the job ends. |

## Billing

A successful request consumes 1 credit a page for most pages, 2 or 3 credits when a page takes more work to load, and 15 credits for the hardest pages to reach. A page that cannot be read costs nothing. Each page is charged when it finishes. Starting the job is free, up to 15000 credits. Requests that fail, or that return no items, are not billed.

## Pagination

This method is not paginated. A single request returns the complete result.


## OpenAPI

````yaml api-reference/openapi.json POST /v1/web/crawl/start
openapi: 3.1.0
info:
  title: Stophy API
  version: '1'
  description: >-
    Every endpoint is a POST that takes its input as a JSON object body. Unknown
    fields are rejected with 400.


    Responses leave out empty values: null, empty strings, empty objects and
    empty nested lists are not sent.


    Text comes back whole. To get shorter responses, add
    ?maxTextLength=<characters> to the URL. Every text value longer than that is
    cut and ends with …. A screenshot is never cut, and a page whose text was
    cut has isTruncated set to true.


    x-credits is what one call costs. Every call costs the same, whatever it
    returns, unless an operation's description says it costs per 10 results, or
    per second of audio. Then x-credits is the lowest price and x-credits-max
    the most one call can cost. Where an operation takes limit, send it to
    return fewer results and pay for fewer.


    When a response has a cursor, send it back unchanged with the same input to
    get the next page. When it has a page, send the next page number.
servers:
  - url: https://api.stophy.dev
security:
  - apiKey: []
paths:
  /v1/web/crawl/start:
    post:
      summary: Crawl a website in the background
      description: >-
        Costs 1 credit a page for most pages, 2 or 3 credits when a page takes
        more work to load, and 15 credits for the hardest pages to reach. A page
        that cannot be read costs nothing. Each page is charged when it
        finishes. Starting the job is free.
      operationId: webCrawlStart
      parameters:
        - $ref: '#/components/parameters/MaxTextLength'
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                url:
                  description: The page to start from, such as https://bun.com/docs.
                  type: string
                goal:
                  description: >-
                    What you are looking for, such as sqlite performance. Pages
                    about it are read first, and the crawl stops once new pages
                    stop mentioning it.
                  type: string
                  minLength: 1
                  maxLength: 300
                includePaths:
                  default: []
                  description: >-
                    Read only pages whose path matches one of these, such as
                    /blog/*.
                  maxItems: 20
                  type: array
                  items:
                    type: string
                    minLength: 1
                    maxLength: 200
                excludePaths:
                  default: []
                  description: Skip pages whose path matches one of these, such as /tag/*.
                  maxItems: 20
                  type: array
                  items:
                    type: string
                    minLength: 1
                    maxLength: 200
                maxDepth:
                  default: 3
                  description: How many links away from the start page to go.
                  type: integer
                  minimum: 1
                  maximum: 10
                maxPages:
                  default: 100
                  description: >-
                    Stop after reading this many pages in total, across every
                    call.
                  type: integer
                  minimum: 1
                  maximum: 1000
                entireSite:
                  default: false
                  description: >-
                    Read the whole site, not only pages under the start page's
                    path.
                  type: boolean
                includeSubdomains:
                  default: false
                  description: >-
                    Also read pages on subdomains, such as docs.bun.com for
                    bun.com.
                  type: boolean
                onlyMainContent:
                  default: true
                  description: >-
                    Keep only each page's main content, without menus, headers
                    and footers.
                  type: boolean
                webhook:
                  description: >-
                    A public address that gets a JSON POST when the job starts,
                    for each finished page, and when the job ends.
                  type: string
              required:
                - url
              additionalProperties: false
      responses:
        '200':
          description: The data, and the credits this call used.
          headers:
            X-Request-ID:
              $ref: '#/components/headers/RequestId'
            x-credits-used:
              $ref: '#/components/headers/CreditsUsed'
            X-RateLimit-Limit:
              $ref: '#/components/headers/RateLimitLimit'
            X-RateLimit-Remaining:
              $ref: '#/components/headers/RateLimitRemaining'
            X-RateLimit-Reset:
              $ref: '#/components/headers/RateLimitReset'
            x-cache:
              $ref: '#/components/headers/Cache'
            x-cache-age:
              $ref: '#/components/headers/CacheAge'
          content:
            application/json:
              schema:
                type: object
                required:
                  - success
                  - data
                  - creditsUsed
                  - requestId
                properties:
                  success:
                    const: true
                  data:
                    type: object
                    additionalProperties: false
                    properties:
                      id:
                        type: string
                      status:
                        type: string
                        enum:
                          - queued
                          - running
                          - completed
                          - cancelled
                          - failed
                      total:
                        type: integer
                        minimum: 0
                        maximum: 9007199254740991
                    required:
                      - status
                      - total
                  creditsUsed:
                    type: integer
                  requestId:
                    type: string
        4XX:
          $ref: '#/components/responses/Error'
        5XX:
          $ref: '#/components/responses/Error'
components:
  parameters:
    MaxTextLength:
      name: maxTextLength
      in: query
      required: false
      description: >-
        Cut every text value in the response to this many characters. A value
        that was cut ends with …, and a page whose text was cut has isTruncated
        set to true. A screenshot is never cut. Leave it out to get the full
        text.
      schema:
        type: integer
        minimum: 1
  headers:
    RequestId:
      description: The request id. Quote it when you report a problem.
      schema:
        type: string
    CreditsUsed:
      description: Credits this call used.
      schema:
        type: integer
    RateLimitLimit:
      description: Requests allowed in the current window.
      schema:
        type: integer
    RateLimitRemaining:
      description: Requests left in the current window.
      schema:
        type: integer
    RateLimitReset:
      description: Unix time in seconds when the current window resets.
      schema:
        type: integer
    Cache:
      description: hit if this result came from the cache, miss if it was fetched live.
      schema:
        type: string
    CacheAge:
      description: Seconds since the returned result was written to the cache.
      schema:
        type: integer
    RetryAfter:
      description: Seconds to wait before retrying.
      schema:
        type: integer
  responses:
    Error:
      description: The request failed. retryable says whether trying again can help.
      headers:
        X-Request-ID:
          $ref: '#/components/headers/RequestId'
        Retry-After:
          $ref: '#/components/headers/RetryAfter'
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
  schemas:
    Error:
      type: object
      required:
        - success
        - error
      properties:
        success:
          const: false
        error:
          type: object
          required:
            - code
            - message
            - retryable
            - requestId
          properties:
            code:
              type: string
              enum:
                - cliSessionCompleted
                - cliSessionExists
                - cliSessionNotFound
                - forbidden
                - insufficientCredits
                - internalError
                - inviteClaimed
                - inviteDisabled
                - inviteExpired
                - inviteUsedUp
                - invalidCliSession
                - invalidCliVerifier
                - invalidRequest
                - notFound
                - rateLimited
                - sourceChanged
                - sourceRefused
                - sourceTimeout
                - sourceUnavailable
                - tooManyInFlight
                - unauthorized
                - unsupportedMediaType
            message:
              type: string
            retryable:
              type: boolean
            retryAfterSeconds:
              type: integer
            requestId:
              type: string
            requiredCredits:
              type: number
            availableCredits:
              type: number
            billingUrl:
              type: string
              const: https://stophy.dev/billing
  securitySchemes:
    apiKey:
      type: http
      scheme: bearer

````

This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.