> ## Documentation Index > Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt > Use this file to discover all available pages before exploring further. # Crawl > Crawl a website in the background. Send one request and get JSON back. Examples in cURL, TypeScript and Python. A failed call costs nothing. Best for reading a whole section of a site without holding a call open. It answers at once with an id; ask crawl.status for the pages, or send webhook to be told as each one finishes. ## Price 1 credit a page, however it is reached. json with a schema or a prompt, summary and highlights add 4 credits each when they return something. CSS rules and attributes add nothing. A PDF adds 1 credit for each page read. Each perform action adds 2 credits. An answer from a stored copy within maxAge is free. A page that comes back with a 404 or 410 status is charged like any page. A page that cannot be read costs nothing. Each page is charged when it finishes. Starting the job is free. See [Pricing](https://docs.stophy.dev/billing.md) for what credits cost. ## Limits An account runs up to 5 crawls and batch scrapes at once. A sixth start answers 400 `invalidRequest`: `jobs: you already have 5 jobs running. Wait for one of them to finish, then try again.` Wait until one is `completed`, then start it again. ```bash cURL theme={null} curl -X POST https://api.stophy.dev/v1/crawl \ -H "Authorization: Bearer $STOPHY_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://bun.com/docs","limit":2}' ``` ```typescript TypeScript theme={null} const response = await fetch("https://api.stophy.dev/v1/crawl", { method: "POST", headers: { Authorization: `Bearer ${process.env.STOPHY_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ url: "https://bun.com/docs", limit: 2 }), }); console.log(await response.json()); ``` ```python Python theme={null} import os import requests response = requests.post( "https://api.stophy.dev/v1/crawl", headers={"Authorization": f"Bearer {os.environ['STOPHY_API_KEY']}"}, json={"url": "https://bun.com/docs", "limit": 2}, ) print(response.json()) ``` ```json 200 theme={null} { "success": true, "id": "job_a-Bqc0NaZ8ADzLB4rhpeKA", "url": "https://api.stophy.dev/v1/crawl/job_a-Bqc0NaZ8ADzLB4rhpeKA" } ``` ## OpenAPI ````yaml api-reference/openapi.json POST /v1/crawl openapi: 3.1.0 info: title: Stophy API version: '1' description: >- Every endpoint is a POST that takes its input as a JSON object body. Unknown fields are rejected with 400. Responses leave out empty values: null, empty strings, empty objects and empty nested lists are not sent. Text comes back whole. To get shorter responses, add ?maxTextLength= to the URL. Every text value longer than that is cut and ends with …. A screenshot is never cut, and a page whose text was cut has isTruncated set to true. x-credits is what one call costs. Every call costs the same, whatever it returns, unless an operation's description says it costs per 10 results, or per second of audio. Then x-credits is the lowest price and x-credits-max the most one call can cost. Where an operation takes limit, send it to return fewer results and pay for fewer. When a response has a cursor, send it back unchanged with the same input to get the next page. When it has a page, send the next page number. servers: - url: https://api.stophy.dev security: - apiKey: [] paths: /v1/crawl: post: summary: Crawl a website in the background description: >- Costs 1 credit a page, however it is reached. json with a schema or a prompt, summary and highlights add 4 credits each when they return something. CSS rules and attributes add nothing. A PDF adds 1 credit for each page read. Each perform action adds 2 credits. An answer from a stored copy within maxAge is free. A page that comes back with a 404 or 410 status is charged like any page. A page that cannot be read costs nothing. Each page is charged when it finishes. Starting the job is free. operationId: crawl parameters: - $ref: '#/components/parameters/MaxTextLength' requestBody: required: true content: application/json: schema: type: object properties: url: description: The page to start from, such as https://bun.com/docs. type: string prompt: description: >- What you are looking for, such as sqlite performance. Pages about it are read first, and the crawl stops once new pages stop mentioning it. type: string minLength: 1 maxLength: 1000 includePaths: default: [] description: >- Read only pages whose path matches one of these regular expressions, such as blog/.*. maxItems: 100 type: array items: type: string minLength: 1 maxLength: 200 excludePaths: default: [] description: >- Skip pages whose path matches one of these regular expressions, such as tag/.*. maxItems: 100 type: array items: type: string minLength: 1 maxLength: 200 maxDiscoveryDepth: default: 3 description: How many links away from the start page to go, up to 10. type: integer minimum: 0 maximum: 10 sitemap: default: include description: >- include also uses the site's sitemaps. only reads just the pages the sitemaps name. skip ignores them. type: string enum: - skip - include - only ignoreQueryParameters: default: false description: >- Treat addresses that differ only in their ?query part as one page. type: boolean regexOnFullURL: default: false description: >- Match includePaths and excludePaths against the whole address, not only the path. type: boolean limit: default: 10000 description: >- Stop after reading this many pages in total, across every call. type: integer minimum: 1 maximum: 10000 crawlEntireDomain: default: false description: >- Read the whole site, not only pages under the start page's path. type: boolean allowExternalLinks: type: boolean const: false allowSubdomains: default: false description: >- Also read pages on subdomains, such as docs.bun.com for bun.com. type: boolean ignoreRobotsTxt: type: boolean const: false robotsUserAgent: not: {} delay: not: {} maxConcurrency: not: {} scrapeOptions: description: >- How to read each page, the same options and formats as scrape. Without formats, each page comes back as markdown. type: object properties: formats: description: >- What to return: markdown (clean page text), summary (a short summary of the page), html (the cleaned page), rawHtml (the page as sent), links (every link on the page), images (every image address), screenshot (a link to a JPEG of the visible page, kept for 7 days), changeTracking (what changed in the markdown since your key last read this page with the same options, kept 30 days) network (the JSON the page loaded while it rendered, up to 30 responses) and branding (the page's colors, fonts, typography, spacing, button and input styles, logo and brand images, read from how the page looks, with a field left out when the page does not show it). Send { "type": "screenshot", "fullPage": true } for the whole page. Send { "type": "highlights", "query": "refund policy" } for the passages that best match a query. Send { "type": "json", "schema": { "type": "object", "properties": { "price": { "type": "number" } } } } or { "type": "json", "prompt": "the price and the stock" } for fields filled from the page, each checked against it. Send { "type": "json", "parser": "amazon-product" } to get a site's own data as json instead of reading the page; GET /v1/parsers lists the parsers. Send { "type": "json", "rules": { "title": "h1", "links": { "selector": "a", "type": "list", "output": "@href" } } } to read the page with CSS rules and no model, at the page's own price. Send { "type": "attributes", "selectors": [{ "selector": "a", "attribute": "href" }] } for attribute values, returned in attributes. Without a parser or rules, json needs a schema or a prompt and is filled from the page. json with a schema or a prompt, summary and highlights cost 4 credits more each when they return something. json with rules or a parser, and attributes, cost nothing more. A screenshot, network or branding read costs 1 credit, the same as any page. A PDF comes back as markdown only, and costs 1 credit more for each page read. default: - markdown minItems: 1 maxItems: 20 type: array items: anyOf: - type: string enum: - markdown - summary - html - rawHtml - links - images - network - branding - screenshot - changeTracking - type: object properties: type: type: string enum: - markdown - summary - html - rawHtml - links - images - network - branding required: - type additionalProperties: false - type: object properties: type: type: string const: changeTracking modes: type: array items: type: string const: git-diff schema: not: {} prompt: not: {} tag: type: 'null' required: - type additionalProperties: false - type: object properties: type: type: string const: screenshot fullPage: default: false description: >- Capture the whole page from top to bottom instead of the visible part. type: boolean quality: not: {} viewport: not: {} required: - type additionalProperties: false - type: object properties: type: type: string const: highlights query: type: string minLength: 1 maxLength: 10000 description: >- What you are looking for on the page, such as refund policy. count: default: 5 description: How many passages to return, best match first. type: integer minimum: 1 maximum: 20 required: - type - query additionalProperties: false - type: object properties: type: type: string const: json parser: description: >- Read the page with a named parser, which gives the site's own data as json. The url must be the kind of page the parser reads. Parsers are the only way to read some sites. GET /v1/parsers lists them. type: string options: description: >- Only with parser: extra inputs for it, such as { "page": 2 } for the next page of results or { "minRating": 4.5 } for a filter. GET /v1/parsers lists what each parser accepts. type: object propertyNames: type: string additionalProperties: $ref: '#/components/schemas/crawl.input.schema0' schema: description: >- The fields to fill, as a JSON Schema (draft-07 or 2020-12) object with properties. Pydantic's model_json_schema() and zod's toJSONSchema() output work as they are. enum, const, anyOf, oneOf, nullable types, $ref with $defs, required, descriptions and any key names are read. Up to 100 top-level fields and 32 KB. A field the page does not state comes back null. type: object propertyNames: type: string additionalProperties: $ref: '#/components/schemas/crawl.input.schema1' prompt: description: >- What to pull from the page in plain words, such as the version and the release date. type: string minLength: 1 maxLength: 10000 rules: description: >- Read the page with CSS rules instead of a model, no extra fee. Each field is a selector such as "h1" (the first match's text), "a@href" (an attribute) or { "selector": "li", "type": "list", "output": "text" | "html" | "@attribute" | { nested fields } }. A missing item is null and an unmatched list is []. Relative links become absolute. Up to 100 fields, 5 levels deep. Not with schema, prompt or parser. type: object propertyNames: type: string additionalProperties: $ref: '#/components/schemas/crawl.input.schema2' checkPromptInjection: type: boolean const: false required: - type additionalProperties: false - type: object properties: type: type: string const: attributes selectors: minItems: 1 maxItems: 50 type: array items: type: object properties: selector: type: string minLength: 1 attribute: type: string minLength: 1 maxLength: 100 required: - selector - attribute additionalProperties: false description: >- The elements and attributes to read, such as { selector: 'a.buy', attribute: 'href' }. required: - type - selectors additionalProperties: false onlyMainContent: default: true description: >- Keep only the page's main content, without menus, headers, footers and ads. type: boolean onlyCleanContent: type: boolean const: false includeTags: description: >- Keep only the parts of the page that match these CSS selectors, such as article or .price. When set, onlyMainContent is not applied. minItems: 1 type: array items: type: string minLength: 1 excludeTags: description: >- Remove the parts of the page that match these CSS selectors, such as .comments. minItems: 1 type: array items: type: string minLength: 1 maxAge: description: >- Accept a stored answer up to this many milliseconds old, at most 30 days. Without it, 2 days, or less for a site whose pages change often, and a copy up to twice that age (at most 7 days) comes back at once as stale while a fresh read runs. 0 always reads the page live. metadata.cacheState says which one you got. type: integer minimum: 0 maximum: 9007199254740991 minAge: description: >- Answer only from a stored copy that is at least this many milliseconds old, and never read the page live. 0 accepts any stored copy. A page with no such copy is notFound. type: integer minimum: 0 maximum: 9007199254740991 headers: not: {} waitFor: description: >- Extra milliseconds to wait before reading, up to 60000, for pages that load their content late. To wait for an element, add a wait action with a selector. A read that waits costs 1 credit, the same as any page. type: integer minimum: 0 maximum: 60000 mobile: type: boolean const: false skipTlsVerification: type: boolean const: false timeout: description: >- How long to keep trying, in milliseconds, between 10 seconds and 2 minutes. Pages behind a bot check can need 10 to 30 seconds the first time. type: integer minimum: 1000 maximum: 300000 parsers: description: 'Only ["pdf"] is supported: PDFs are read as markdown.' minItems: 1 type: array items: anyOf: - type: string const: pdf - type: object properties: type: type: string const: pdf mode: type: string const: auto maxPages: not: {} pages: type: boolean const: false blocks: type: boolean const: false pageMarkers: type: boolean const: false required: - type additionalProperties: false actions: description: >- Up to 50 steps to run on the page, in order, before reading it: { "type": "wait", "milliseconds": 500 } (up to 30 seconds), { "type": "wait", "selector": ".results" }, { "type": "click", "selector": "button.more" }, { "type": "write", "text": "tides" }, { "type": "press", "key": "Enter" }, { "type": "scroll", "direction": "down" } (also up, left or right; amount is pixels, "viewport" or "max"; maxScrolls 1 to 50; selector scrolls an inner container; "max" keeps scrolling until the page stops growing, for feeds that load as you go), { "type": "screenshot", "fullPage": true } (links come back in actions.screenshots), { "type": "scrape" } (the page at that point, in actions.scrapes), { "type": "executeJavascript", "script": "document.title" } (results in actions.javascriptReturns), { "type": "pdf", "format": "A4" } (links in actions.pdfs) or { "type": "perform", "action": "Click the Load more button" } (a step in plain words, up to 5 browser steps on this page, never leaving the site; at most 5 per call). A read with actions costs 1 credit, the same as any page, and each perform action costs 2 credits more. If an action fails, you still get the page as it stood, with a warning that names the action and says why. minItems: 1 maxItems: 50 type: array items: anyOf: - type: object properties: type: type: string const: wait milliseconds: type: integer minimum: 1 maximum: 30000 required: - type - milliseconds additionalProperties: false - type: object properties: type: type: string const: wait selector: type: string minLength: 1 required: - type - selector additionalProperties: false - type: object properties: type: type: string const: click selector: type: string minLength: 1 all: type: boolean const: false required: - type - selector additionalProperties: false - type: object properties: type: type: string const: write text: type: string minLength: 1 required: - type - text additionalProperties: false - type: object properties: type: type: string const: press key: type: string minLength: 1 maxLength: 40 required: - type - key additionalProperties: false - type: object properties: type: type: string const: scroll direction: default: down type: string enum: - up - down - left - right amount: anyOf: - type: integer minimum: 1 maximum: 100000 - type: string enum: - viewport - max maxScrolls: type: integer minimum: 1 maximum: 50 selector: type: string minLength: 1 required: - type additionalProperties: false - type: object properties: type: type: string const: screenshot fullPage: type: boolean quality: type: integer minimum: 1 maximum: 100 viewport: type: object properties: width: type: integer minimum: 200 maximum: 3840 height: type: integer minimum: 200 maximum: 2160 required: - width - height additionalProperties: false required: - type additionalProperties: false - type: object properties: type: type: string const: scrape required: - type additionalProperties: false - type: object properties: type: type: string const: executeJavascript script: type: string minLength: 1 maxLength: 20000 required: - type - script additionalProperties: false - type: object properties: type: type: string const: pdf format: type: string enum: - Letter - Legal - Tabloid - Ledger - A0 - A1 - A2 - A3 - A4 - A5 - A6 landscape: type: boolean scale: type: number minimum: 0.1 maximum: 2 required: - type additionalProperties: false - type: object properties: type: type: string const: perform action: type: string minLength: 1 maxLength: 500 required: - type - action additionalProperties: false location: description: >- Where to read the page from. The default is the United States. type: object properties: country: description: >- Two-letter country code to read the page from, such as DE or JP. type: string languages: not: {} additionalProperties: false removeBase64Images: type: boolean const: true blockAds: type: boolean const: true proxy: type: string const: auto storeInCache: default: true description: >- Send false to neither read nor keep a stored copy of this page. type: boolean lockdown: type: boolean const: false checkPromptInjection: type: boolean const: false redactPII: type: boolean const: false profile: description: >- Read the page with the cookies and local storage that earlier calls saved under this name, kept for your account for 30 days after the last use. A call with a profile always loads the page in a browser, costs 1 credit, the same as any page, and is never answered from a stored copy. Only one call at a time can save to a profile. type: object properties: name: type: string minLength: 1 maxLength: 128 description: >- A name you choose, such as shop-login. Calls with the same name share state. saveChanges: default: true description: >- Keep the cookies and local storage this read leaves behind for the next call with this name. Send false to read with the profile without changing it. type: boolean required: - name additionalProperties: false threatProtection: not: {} auditMetadata: not: {} zeroDataRetention: default: false description: >- Send true to keep nothing from this call: no stored copy is read or made, and nothing from it is logged. Costs 1 credit more per page. Not with a screenshot. type: boolean alexandria: not: {} domainTools: type: boolean const: false toolDetail: not: {} zeroDataRetention: default: false description: >- Send true to keep nothing from this crawl: no stored copy is read or made, page results are held for one hour at most, and nothing is logged. Costs 1 credit more per page. type: boolean webhook: description: >- Where to be told about the job: { url } gets a JSON POST when the job starts, for each finished page, and when the job ends. If your address answers slowly, a page event can carry only the page url. Read that page with the status call. type: object properties: url: type: string headers: not: {} metadata: not: {} events: not: {} required: - url additionalProperties: false required: - url additionalProperties: false example: url: https://bun.com/docs limit: 2 responses: '200': description: The data, and the credits this call used. headers: X-Request-ID: $ref: '#/components/headers/RequestId' x-credits-used: $ref: '#/components/headers/CreditsUsed' X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' X-RateLimit-Reset: $ref: '#/components/headers/RateLimitReset' x-cache: $ref: '#/components/headers/Cache' x-cache-age: $ref: '#/components/headers/CacheAge' content: application/json: schema: type: object additionalProperties: false properties: success: type: boolean const: true id: type: string url: type: string required: - success 4XX: $ref: '#/components/responses/Error' 5XX: $ref: '#/components/responses/Error' components: parameters: MaxTextLength: name: maxTextLength in: query required: false description: >- Cut every text value in the response to this many characters. A value that was cut ends with …, and a page whose text was cut has isTruncated set to true. A screenshot is never cut. Leave it out to get the full text. schema: type: integer minimum: 1 schemas: crawl.input.schema0: anyOf: - type: string - type: number - type: boolean - type: 'null' - type: array items: $ref: '#/components/schemas/crawl.input.schema0' - type: object propertyNames: type: string additionalProperties: $ref: '#/components/schemas/crawl.input.schema0' crawl.input.schema1: anyOf: - type: string - type: number - type: boolean - type: 'null' - type: array items: $ref: '#/components/schemas/crawl.input.schema1' - type: object propertyNames: type: string additionalProperties: $ref: '#/components/schemas/crawl.input.schema1' crawl.input.schema2: anyOf: - type: string - type: number - type: boolean - type: 'null' - type: array items: $ref: '#/components/schemas/crawl.input.schema2' - type: object propertyNames: type: string additionalProperties: $ref: '#/components/schemas/crawl.input.schema2' Error: type: object required: - success - error - code properties: success: const: false error: type: string description: What went wrong and what to do next. code: type: string enum: - apiKeyRequired - cliSessionCompleted - cliSessionExists - cliSessionNotFound - forbidden - freeLimit - insufficientCredits - internalError - inviteClaimed - inviteDisabled - inviteExpired - inviteUsedUp - invalidCliSession - invalidCliVerifier - invalidRequest - jobFinished - methodNotAllowed - notFound - payloadTooLarge - profileBusy - profileTooLarge - rateLimited - sourceChanged - sourceRefused - siteNotRead - sourceTimeout - sourceUnavailable - unauthorized - unsupportedMediaType retryAfterSeconds: type: integer signupUrl: type: string const: https://stophy.dev/signup description: >- Where a person gets a key. Sent with apiKeyRequired, freeLimit and an invalid key. signup: type: string const: POST https://api.stophy.dev/v1/signup description: >- The call that gets an agent a key with no human. Sent next to signupUrl. headers: RequestId: description: The request id. Quote it when you report a problem. schema: type: string CreditsUsed: description: Credits this call used. schema: type: integer RateLimitLimit: description: Requests allowed in the current window. schema: type: integer RateLimitRemaining: description: Requests left in the current window. schema: type: integer RateLimitReset: description: Unix time in seconds when the current window resets. schema: type: integer Cache: description: hit if this result came from the cache, miss if it was fetched live. schema: type: string CacheAge: description: Seconds since the returned result was written to the cache. schema: type: integer RetryAfter: description: Seconds to wait before retrying. schema: type: integer responses: Error: description: The request failed. Retry-After says when trying again can help. headers: X-Request-ID: $ref: '#/components/headers/RequestId' Retry-After: $ref: '#/components/headers/RetryAfter' content: application/json: schema: $ref: '#/components/schemas/Error' securitySchemes: apiKey: type: http scheme: bearer ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.