Web web Crawl API

Crawl a website and scrape its pages as markdown

curl --request POST \
  --url https://api.stophy.dev/v1/web/crawl \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "url": "https://bun.com/docs",
  "goal": "sqlite",
  "maxPages": 4
}
'

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({url: 'https://bun.com/docs', goal: 'sqlite', maxPages: 4})
};

fetch('https://api.stophy.dev/v1/web/crawl', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

import requests

url = "https://api.stophy.dev/v1/web/crawl"

payload = {
    "url": "https://bun.com/docs",
    "goal": "sqlite",
    "maxPages": 4
}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

{
  "success": true,
  "data": {
    "pagesRead": 0,
    "results": [
      {
        "depth": 0,
        "url": "<string>",
        "title": "<string>",
        "markdown": "<string>"
      }
    ],
    "cursor": "<string>"
  },
  "creditsUsed": 123,
  "requestId": "<string>"
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

POST

web

crawl

Crawl a website and scrape its pages as markdown

curl --request POST \
  --url https://api.stophy.dev/v1/web/crawl \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "url": "https://bun.com/docs",
  "goal": "sqlite",
  "maxPages": 4
}
'

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({url: 'https://bun.com/docs', goal: 'sqlite', maxPages: 4})
};

fetch('https://api.stophy.dev/v1/web/crawl', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

import requests

url = "https://api.stophy.dev/v1/web/crawl"

payload = {
    "url": "https://bun.com/docs",
    "goal": "sqlite",
    "maxPages": 4
}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

{
  "success": true,
  "data": {
    "pagesRead": 0,
    "results": [
      {
        "depth": 0,
        "url": "<string>",
        "title": "<string>",
        "markdown": "<string>"
      }
    ],
    "cursor": "<string>"
  },
  "creditsUsed": 123,
  "requestId": "<string>"
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

Crawl a website and scrape its pages as markdown.

Request

url is required.

import { Stophy } from "stophy";

const stophy = new Stophy({ apiKey: process.env.STOPHY_API_KEY });
const result = await stophy.web.crawl({ url: "https://bun.com/docs", goal: "sqlite", maxPages: 4 });
console.log(result.data.results);

from stophy import Stophy

stophy = Stophy()  # reads STOPHY_API_KEY
result = stophy.web.crawl(url="https://bun.com/docs", goal="sqlite", max_pages=4)
print(result["data"]["results"])

curl -X POST https://api.stophy.dev/v1/web/crawl \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://bun.com/docs","goal":"sqlite","maxPages":4}'

stophy web crawl https://bun.com/docs --goal sqlite --maxPages 4

Response

If successful, the response body contains the standard envelope. data contains pagesRead, results and cursor. The following example is a real response, truncated: each list shows one item and long strings are cut.

{
  "success": true,
  "data": {
    "pagesRead": 4,
    "results": [
      {
        "url": "https://bun.com/docs",
        "title": "Welcome to Bun | Bun Docs",
        "depth": 0,
        "markdown": "[Bun RuntimeA fast JavaScript runtime designed as a drop-in replacement for Node.jsGet started with `bun…"
      }
    ]
  },
  "creditsUsed": 1,
  "requestId": "2fe9dbad-fc48-4890-9751-0b24b596035b"
}

Field	Type	Description
`depth`	integer	The nesting level of the comment in the thread. Top-level comments are 0.

For every field, see the response schema on this page.

Set goal to read the pages about one topic first and stop when the site has no more about it. Use Web web Map to only list pages.

Optional parameters

Parameter	Type	Description
`goal`	string	What you are looking for, such as sqlite performance. Pages about it are read first, and the crawl stops once new pages stop mentioning it.
`includePaths`	string[]	Read only pages whose path matches one of these, such as /blog/*. Accepts up to 20 values.
`excludePaths`	string[]	Skip pages whose path matches one of these, such as /tag/*. Accepts up to 20 values.
`maxDepth`	integer	The number of links away from the start page to go. Must be between 1 and 10. If unset, defaults to `3`.
`maxPages`	integer	Stop after reading this many pages in total, across every call. Must be between 1 and 1000. If unset, defaults to `100`.
`entireSite`	boolean	Read the whole site, not only pages under the start page’s path. If unset, defaults to `false`.
`includeSubdomains`	boolean	Also read pages on subdomains, such as docs.bun.com for bun.com. If unset, defaults to `false`.
`onlyMainContent`	boolean	Keep only each page’s main content, without menus, headers and footers. If unset, defaults to `true`.
`cursor`	string	The pagination token from the previous response.

Billing

A successful request consumes 1 credit a page for most pages, 2 or 3 credits when a page takes more work to load, and 15 credits for the hardest pages to reach. A page that cannot be read costs nothing. Each call reads up to 4 pages, up to 60 credits. Requests that fail, or that return no items, are not billed.

Pagination

Results are paginated. To retrieve the next page, pass the value of data.cursor as cursor in a subsequent request. A response that omits cursor is the last page.

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Query Parameters

maxTextLength

integer

Cut every text value in the response to this many characters. A value that was cut ends with …, and a page whose text was cut has isTruncated set to true. A screenshot is never cut. Leave it out to get the full text.

Required range: x >= 1

Body

application/json

url

string

required

The page to start from, such as https://bun.com/docs.

goal

string

What you are looking for, such as sqlite performance. Pages about it are read first, and the crawl stops once new pages stop mentioning it.

Required string length: 1 - 300

includePaths

string[]

Read only pages whose path matches one of these, such as /blog/*.

Maximum array length: 20

Required string length: 1 - 200

excludePaths

string[]

Skip pages whose path matches one of these, such as /tag/*.

Maximum array length: 20

Required string length: 1 - 200

maxDepth

integer

default:3

How many links away from the start page to go.

Required range: 1 <= x <= 10

maxPages

integer

default:100

Stop after reading this many pages in total, across every call.

Required range: 1 <= x <= 1000

entireSite

boolean

default:false

Read the whole site, not only pages under the start page's path.

includeSubdomains

boolean

default:false

Also read pages on subdomains, such as docs.bun.com for bun.com.

onlyMainContent

boolean

default:true

Keep only each page's main content, without menus, headers and footers.

cursor

string

The cursor from the previous response. Send it as is.

Minimum string length: 1

Response

The data, and the credits this call used.

success

any

required

data

object

required

Show child attributes

creditsUsed

integer

required

requestId

string

required

Using the API

Catalog

Account

Search and AI answers

Video

Social

Places and travel

Jobs

Shopping

Apps

Real estate

Ads

Web web Crawl API

Request

Response

Optional parameters

Billing

Authorizations

Query Parameters

Body

Response

​Request

​Response

​Optional parameters

​Billing

​Pagination

Authorizations

Query Parameters

Body

Response

Request

Response

Optional parameters

Billing

Pagination