Web web Crawl Start API

Crawl a website in the background

curl --request POST \
  --url https://api.stophy.dev/v1/web/crawl/start \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "url": "<string>",
  "goal": "<string>",
  "includePaths": [],
  "excludePaths": [],
  "maxDepth": 3,
  "maxPages": 100,
  "entireSite": false,
  "includeSubdomains": false,
  "onlyMainContent": true,
  "webhook": "<string>"
}
'

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({
    url: '<string>',
    goal: '<string>',
    includePaths: [],
    excludePaths: [],
    maxDepth: 3,
    maxPages: 100,
    entireSite: false,
    includeSubdomains: false,
    onlyMainContent: true,
    webhook: '<string>'
  })
};

fetch('https://api.stophy.dev/v1/web/crawl/start', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

import requests

url = "https://api.stophy.dev/v1/web/crawl/start"

payload = {
    "url": "<string>",
    "goal": "<string>",
    "includePaths": [],
    "excludePaths": [],
    "maxDepth": 3,
    "maxPages": 100,
    "entireSite": False,
    "includeSubdomains": False,
    "onlyMainContent": True,
    "webhook": "<string>"
}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

{
  "success": true,
  "data": {
    "status": "queued",
    "total": 4503599627370495,
    "id": "<string>"
  },
  "creditsUsed": 123,
  "requestId": "<string>"
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

POST

web

crawl

start

Crawl a website in the background

curl --request POST \
  --url https://api.stophy.dev/v1/web/crawl/start \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "url": "<string>",
  "goal": "<string>",
  "includePaths": [],
  "excludePaths": [],
  "maxDepth": 3,
  "maxPages": 100,
  "entireSite": false,
  "includeSubdomains": false,
  "onlyMainContent": true,
  "webhook": "<string>"
}
'

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({
    url: '<string>',
    goal: '<string>',
    includePaths: [],
    excludePaths: [],
    maxDepth: 3,
    maxPages: 100,
    entireSite: false,
    includeSubdomains: false,
    onlyMainContent: true,
    webhook: '<string>'
  })
};

fetch('https://api.stophy.dev/v1/web/crawl/start', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

import requests

url = "https://api.stophy.dev/v1/web/crawl/start"

payload = {
    "url": "<string>",
    "goal": "<string>",
    "includePaths": [],
    "excludePaths": [],
    "maxDepth": 3,
    "maxPages": 100,
    "entireSite": False,
    "includeSubdomains": False,
    "onlyMainContent": True,
    "webhook": "<string>"
}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

{
  "success": true,
  "data": {
    "status": "queued",
    "total": 4503599627370495,
    "id": "<string>"
  },
  "creditsUsed": 123,
  "requestId": "<string>"
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

Crawl a website in the background.

Request

url is required.

import { Stophy } from "stophy";

const stophy = new Stophy({ apiKey: process.env.STOPHY_API_KEY });
const result = await stophy.web.crawl.start({ url: "https://bun.com/docs", goal: "sqlite", maxPages: 4 });
console.log(result.data);

from stophy import Stophy

stophy = Stophy()  # reads STOPHY_API_KEY
result = stophy.web.crawl.start(url="https://bun.com/docs", goal="sqlite", max_pages=4)
print(result["data"])

curl -X POST https://api.stophy.dev/v1/web/crawl/start \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://bun.com/docs","goal":"sqlite","maxPages":4}'

stophy web crawl start https://bun.com/docs --goal sqlite --maxPages 4

Response

If successful, the response body contains the standard envelope. data contains id, status and total. The following example is a real response, truncated: each list shows one item and long strings are cut.

{
  "success": true,
  "data": {
    "id": "job_Qg9km6Ss5DAJktZgPYOUkA",
    "status": "queued",
    "total": 1
  },
  "creditsUsed": 0,
  "requestId": "2fe9dbad-fc48-4890-9751-0b24b596035b"
}

Field	Type	Description
`total`	integer	The total number of matching results, across all pages.

For every field, see the response schema on this page.

It answers at once with an id; ask Web web Crawl Status for the pages, or send webhook to be told as each one finishes.

Optional parameters

Parameter	Type	Description
`goal`	string	What you are looking for, such as sqlite performance. Pages about it are read first, and the crawl stops once new pages stop mentioning it.
`includePaths`	string[]	Read only pages whose path matches one of these, such as /blog/*. Accepts up to 20 values.
`excludePaths`	string[]	Skip pages whose path matches one of these, such as /tag/*. Accepts up to 20 values.
`maxDepth`	integer	The number of links away from the start page to go. Must be between 1 and 10. If unset, defaults to `3`.
`maxPages`	integer	Stop after reading this many pages in total, across every call. Must be between 1 and 1000. If unset, defaults to `100`.
`entireSite`	boolean	Read the whole site, not only pages under the start page’s path. If unset, defaults to `false`.
`includeSubdomains`	boolean	Also read pages on subdomains, such as docs.bun.com for bun.com. If unset, defaults to `false`.
`onlyMainContent`	boolean	Keep only each page’s main content, without menus, headers and footers. If unset, defaults to `true`.
`webhook`	string	A public address that gets a JSON POST when the job starts, for each finished page, and when the job ends.

Billing

A successful request consumes 1 credit a page for most pages, 2 or 3 credits when a page takes more work to load, and 15 credits for the hardest pages to reach. A page that cannot be read costs nothing. Each page is charged when it finishes. Starting the job is free, up to 15000 credits. Requests that fail, or that return no items, are not billed.

Pagination

This method is not paginated. A single request returns the complete result.

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Query Parameters

maxTextLength

integer

Cut every text value in the response to this many characters. A value that was cut ends with …, and a page whose text was cut has isTruncated set to true. A screenshot is never cut. Leave it out to get the full text.

Required range: x >= 1

Body

application/json

url

string

required

The page to start from, such as https://bun.com/docs.

goal

string

What you are looking for, such as sqlite performance. Pages about it are read first, and the crawl stops once new pages stop mentioning it.

Required string length: 1 - 300

includePaths

string[]

Read only pages whose path matches one of these, such as /blog/*.

Maximum array length: 20

Required string length: 1 - 200

excludePaths

string[]

Skip pages whose path matches one of these, such as /tag/*.

Maximum array length: 20

Required string length: 1 - 200

maxDepth

integer

default:3

How many links away from the start page to go.

Required range: 1 <= x <= 10

maxPages

integer

default:100

Stop after reading this many pages in total, across every call.

Required range: 1 <= x <= 1000

entireSite

boolean

default:false

Read the whole site, not only pages under the start page's path.

includeSubdomains

boolean

default:false

Also read pages on subdomains, such as docs.bun.com for bun.com.

onlyMainContent

boolean

default:true

Keep only each page's main content, without menus, headers and footers.

webhook

string

A public address that gets a JSON POST when the job starts, for each finished page, and when the job ends.

Response

The data, and the credits this call used.

success

any

required

data

object

required

Show child attributes

creditsUsed

integer

required

requestId

string

required

Using the API

Catalog

Account

Search and AI answers

Video

Social

Places and travel

Jobs

Shopping

Apps

Real estate

Ads

Web web Crawl Start API

Request

Response

Optional parameters

Billing

Authorizations

Query Parameters

Body

Response

​Request

​Response

​Optional parameters

​Billing

​Pagination

Authorizations

Query Parameters

Body

Response

Request

Response

Optional parameters

Billing

Pagination