Web web Scrape API

Scrape any public web page as markdown, HTML, links or a screenshot

curl --request POST \
  --url https://api.stophy.dev/v1/web/scrape \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "url": "https://bun.com/docs/runtime/bunfig"
}
'

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({url: 'https://bun.com/docs/runtime/bunfig'})
};

fetch('https://api.stophy.dev/v1/web/scrape', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

import requests

url = "https://api.stophy.dev/v1/web/scrape"

payload = { "url": "https://bun.com/docs/runtime/bunfig" }
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

{
  "success": true,
  "data": {
    "isTruncated": true,
    "metadata": {
      "statusCode": 0,
      "cacheState": "hit",
      "title": "<string>",
      "description": "<string>",
      "language": "<string>",
      "ogImage": "<string>",
      "sourceURL": "<string>",
      "contentType": "<string>",
      "cachedAt": "2023-11-07T05:31:56Z",
      "numPages": 0
    },
    "markdown": "<string>",
    "html": "<string>",
    "rawHtml": "<string>",
    "links": [
      "<string>"
    ],
    "screenshot": "<string>"
  },
  "creditsUsed": 123,
  "requestId": "<string>"
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

POST

web

scrape

Scrape any public web page as markdown, HTML, links or a screenshot

curl --request POST \
  --url https://api.stophy.dev/v1/web/scrape \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "url": "https://bun.com/docs/runtime/bunfig"
}
'

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({url: 'https://bun.com/docs/runtime/bunfig'})
};

fetch('https://api.stophy.dev/v1/web/scrape', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

import requests

url = "https://api.stophy.dev/v1/web/scrape"

payload = { "url": "https://bun.com/docs/runtime/bunfig" }
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

{
  "success": true,
  "data": {
    "isTruncated": true,
    "metadata": {
      "statusCode": 0,
      "cacheState": "hit",
      "title": "<string>",
      "description": "<string>",
      "language": "<string>",
      "ogImage": "<string>",
      "sourceURL": "<string>",
      "contentType": "<string>",
      "cachedAt": "2023-11-07T05:31:56Z",
      "numPages": 0
    },
    "markdown": "<string>",
    "html": "<string>",
    "rawHtml": "<string>",
    "links": [
      "<string>"
    ],
    "screenshot": "<string>"
  },
  "creditsUsed": 123,
  "requestId": "<string>"
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

{
  "success": false,
  "error": {
    "code": "cliSessionCompleted",
    "message": "<string>",
    "retryable": true,
    "requestId": "<string>",
    "retryAfterSeconds": 123,
    "requiredCredits": 123,
    "availableCredits": 123,
    "billingUrl": "https://stophy.dev/billing"
  }
}

Scrape any public web page as markdown, HTML, links or a screenshot.

Request

url is required.

This endpoint can be called without an API key, subject to per-address limits. Pagination requires a key.

import { Stophy } from "stophy";

const stophy = new Stophy({ apiKey: process.env.STOPHY_API_KEY });
const result = await stophy.web.scrape({ url: "https://bun.com/docs/runtime/bunfig" });
console.log(result.data);

from stophy import Stophy

stophy = Stophy()  # reads STOPHY_API_KEY
result = stophy.web.scrape(url="https://bun.com/docs/runtime/bunfig")
print(result["data"])

curl -X POST https://api.stophy.dev/v1/web/scrape \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://bun.com/docs/runtime/bunfig"}'

stophy web scrape https://bun.com/docs/runtime/bunfig

Response

If successful, the response body contains the standard envelope. data contains markdown, html, rawHtml, links, screenshot, isTruncated and metadata. The following example is a real response, truncated: each list shows one item and long strings are cut.

{
  "success": true,
  "data": {
    "markdown": "`bunfig.toml` is Bun's configuration file.\n\nBun relies on pre-existing configuration files like `package.json` and…",
    "isTruncated": false,
    "metadata": {
      "title": "bunfig.toml | Bun Docs",
      "description": "Configure Bun's behavior using its configuration file bunfig.toml",
      "language": "en",
      "ogImage": "https://bun.com/og/docs/runtime/bunfig.png",
      "sourceURL": "https://bun.com/docs/runtime/bunfig",
      "statusCode": 200,
      "contentType": "text/html",
      "cacheState": "miss"
    }
  },
  "creditsUsed": 1,
  "requestId": "2fe9dbad-fc48-4890-9751-0b24b596035b"
}

Social, video, shopping and jobs pages have their own endpoints with structured data.

Optional parameters

Parameter	Type	Description
`formats`	object[]	What to return: markdown (clean page text), html (the cleaned page), `rawHtml` (the page as sent), links (every link on the page) and screenshot (a JPEG of the visible page, as base64). Send `{ "type": "screenshot", "fullPage": true }` for the whole page. A screenshot costs 3 credits, or 15 on the hardest pages to reach. A PDF comes back as markdown only. Accepts up to 5 values.
`onlyMainContent`	boolean	Keep only the page’s main content, without menus, headers, footers and ads. If unset, defaults to `true`.
`includeTags`	string[]	Keep only the parts of the page that match these CSS selectors, such as article or .price. When set, `onlyMainContent` is not applied. Accepts up to 20 values.
`excludeTags`	string[]	Remove the parts of the page that match these CSS selectors, such as .comments. Accepts up to 20 values.
`waitFor`	integer	Extra milliseconds to wait before reading, for pages that load their content late. To wait for an element, add a wait action with a selector. A read that waits costs 3 credits, or 15 on the hardest pages to reach. Must be between 0 and 10000.
`timeout`	integer	How long to keep trying, in milliseconds. Pages behind a bot check can need 10 to 30 seconds the first time. Must be between 10000 and 60000. If unset, defaults to `30000`.
`maxAge`	integer	Accept a stored answer up to this many milliseconds old. 0 always reads the page live. metadata.`cacheState` says which one you got. Must be between 0 and 172800000. If unset, defaults to `3600000`.
`location`	object	Where to read the page from. The default is the United States.
`actions`	object[]	Steps to run on the page, in order, before reading it: `{ "type": "wait", "milliseconds": 500 }`, `{ "type": "wait", "selector": ".results" }`, `{ "type": "click", "selector": "button.more" }`, `{ "type": "write", "text": "tides" }`, `{ "type": "press", "key": "Enter" }` or `{ "type": "scroll", "direction": "down" }`. A read with actions costs 3 credits, or 15 on the hardest pages to reach. Accepts up to 20 values.
`maxChars`	integer	Cut the markdown at this many characters; `isTruncated` says when that happened. Must be between 500 and 1000000.

Billing

A successful request consumes 1 credit for most pages. 2 or 3 credits when a page takes more work to load, and 15 credits for the hardest pages to reach. A read with a screenshot, actions or waitFor costs 3 credits, or 15 on the hardest pages to reach. An answer from a stored copy costs 1 credit. A page that cannot be read costs nothing. creditsUsed shows what the call cost, up to 15 credits. Requests that fail, or that return no items, are not billed.

Pagination

This method is not paginated. A single request returns the complete result.

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Query Parameters

maxTextLength

integer

Cut every text value in the response to this many characters. A value that was cut ends with …, and a page whose text was cut has isTruncated set to true. A screenshot is never cut. Leave it out to get the full text.

Required range: x >= 1

Body

application/json

url

string

required

The page or PDF to read, such as https://bun.com/docs.

formats

(enum<string> | object)[]

What to return: markdown (clean page text), html (the cleaned page), rawHtml (the page as sent), links (every link on the page) and screenshot (a JPEG of the visible page, as base64). Send { "type": "screenshot", "fullPage": true } for the whole page. A screenshot costs 3 credits, or 15 on the hardest pages to reach. A PDF comes back as markdown only.

Required array length: 1 - 5 elements

Available options:

markdown,

html,

rawHtml,

links,

screenshot

onlyMainContent

boolean

default:true

Keep only the page's main content, without menus, headers, footers and ads.

includeTags

string[]

Keep only the parts of the page that match these CSS selectors, such as article or .price. When set, onlyMainContent is not applied.

Required array length: 1 - 20 elements

Required string length: 1 - 200

excludeTags

string[]

Remove the parts of the page that match these CSS selectors, such as .comments.

Required array length: 1 - 20 elements

Required string length: 1 - 200

waitFor

integer

Extra milliseconds to wait before reading, for pages that load their content late. To wait for an element, add a wait action with a selector. A read that waits costs 3 credits, or 15 on the hardest pages to reach.

Required range: 0 <= x <= 10000

timeout

integer

default:30000

How long to keep trying, in milliseconds. Pages behind a bot check can need 10 to 30 seconds the first time.

Required range: 10000 <= x <= 60000

maxAge

integer

default:3600000

Accept a stored answer up to this many milliseconds old. 0 always reads the page live. metadata.cacheState says which one you got.

Required range: 0 <= x <= 172800000

location

object

Where to read the page from. The default is the United States.

Show child attributes

actions

object[]

Steps to run on the page, in order, before reading it: { "type": "wait", "milliseconds": 500 }, { "type": "wait", "selector": ".results" }, { "type": "click", "selector": "button.more" }, { "type": "write", "text": "tides" }, { "type": "press", "key": "Enter" } or { "type": "scroll", "direction": "down" }. A read with actions costs 3 credits, or 15 on the hardest pages to reach.

Required array length: 1 - 20 elements

Show child attributes

maxChars

integer

Cut the markdown at this many characters; isTruncated says when that happened.

Required range: 500 <= x <= 1000000

Response

The data, and the credits this call used.

success

any

required

data

object

required

Show child attributes

creditsUsed

integer

required

requestId

string

required

Using the API

Catalog

Account

Search and AI answers

Video

Social

Places and travel

Jobs

Shopping

Apps

Real estate

Ads

Web web Scrape API

Request

Response

Optional parameters

Billing

Authorizations

Query Parameters

Body

Response

​Request

​Response

​Optional parameters

​Billing

​Pagination

Authorizations

Query Parameters

Body

Response

Request

Response

Optional parameters

Billing

Pagination