> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# Web web Scrape API
> Scrape any public web page as markdown, HTML, links or a screenshot. Call the Web web Scrape API for JSON. From 1 credit per call.
Scrape any public web page as markdown, HTML, links or a screenshot.
## Request
`url` is required.
This endpoint can be called without an API key, subject to per-address limits. Pagination requires a key.
```typescript TypeScript icon="https://mintlify.s3.us-west-1.amazonaws.com/sirathic/images/icons/typescript.svg" theme={null} theme={null}
import { Stophy } from "stophy";
const stophy = new Stophy({ apiKey: process.env.STOPHY_API_KEY });
const result = await stophy.web.scrape({ url: "https://bun.com/docs/runtime/bunfig" });
console.log(result.data);
```
```python Python icon="python" theme={null} theme={null}
from stophy import Stophy
stophy = Stophy() # reads STOPHY_API_KEY
result = stophy.web.scrape(url="https://bun.com/docs/runtime/bunfig")
print(result["data"])
```
```bash cURL icon="terminal" theme={null} theme={null}
curl -X POST https://api.stophy.dev/v1/web/scrape \
-H "Authorization: Bearer $STOPHY_API_KEY" \
-H "content-type: application/json" \
-d '{"url":"https://bun.com/docs/runtime/bunfig"}'
```
```bash CLI icon="square-terminal" theme={null} theme={null}
stophy web scrape https://bun.com/docs/runtime/bunfig
```
## Response
If successful, the response body contains the standard envelope. `data` contains the page in the formats requested, as markdown by default, with its metadata.
The following example is a real response, truncated: each list shows one item and long strings are cut.
```json theme={null}
{
"success": true,
"data": {
"markdown": "`bunfig.toml` is Bun's configuration file.\n\nBun relies on pre-existing configuration files like `package.json` and…",
"isTruncated": false,
"metadata": {
"title": "bunfig.toml | Bun Docs",
"description": "Configure Bun's behavior using its configuration file bunfig.toml",
"language": "en",
"ogImage": "https://bun.com/og/docs/runtime/bunfig.png",
"sourceURL": "https://bun.com/docs/runtime/bunfig",
"statusCode": 200,
"contentType": "text/html",
"cacheState": "miss"
}
},
"creditsUsed": 1,
"requestId": "2fe9dbad-fc48-4890-9751-0b24b596035b"
}
```
Social, video, shopping and jobs pages have their own endpoints with structured data.
## Optional parameters
| Parameter | Type | Description |
| - | - | - |
| `formats` | object\[] | What to return: markdown (clean page text), html (the cleaned page), `rawHtml` (the page as sent), links (every link on the page) and screenshot (a JPEG of the visible page, as base64). Send `{ "type": "screenshot", "fullPage": true }` for the whole page. A screenshot costs 3 credits, or 15 on the hardest pages to reach. A PDF comes back as markdown only. Accepts up to 5 values. |
| `onlyMainContent` | boolean | Keep only the page's main content, without menus, headers, footers and ads. If unset, defaults to `true`. |
| `includeTags` | string\[] | Keep only the parts of the page that match these CSS selectors, such as article or .price. When set, `onlyMainContent` is not applied. Accepts up to 20 values. |
| `excludeTags` | string\[] | Remove the parts of the page that match these CSS selectors, such as .comments. Accepts up to 20 values. |
| `waitFor` | integer | Extra milliseconds to wait before reading, for pages that load their content late. To wait for an element, add a wait action with a selector. A read that waits costs 3 credits, or 15 on the hardest pages to reach. Must be between 0 and 10000. |
| `timeout` | integer | How long to keep trying, in milliseconds. Pages behind a bot check can need 10 to 30 seconds the first time. Must be between 10000 and 60000. If unset, defaults to `30000`. |
| `maxAge` | integer | Accept a stored answer up to this many milliseconds old. 0 always reads the page live. metadata.`cacheState` says which one you got. Must be between 0 and 172800000. If unset, defaults to `3600000`. |
| `location` | object | Where to read the page from. The default is the United States. |
| `actions` | object\[] | Steps to run on the page, in order, before reading it: `{ "type": "wait", "milliseconds": 500 }`, `{ "type": "wait", "selector": ".results" }`, `{ "type": "click", "selector": "button.more" }`, `{ "type": "write", "text": "tides" }`, `{ "type": "press", "key": "Enter" }` or `{ "type": "scroll", "direction": "down" }`. A read with actions costs 3 credits, or 15 on the hardest pages to reach. Accepts up to 20 values. |
| `maxChars` | integer | Cut the markdown at this many characters; `isTruncated` says when that happened. Must be between 500 and 1000000. |
## Billing
A successful request consumes 1 credit for most pages. 2 or 3 credits when a page takes more work to load, and 15 credits for the hardest pages to reach. A read with a screenshot, actions or waitFor costs 3 credits, or 15 on the hardest pages to reach. An answer from a stored copy costs 1 credit. A page that cannot be read costs nothing. creditsUsed shows what the call cost, up to 15 credits. Requests that fail, or that return no items, are not billed.
## Pagination
This method is not paginated. A single request returns the complete result.
## OpenAPI
````yaml api-reference/openapi.json POST /v1/web/scrape
openapi: 3.1.0
info:
title: Stophy API
version: '1'
description: >-
Every endpoint is a POST that takes its input as a JSON object body. Unknown
fields are rejected with 400.
Responses leave out empty values: null, empty strings, empty objects and
empty nested lists are not sent.
Text comes back whole. To get shorter responses, add
?maxTextLength= to the URL. Every text value longer than that is
cut and ends with …. A screenshot is never cut, and a page whose text was
cut has isTruncated set to true.
x-credits is what one call costs. Every call costs the same, whatever it
returns, unless an operation's description says it costs per 10 results, or
per second of audio. Then x-credits is the lowest price and x-credits-max
the most one call can cost. Where an operation takes limit, send it to
return fewer results and pay for fewer.
When a response has a cursor, send it back unchanged with the same input to
get the next page. When it has a page, send the next page number.
servers:
- url: https://api.stophy.dev
security:
- apiKey: []
paths:
/v1/web/scrape:
post:
summary: Scrape any public web page as markdown, HTML, links or a screenshot
description: >-
Costs 1 credit for most pages. 2 or 3 credits when a page takes more
work to load, and 15 credits for the hardest pages to reach. A read with
a screenshot, actions or waitFor costs 3 credits, or 15 on the hardest
pages to reach. An answer from a stored copy costs 1 credit. A page that
cannot be read costs nothing. creditsUsed shows what the call cost.
operationId: webScrape
parameters:
- $ref: '#/components/parameters/MaxTextLength'
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
url:
description: The page or PDF to read, such as https://bun.com/docs.
type: string
formats:
description: >-
What to return: markdown (clean page text), html (the
cleaned page), rawHtml (the page as sent), links (every link
on the page) and screenshot (a JPEG of the visible page, as
base64). Send { "type": "screenshot", "fullPage": true } for
the whole page. A screenshot costs 3 credits, or 15 on the
hardest pages to reach. A PDF comes back as markdown only.
default:
- markdown
minItems: 1
maxItems: 5
type: array
items:
anyOf:
- type: string
enum:
- markdown
- html
- rawHtml
- links
- screenshot
- type: object
properties:
type:
type: string
const: screenshot
fullPage:
default: false
description: >-
Capture the whole page from top to bottom instead
of the visible part.
type: boolean
required:
- type
additionalProperties: false
onlyMainContent:
default: true
description: >-
Keep only the page's main content, without menus, headers,
footers and ads.
type: boolean
includeTags:
description: >-
Keep only the parts of the page that match these CSS
selectors, such as article or .price. When set,
onlyMainContent is not applied.
minItems: 1
maxItems: 20
type: array
items:
type: string
minLength: 1
maxLength: 200
excludeTags:
description: >-
Remove the parts of the page that match these CSS selectors,
such as .comments.
minItems: 1
maxItems: 20
type: array
items:
type: string
minLength: 1
maxLength: 200
waitFor:
description: >-
Extra milliseconds to wait before reading, for pages that
load their content late. To wait for an element, add a wait
action with a selector. A read that waits costs 3 credits,
or 15 on the hardest pages to reach.
type: integer
minimum: 0
maximum: 10000
timeout:
default: 30000
description: >-
How long to keep trying, in milliseconds. Pages behind a bot
check can need 10 to 30 seconds the first time.
type: integer
minimum: 10000
maximum: 60000
maxAge:
default: 3600000
description: >-
Accept a stored answer up to this many milliseconds old. 0
always reads the page live. metadata.cacheState says which
one you got.
type: integer
minimum: 0
maximum: 172800000
location:
description: >-
Where to read the page from. The default is the United
States.
type: object
properties:
country:
description: >-
Two-letter country code to read the page from, such as
DE or JP.
type: string
required:
- country
additionalProperties: false
actions:
description: >-
Steps to run on the page, in order, before reading it: {
"type": "wait", "milliseconds": 500 }, { "type": "wait",
"selector": ".results" }, { "type": "click", "selector":
"button.more" }, { "type": "write", "text": "tides" }, {
"type": "press", "key": "Enter" } or { "type": "scroll",
"direction": "down" }. A read with actions costs 3 credits,
or 15 on the hardest pages to reach.
minItems: 1
maxItems: 20
type: array
items:
anyOf:
- type: object
properties:
type:
type: string
const: wait
milliseconds:
type: integer
minimum: 1
maximum: 10000
required:
- type
- milliseconds
additionalProperties: false
- type: object
properties:
type:
type: string
const: wait
selector:
type: string
minLength: 1
maxLength: 200
required:
- type
- selector
additionalProperties: false
- type: object
properties:
type:
type: string
const: click
selector:
type: string
minLength: 1
maxLength: 200
required:
- type
- selector
additionalProperties: false
- type: object
properties:
type:
type: string
const: write
text:
type: string
minLength: 1
maxLength: 2000
required:
- type
- text
additionalProperties: false
- type: object
properties:
type:
type: string
const: press
key:
type: string
minLength: 1
maxLength: 40
required:
- type
- key
additionalProperties: false
- type: object
properties:
type:
type: string
const: scroll
direction:
type: string
enum:
- up
- down
required:
- type
- direction
additionalProperties: false
maxChars:
description: >-
Cut the markdown at this many characters; isTruncated says
when that happened.
type: integer
minimum: 500
maximum: 1000000
required:
- url
additionalProperties: false
example:
url: https://bun.com/docs/runtime/bunfig
responses:
'200':
description: The data, and the credits this call used.
headers:
X-Request-ID:
$ref: '#/components/headers/RequestId'
x-credits-used:
$ref: '#/components/headers/CreditsUsed'
X-RateLimit-Limit:
$ref: '#/components/headers/RateLimitLimit'
X-RateLimit-Remaining:
$ref: '#/components/headers/RateLimitRemaining'
X-RateLimit-Reset:
$ref: '#/components/headers/RateLimitReset'
x-cache:
$ref: '#/components/headers/Cache'
x-cache-age:
$ref: '#/components/headers/CacheAge'
content:
application/json:
schema:
type: object
required:
- success
- data
- creditsUsed
- requestId
properties:
success:
const: true
data:
type: object
additionalProperties: false
properties:
markdown:
type: string
html:
type: string
rawHtml:
type: string
links:
type: array
items:
type: string
screenshot:
type: string
isTruncated:
type: boolean
metadata:
type: object
additionalProperties: false
properties:
title:
type: string
description:
type: string
language:
type: string
ogImage:
type: string
sourceURL:
type: string
statusCode:
type: integer
minimum: -9007199254740991
maximum: 9007199254740991
contentType:
type: string
cacheState:
type: string
enum:
- hit
- miss
cachedAt:
type: string
format: date-time
pattern: >-
^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z))$
numPages:
type: integer
minimum: -9007199254740991
maximum: 9007199254740991
required:
- statusCode
- cacheState
required:
- isTruncated
- metadata
creditsUsed:
type: integer
requestId:
type: string
4XX:
$ref: '#/components/responses/Error'
5XX:
$ref: '#/components/responses/Error'
security:
- apiKey: []
components:
parameters:
MaxTextLength:
name: maxTextLength
in: query
required: false
description: >-
Cut every text value in the response to this many characters. A value
that was cut ends with …, and a page whose text was cut has isTruncated
set to true. A screenshot is never cut. Leave it out to get the full
text.
schema:
type: integer
minimum: 1
headers:
RequestId:
description: The request id. Quote it when you report a problem.
schema:
type: string
CreditsUsed:
description: Credits this call used.
schema:
type: integer
RateLimitLimit:
description: Requests allowed in the current window.
schema:
type: integer
RateLimitRemaining:
description: Requests left in the current window.
schema:
type: integer
RateLimitReset:
description: Unix time in seconds when the current window resets.
schema:
type: integer
Cache:
description: hit if this result came from the cache, miss if it was fetched live.
schema:
type: string
CacheAge:
description: Seconds since the returned result was written to the cache.
schema:
type: integer
RetryAfter:
description: Seconds to wait before retrying.
schema:
type: integer
responses:
Error:
description: The request failed. retryable says whether trying again can help.
headers:
X-Request-ID:
$ref: '#/components/headers/RequestId'
Retry-After:
$ref: '#/components/headers/RetryAfter'
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
schemas:
Error:
type: object
required:
- success
- error
properties:
success:
const: false
error:
type: object
required:
- code
- message
- retryable
- requestId
properties:
code:
type: string
enum:
- cliSessionCompleted
- cliSessionExists
- cliSessionNotFound
- forbidden
- insufficientCredits
- internalError
- inviteClaimed
- inviteDisabled
- inviteExpired
- inviteUsedUp
- invalidCliSession
- invalidCliVerifier
- invalidRequest
- notFound
- rateLimited
- sourceChanged
- sourceRefused
- sourceTimeout
- sourceUnavailable
- tooManyInFlight
- unauthorized
- unsupportedMediaType
message:
type: string
retryable:
type: boolean
retryAfterSeconds:
type: integer
requestId:
type: string
requiredCredits:
type: number
availableCredits:
type: number
billingUrl:
type: string
const: https://stophy.dev/billing
securitySchemes:
apiKey:
type: http
scheme: bearer
````
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.