> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# Web web Batch API
> Scrape up to 100 pages in the background. Call the Web web Batch API and get JSON back in one request. From 0 credits per call. Failed calls cost nothing.
Scrape up to 100 pages in the background.
## Request
`urls` is required.
```typescript TypeScript icon="https://mintlify.s3.us-west-1.amazonaws.com/sirathic/images/icons/typescript.svg" theme={null} theme={null}
import { Stophy } from "stophy";
const stophy = new Stophy({ apiKey: process.env.STOPHY_API_KEY });
const result = await stophy.web.batch({ urls: ["https://bun.com/docs", "https://bun.com/blog"] });
console.log(result.data);
```
```python Python icon="python" theme={null} theme={null}
from stophy import Stophy
stophy = Stophy() # reads STOPHY_API_KEY
result = stophy.web.batch(urls=["https://bun.com/docs", "https://bun.com/blog"])
print(result["data"])
```
```bash cURL icon="terminal" theme={null} theme={null}
curl -X POST https://api.stophy.dev/v1/web/batch \
-H "Authorization: Bearer $STOPHY_API_KEY" \
-H "content-type: application/json" \
-d '{"urls":["https://bun.com/docs","https://bun.com/blog"]}'
```
```bash CLI icon="square-terminal" theme={null} theme={null}
stophy web batch --urls https://bun.com/docs,https://bun.com/blog
```
## Response
If successful, the response body contains the standard envelope. `data` contains the batch's id and status, to follow with `web.batch.status`.
The following example is a real response, truncated: each list shows one item and long strings are cut.
```json theme={null}
{
"success": true,
"data": {
"id": "job_BC0vf8IzmDUA4VeGy9M7tA",
"status": "queued",
"total": 2
},
"creditsUsed": 0,
"requestId": "2fe9dbad-fc48-4890-9751-0b24b596035b"
}
```
| Field | Type | Description |
| - | - | - |
| `total` | integer | The total number of matching results, across all pages. |
For every field, see the response schema on this page.
It answers at once with an id; ask [Web web Batch Status](/api-reference/endpoint/web-batch-status) for the pages, or send webhook to be told as each one finishes.
## Optional parameters
| Parameter | Type | Description |
| - | - | - |
| `webhook` | string | A public address that gets a JSON POST when the job starts, for each finished page, and when the job ends. |
| `formats` | object\[] | What to return: markdown (clean page text), html (the cleaned page), `rawHtml` (the page as sent), links (every link on the page) and screenshot (a JPEG of the visible page, as base64). Send `{ "type": "screenshot", "fullPage": true }` for the whole page. A screenshot costs 3 credits, or 15 on the hardest pages to reach. A PDF comes back as markdown only. Accepts up to 5 values. |
| `onlyMainContent` | boolean | Keep only the page's main content, without menus, headers, footers and ads. If unset, defaults to `true`. |
| `includeTags` | string\[] | Keep only the parts of the page that match these CSS selectors, such as article or .price. When set, `onlyMainContent` is not applied. Accepts up to 20 values. |
| `excludeTags` | string\[] | Remove the parts of the page that match these CSS selectors, such as .comments. Accepts up to 20 values. |
| `waitFor` | integer | Extra milliseconds to wait before reading, for pages that load their content late. To wait for an element, add a wait action with a selector. A read that waits costs 3 credits, or 15 on the hardest pages to reach. Must be between 0 and 10000. |
| `timeout` | integer | How long to keep trying, in milliseconds. Pages behind a bot check can need 10 to 30 seconds the first time. Must be between 10000 and 60000. If unset, defaults to `30000`. |
| `maxAge` | integer | Accept a stored answer up to this many milliseconds old. 0 always reads the page live. metadata.`cacheState` says which one you got. Must be between 0 and 172800000. If unset, defaults to `3600000`. |
| `location` | object | Where to read the page from. The default is the United States. |
| `actions` | object\[] | Steps to run on the page, in order, before reading it: `{ "type": "wait", "milliseconds": 500 }`, `{ "type": "wait", "selector": ".results" }`, `{ "type": "click", "selector": "button.more" }`, `{ "type": "write", "text": "tides" }`, `{ "type": "press", "key": "Enter" }` or `{ "type": "scroll", "direction": "down" }`. A read with actions costs 3 credits, or 15 on the hardest pages to reach. Accepts up to 20 values. |
| `maxChars` | integer | Cut the markdown at this many characters; `isTruncated` says when that happened. Must be between 500 and 1000000. |
## Billing
A successful request consumes 1 credit a page for most pages, 2 or 3 credits when a page takes more work to load, and 15 credits for the hardest pages to reach. A read with a screenshot, actions or waitFor costs 3 credits, or 15 on the hardest pages to reach. An answer from a stored copy costs 1 credit. A page that cannot be read costs nothing. Each page is charged when it finishes. Starting the job is free, up to 1500 credits. Requests that fail, or that return no items, are not billed.
## Pagination
This method is not paginated. A single request returns the complete result.
## OpenAPI
````yaml api-reference/openapi.json POST /v1/web/batch
openapi: 3.1.0
info:
title: Stophy API
version: '1'
description: >-
Every endpoint is a POST that takes its input as a JSON object body. Unknown
fields are rejected with 400.
Responses leave out empty values: null, empty strings, empty objects and
empty nested lists are not sent.
Text comes back whole. To get shorter responses, add
?maxTextLength= to the URL. Every text value longer than that is
cut and ends with …. A screenshot is never cut, and a page whose text was
cut has isTruncated set to true.
x-credits is what one call costs. Every call costs the same, whatever it
returns, unless an operation's description says it costs per 10 results, or
per second of audio. Then x-credits is the lowest price and x-credits-max
the most one call can cost. Where an operation takes limit, send it to
return fewer results and pay for fewer.
When a response has a cursor, send it back unchanged with the same input to
get the next page. When it has a page, send the next page number.
servers:
- url: https://api.stophy.dev
security:
- apiKey: []
paths:
/v1/web/batch:
post:
summary: Scrape up to 100 pages in the background
description: >-
Costs 1 credit a page for most pages, 2 or 3 credits when a page takes
more work to load, and 15 credits for the hardest pages to reach. A read
with a screenshot, actions or waitFor costs 3 credits, or 15 on the
hardest pages to reach. An answer from a stored copy costs 1 credit. A
page that cannot be read costs nothing. Each page is charged when it
finishes. Starting the job is free.
operationId: webBatch
parameters:
- $ref: '#/components/parameters/MaxTextLength'
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
urls:
minItems: 1
maxItems: 100
type: array
items:
type: string
description: >-
The pages or PDFs to read, up to 100. An address sent twice
is read once.
webhook:
description: >-
A public address that gets a JSON POST when the job starts,
for each finished page, and when the job ends.
type: string
formats:
description: >-
What to return: markdown (clean page text), html (the
cleaned page), rawHtml (the page as sent), links (every link
on the page) and screenshot (a JPEG of the visible page, as
base64). Send { "type": "screenshot", "fullPage": true } for
the whole page. A screenshot costs 3 credits, or 15 on the
hardest pages to reach. A PDF comes back as markdown only.
default:
- markdown
minItems: 1
maxItems: 5
type: array
items:
anyOf:
- type: string
enum:
- markdown
- html
- rawHtml
- links
- screenshot
- type: object
properties:
type:
type: string
const: screenshot
fullPage:
default: false
description: >-
Capture the whole page from top to bottom instead
of the visible part.
type: boolean
required:
- type
additionalProperties: false
onlyMainContent:
default: true
description: >-
Keep only the page's main content, without menus, headers,
footers and ads.
type: boolean
includeTags:
description: >-
Keep only the parts of the page that match these CSS
selectors, such as article or .price. When set,
onlyMainContent is not applied.
minItems: 1
maxItems: 20
type: array
items:
type: string
minLength: 1
maxLength: 200
excludeTags:
description: >-
Remove the parts of the page that match these CSS selectors,
such as .comments.
minItems: 1
maxItems: 20
type: array
items:
type: string
minLength: 1
maxLength: 200
waitFor:
description: >-
Extra milliseconds to wait before reading, for pages that
load their content late. To wait for an element, add a wait
action with a selector. A read that waits costs 3 credits,
or 15 on the hardest pages to reach.
type: integer
minimum: 0
maximum: 10000
timeout:
default: 30000
description: >-
How long to keep trying, in milliseconds. Pages behind a bot
check can need 10 to 30 seconds the first time.
type: integer
minimum: 10000
maximum: 60000
maxAge:
default: 3600000
description: >-
Accept a stored answer up to this many milliseconds old. 0
always reads the page live. metadata.cacheState says which
one you got.
type: integer
minimum: 0
maximum: 172800000
location:
description: >-
Where to read the page from. The default is the United
States.
type: object
properties:
country:
description: >-
Two-letter country code to read the page from, such as
DE or JP.
type: string
required:
- country
additionalProperties: false
actions:
description: >-
Steps to run on the page, in order, before reading it: {
"type": "wait", "milliseconds": 500 }, { "type": "wait",
"selector": ".results" }, { "type": "click", "selector":
"button.more" }, { "type": "write", "text": "tides" }, {
"type": "press", "key": "Enter" } or { "type": "scroll",
"direction": "down" }. A read with actions costs 3 credits,
or 15 on the hardest pages to reach.
minItems: 1
maxItems: 20
type: array
items:
anyOf:
- type: object
properties:
type:
type: string
const: wait
milliseconds:
type: integer
minimum: 1
maximum: 10000
required:
- type
- milliseconds
additionalProperties: false
- type: object
properties:
type:
type: string
const: wait
selector:
type: string
minLength: 1
maxLength: 200
required:
- type
- selector
additionalProperties: false
- type: object
properties:
type:
type: string
const: click
selector:
type: string
minLength: 1
maxLength: 200
required:
- type
- selector
additionalProperties: false
- type: object
properties:
type:
type: string
const: write
text:
type: string
minLength: 1
maxLength: 2000
required:
- type
- text
additionalProperties: false
- type: object
properties:
type:
type: string
const: press
key:
type: string
minLength: 1
maxLength: 40
required:
- type
- key
additionalProperties: false
- type: object
properties:
type:
type: string
const: scroll
direction:
type: string
enum:
- up
- down
required:
- type
- direction
additionalProperties: false
maxChars:
description: >-
Cut the markdown at this many characters; isTruncated says
when that happened.
type: integer
minimum: 500
maximum: 1000000
required:
- urls
additionalProperties: false
responses:
'200':
description: The data, and the credits this call used.
headers:
X-Request-ID:
$ref: '#/components/headers/RequestId'
x-credits-used:
$ref: '#/components/headers/CreditsUsed'
X-RateLimit-Limit:
$ref: '#/components/headers/RateLimitLimit'
X-RateLimit-Remaining:
$ref: '#/components/headers/RateLimitRemaining'
X-RateLimit-Reset:
$ref: '#/components/headers/RateLimitReset'
x-cache:
$ref: '#/components/headers/Cache'
x-cache-age:
$ref: '#/components/headers/CacheAge'
content:
application/json:
schema:
type: object
required:
- success
- data
- creditsUsed
- requestId
properties:
success:
const: true
data:
type: object
additionalProperties: false
properties:
id:
type: string
status:
type: string
enum:
- queued
- running
- completed
- cancelled
- failed
total:
type: integer
minimum: 0
maximum: 9007199254740991
required:
- status
- total
creditsUsed:
type: integer
requestId:
type: string
4XX:
$ref: '#/components/responses/Error'
5XX:
$ref: '#/components/responses/Error'
components:
parameters:
MaxTextLength:
name: maxTextLength
in: query
required: false
description: >-
Cut every text value in the response to this many characters. A value
that was cut ends with …, and a page whose text was cut has isTruncated
set to true. A screenshot is never cut. Leave it out to get the full
text.
schema:
type: integer
minimum: 1
headers:
RequestId:
description: The request id. Quote it when you report a problem.
schema:
type: string
CreditsUsed:
description: Credits this call used.
schema:
type: integer
RateLimitLimit:
description: Requests allowed in the current window.
schema:
type: integer
RateLimitRemaining:
description: Requests left in the current window.
schema:
type: integer
RateLimitReset:
description: Unix time in seconds when the current window resets.
schema:
type: integer
Cache:
description: hit if this result came from the cache, miss if it was fetched live.
schema:
type: string
CacheAge:
description: Seconds since the returned result was written to the cache.
schema:
type: integer
RetryAfter:
description: Seconds to wait before retrying.
schema:
type: integer
responses:
Error:
description: The request failed. retryable says whether trying again can help.
headers:
X-Request-ID:
$ref: '#/components/headers/RequestId'
Retry-After:
$ref: '#/components/headers/RetryAfter'
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
schemas:
Error:
type: object
required:
- success
- error
properties:
success:
const: false
error:
type: object
required:
- code
- message
- retryable
- requestId
properties:
code:
type: string
enum:
- cliSessionCompleted
- cliSessionExists
- cliSessionNotFound
- forbidden
- insufficientCredits
- internalError
- inviteClaimed
- inviteDisabled
- inviteExpired
- inviteUsedUp
- invalidCliSession
- invalidCliVerifier
- invalidRequest
- notFound
- rateLimited
- sourceChanged
- sourceRefused
- sourceTimeout
- sourceUnavailable
- tooManyInFlight
- unauthorized
- unsupportedMediaType
message:
type: string
retryable:
type: boolean
retryAfterSeconds:
type: integer
requestId:
type: string
requiredCredits:
type: number
availableCredits:
type: number
billingUrl:
type: string
const: https://stophy.dev/billing
securitySchemes:
apiKey:
type: http
scheme: bearer
````
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.