Page through results

Two things in Stophy come back in pages: parsers that read lists, and the results of a crawl or batch scrape. Paging needs an API key.

Parsers

A parser that reads a list returns cursor or page in data.json when there is more. Send the same request again with that value in the format’s options. This asks for the second page of a Reddit search. The first response had "cursor": "MjQ=".

curl -X POST https://api.stophy.dev/v1/scrape \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://www.reddit.com/search/?q=web+scraping&sort=top&t=year","formats":[{"type":"json","parser":"reddit-search","options":{"cursor":"MjQ="}}]}'

It returned the next 25 posts and "cursor": "NDk=" for the page after. Keep the URL and the rest of the format the same between pages. The last page has no cursor. A parser that takes page works the same way: send "options": {"page": 2}, then 3, until the results are empty. Each parser’s options are on its parser page and in the parser list. Each page is a new call and costs the parser’s price.

const format = { type: "json", parser: "reddit-search" };
const url = "https://www.reddit.com/search/?q=web+scraping&sort=top&t=year";
const posts = [];
let cursor: string | undefined;
for (let page = 0; page < 3; page += 1) {
  const response = await fetch("https://api.stophy.dev/v1/scrape", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.STOPHY_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url, formats: [cursor === undefined ? format : { ...format, options: { cursor } }] }),
  });
  const { data } = await response.json();
  posts.push(...data.json.results);
  cursor = data.json.cursor;
  if (cursor === undefined) break;
}

import os

import requests

fmt = {"type": "json", "parser": "reddit-search"}
url = "https://www.reddit.com/search/?q=web+scraping&sort=top&t=year"
posts = []
cursor = None
for _ in range(3):
    options = {"options": {"cursor": cursor}} if cursor else {}
    response = requests.post(
        "https://api.stophy.dev/v1/scrape",
        headers={"Authorization": f"Bearer {os.environ['STOPHY_API_KEY']}"},
        json={"url": url, "formats": [{**fmt, **options}]},
    )
    data = response.json()["data"]
    posts.extend(data["json"]["results"])
    cursor = data["json"].get("cursor")
    if not cursor:
        break

Crawl and batch scrape

Crawl and batch scrape answer at once with an id. Read the pages with crawl status or batch scrape status. Reading status is free, and it needs the same key. A status answer holds up to 100 finished pages or 2 MB, whichever comes first. This is separate from the limit you sent to crawl, which is how many pages the crawl reads in all. A crawl with limit 30 is usually read in one status call once it is done, and one with limit 250 in three. While the job runs, status is scraping. total is the number of pages found so far, and it grows. completed is how many are finished. data holds the finished pages. next appears only when more finished pages are waiting, so it never leads to an empty data.

Poll the status URL every few seconds until status is completed, failed or cancelled.
Follow next until the answer has none. Each next is a full URL on api.stophy.dev. Call it with the same key.

{ "status": "completed", "total": 250, "completed": 250, "next": "https://api.stophy.dev/v1/crawl/job_3a2Bj6qUFGZFq-y-1MWdaA?cursor=-IEUnTHVpage_100", "data": [ ... ] }
{ "status": "completed", "total": 250, "completed": 250, "next": "https://api.stophy.dev/v1/crawl/job_3a2Bj6qUFGZFq-y-1MWdaA?cursor=1owY_-gJpage_200", "data": [ ... ] }
{ "status": "completed", "total": 250, "completed": 250, "data": [ ... ] }

const headers = { Authorization: `Bearer ${process.env.STOPHY_API_KEY}` };
const url = `https://api.stophy.dev/v1/crawl/${id}`;
const over = new Set(["completed", "failed", "cancelled"]);

let status = await (await fetch(url, { headers })).json();
while (!over.has(status.status)) {
  await new Promise((resolve) => setTimeout(resolve, 2000));
  status = await (await fetch(url, { headers })).json();
}

const pages = [...status.data];
let next: string | undefined = status.next;
while (next !== undefined) {
  const more = await (await fetch(next, { headers })).json();
  pages.push(...more.data);
  next = more.next;
}

import os
import time

import requests

headers = {"Authorization": f"Bearer {os.environ['STOPHY_API_KEY']}"}
url = f"https://api.stophy.dev/v1/crawl/{job_id}"

status = requests.get(url, headers=headers).json()
while status["status"] not in ("completed", "failed", "cancelled"):
    time.sleep(2)
    status = requests.get(url, headers=headers).json()

pages = list(status["data"])
next_url = status.get("next")
while next_url:
    more = requests.get(next_url, headers=headers).json()
    pages.extend(more["data"])
    next_url = more.get("next")

Batch scrape works the same way at https://api.stophy.dev/v1/batch/scrape/<id>. A crawl and its pages are kept for 24 hours.

Get started

Parsers

Guides

Plans and billing

Page through results

Parsers

Crawl and batch scrape

​Parsers

​Crawl and batch scrape

Parsers

Crawl and batch scrape