Handle errors and retries | Stophy

Decide what to do

retryAfterSeconds in the body and the Retry-After header are the same number of seconds, so read either one. When neither is there, back off: wait 0.5 s, then 1 s, then 2 s. If the wait is longer than a minute, stop retrying. A freeLimit wait can be hours, so get a key instead.

Code	Status	Retry	Wait	Most tries
`invalidRequest`, `unsupportedMediaType`, `payloadTooLarge`	400, 413, 415	No	Fix the request, or read `error`.	1
`apiKeyRequired`, `unauthorized`	401	No	Get a key at `signupUrl`, or call `signup`.	1
`insufficientCredits`	402	No	Top up at stophy.dev/billing. The `x-required-credits` and `x-available-credits` headers say by how much.	1
`forbidden`, `siteNotRead`, `notFound`, `jobFinished`	403, 404, 409	No	Change the request. `siteNotRead` needs a parser.	1
`freeLimit`, `rateLimited`	429	Yes	`retryAfterSeconds`, else back off.	3
`internalError`, `sourceUnavailable`, `sourceRefused`, `sourceTimeout`	500, 503, 504	Yes	`retryAfterSeconds`, else back off.	3
`sourceChanged`	502	No	Retrying will not help yet.	1

After the last try, stop and log the X-Request-ID header. A failed call costs nothing, so a retry costs nothing extra. Errors lists every code.

A retry loop

const RETRY = new Set(["freeLimit", "rateLimited", "internalError", "sourceUnavailable", "sourceRefused", "sourceTimeout"]);

async function call(path: string, input: object, attempts = 3) {
  for (let attempt = 0; ; attempt++) {
    const response = await fetch(`https://api.stophy.dev${path}`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.STOPHY_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(input),
      signal: AbortSignal.timeout(65_000),
    });
    const body = await response.json();
    if (response.ok) return body;
    if (!RETRY.has(body.code) || attempt + 1 >= attempts) {
      throw new Error(`${body.code}: ${body.error} (${response.headers.get("x-request-id")})`);
    }
    const seconds = body.retryAfterSeconds ?? 0.5 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, seconds * 1000));
  }
}

const page = await call("/v1/scrape", { url: "https://bun.com/docs/runtime/bunfig" });

import os
import time

import requests

RETRY = {"freeLimit", "rateLimited", "internalError", "sourceUnavailable", "sourceRefused", "sourceTimeout"}


def call(path, payload, attempts=3):
    for attempt in range(attempts):
        response = requests.post(
            f"https://api.stophy.dev{path}",
            headers={"Authorization": f"Bearer {os.environ['STOPHY_API_KEY']}"},
            json=payload,
            timeout=65,
        )
        body = response.json()
        if response.ok:
            return body
        if body["code"] not in RETRY or attempt + 1 >= attempts:
            raise RuntimeError(f"{body['code']}: {body['error']} ({response.headers.get('x-request-id')})")
        time.sleep(body.get("retryAfterSeconds", 0.5 * 2**attempt))


page = call("/v1/scrape", {"url": "https://bun.com/docs/runtime/bunfig"})

Timeouts

Most calls answer within 15 seconds. A slow page, a long PDF or a transcript can take up to a minute. Set your client timeout to at least 65 seconds. Crawl, batch scrape, extract and browser tasks answer at once with an id, and you read the result with their status calls.