Skip to main content
You have a list of pages and want a table: one row per page, the same columns every time. You describe the columns as a JSON schema, a batch reads every page in the background, and you flatten the answers into rows. You need an API key in STOPHY_API_KEY. See the Quickstart. Each page costs $0.004 with a schema. The run on this page read 3 pages and cost $0.012.

1. Describe the row

The schema is one row. Mark the fields every page has as required, and leave the others optional, so a page that lacks one still returns a row. description tells the model the format you want.
See JSON for what a schema can hold.

2. Submit the batch

A batch reads a list of URLs in the background. It answers at once with an id. Send formats with the json format and your schema.
invalidURLs lists addresses that were not public pages. They are left out and cost nothing. Starting a batch is free, and each page is charged when it finishes.

3. Read the answers

Call url until status is completed. This batch took 6 seconds. Each page in data holds its object in json, and its address in metadata.url. This is the real answer, cut.
The second page has no topics. The field was optional, and the page lists none. Your code has to expect that. creditsUsed is the charge in steps of $0.002, so 6 is $0.012. An answer holds up to 100 pages or 2 MB, so a long batch comes back in parts. Follow next until it is missing. errors lists the pages that could not be read, and a failed page costs nothing. See Read a crawl, which a batch works like.

4. Write the rows

This function waits for completed, follows next, turns each json into a flat row with its url, fills a missing field with an empty string, and writes a CSV.
On the batch above it returned this CSV. The third page has an empty topics cell.

What to check

  • completed plus failed equals total, and your row count equals completed.
  • Open three pages and compare each cell with the page. The model fills a field from the page text, so a field the page does not state can come back wrong or empty.
  • required fields are never empty. Count the empty cells of each optional column, and decide if the column is worth keeping.
  • Dates are in one format. If not, tighten the description and read a few pages again.
  • Store url and the time you read it with each row, so you can tell how old a row is.

What it costs

1,000 rows cost $4. Test the schema on 3 to 5 pages before you send the list. See Pricing.
  • JSON describes schemas and prompts for one page.
  • Batches covers submitting, reading and cancelling a batch.
  • Extract fills a schema across several pages and checks each value against the page.
  • Monitors keep a dataset fresh by telling you which pages changed.