STOPHY_API_KEY. See the Quickstart. Each page costs $0.004 with a schema. The run on this page read 3 pages and cost $0.012.
1. Describe the row
The schema is one row. Mark the fields every page has asrequired, and leave the others optional, so a page that lacks one still returns a row. description tells the model the format you want.
2. Submit the batch
A batch reads a list of URLs in the background. It answers at once with anid. Send formats with the json format and your schema.
invalidURLs lists addresses that were not public pages. They are left out and cost nothing. Starting a batch is free, and each page is charged when it finishes.
3. Read the answers
Callurl until status is completed. This batch took 6 seconds. Each page in data holds its object in json, and its address in metadata.url. This is the real answer, cut.
topics. The field was optional, and the page lists none. Your code has to expect that. creditsUsed is the charge in steps of $0.002, so 6 is $0.012.
An answer holds up to 100 pages or 2 MB, so a long batch comes back in parts. Follow next until it is missing. errors lists the pages that could not be read, and a failed page costs nothing. See Read a crawl, which a batch works like.
4. Write the rows
This function waits forcompleted, follows next, turns each json into a flat row with its url, fills a missing field with an empty string, and writes a CSV.
topics cell.
What to check
completedplusfailedequalstotal, and your row count equalscompleted.- Open three pages and compare each cell with the page. The model fills a field from the page text, so a field the page does not state can come back wrong or empty.
requiredfields are never empty. Count the empty cells of each optional column, and decide if the column is worth keeping.- Dates are in one format. If not, tighten the
descriptionand read a few pages again. - Store
urland the time you read it with each row, so you can tell how old a row is.
What it costs
1,000 rows cost $4. Test the schema on 3 to 5 pages before you send the list. See Pricing.