STOPHY_API_KEY. See the Quickstart. The run on this page read 3 pages and cost $0.003.
1. Crawl the section
Crawl starts at one page, follows its links, and reads every page it finds. Starting is free, and each page is charged when it finishes. This request stops after 3 pages, which is enough to build and check the pipeline. Raiselimit for the whole section.
id.
includePaths and excludePaths. See Scope and limits. To read only chosen pages, list them with Map and send the ones you want to a batch.
2. Read every page
PollGET /v1/crawl/{id} until status is completed. Then read the pages with GET /v1/crawl/{id}/results, up to 100 at a time, and send cursor from data.cursor until it is missing. Both calls are free, and the pages are kept for 24 hours. See Read a crawl.
Each result has the page’s markdown, its url, its title and its depth. This is the real answer for the 3 pages, cut to the first result.
3. Cut each page into chunks
Split at headings, and keep a fenced code block whole, so a shell comment that starts with# is not read as a heading. Pack paragraphs into chunks of about 1,200 characters. Every chunk keeps the url, the page title and the heading it sits under. Save this file as chunk.ts.
4. Index with the source
Embedtext with your model and store the whole chunk next to the vector: id, url, title, heading and text. The id is the page URL and the chunk number. When you crawl a page again, delete the chunks with that url first, then add the new ones, so no old chunk stays behind.
When your retrieval returns a chunk, show title, heading and a link to url. Put the url in the prompt too, and tell the model to cite it.
5. Keep it fresh
Crawl again to rebuild the index. To re-read only what changed, create a monitor with acrawl target on the same section. It sends a monitor.page event for each page that changed or appeared. Read those pages again with Scrape and replace their chunks.
What to check
- The crawl’s
completedequals itstotal, anderrorsis empty. Pages that failed are listed inGET /v1/crawl/{id}/errors. - Every chunk has a
urlunder your start page. - The chunk count per page is plausible. A page with 1 chunk and 20,000 characters means the headings were not split.
- Pick three chunks, open their
urland find the text. If you cannot, fix the chunker before you embed. - Ask three questions you know the answer to. The right page should come back in the top results.
What it costs
Crawl costs $1 per 1,000 pages, which is $0.001 a page. A 100-page section costs $0.10, and a 1,000-page section costs $1. Reading the results costs nothing, and neither does a page that fails. A monitor on the same section costs the same per page, on every check. See Pricing.Related
- Crawl explains the start options.
- Page content sets what each page returns, such as
onlyMainContent. - Website change digest turns page changes into a summary.
- Parse documents turns your own PDFs into Markdown for the same pipeline.