Skip to main content
A retrieval index is only as good as its chunks. You crawl a docs section, cut each page at its headings, and keep the page URL and heading on every chunk. When the index returns a chunk, you can link to where it came from. You need an API key in STOPHY_API_KEY. See the Quickstart. The run on this page read 3 pages and cost $0.003.

1. Crawl the section

Crawl starts at one page, follows its links, and reads every page it finds. Starting is free, and each page is charged when it finishes. This request stops after 3 pages, which is enough to build and check the pipeline. Raise limit for the whole section.
The call answers at once with the crawl’s id.
To keep a crawl to one part of a site, set includePaths and excludePaths. See Scope and limits. To read only chosen pages, list them with Map and send the ones you want to a batch.

2. Read every page

Poll GET /v1/crawl/{id} until status is completed. Then read the pages with GET /v1/crawl/{id}/results, up to 100 at a time, and send cursor from data.cursor until it is missing. Both calls are free, and the pages are kept for 24 hours. See Read a crawl. Each result has the page’s markdown, its url, its title and its depth. This is the real answer for the 3 pages, cut to the first result.

3. Cut each page into chunks

Split at headings, and keep a fenced code block whole, so a shell comment that starts with # is not read as a heading. Pack paragraphs into chunks of about 1,200 characters. Every chunk keeps the url, the page title and the heading it sits under. Save this file as chunk.ts.
This function joins steps 1 to 3. It starts the crawl, waits, reads every result and returns the chunks.
On the 3 pages above it returned 61 chunks, from 3 different URLs. The longest was 1,483 characters, because a code block is never split. This is one of them.

4. Index with the source

Embed text with your model and store the whole chunk next to the vector: id, url, title, heading and text. The id is the page URL and the chunk number. When you crawl a page again, delete the chunks with that url first, then add the new ones, so no old chunk stays behind. When your retrieval returns a chunk, show title, heading and a link to url. Put the url in the prompt too, and tell the model to cite it.

5. Keep it fresh

Crawl again to rebuild the index. To re-read only what changed, create a monitor with a crawl target on the same section. It sends a monitor.page event for each page that changed or appeared. Read those pages again with Scrape and replace their chunks.

What to check

  • The crawl’s completed equals its total, and errors is empty. Pages that failed are listed in GET /v1/crawl/{id}/errors.
  • Every chunk has a url under your start page.
  • The chunk count per page is plausible. A page with 1 chunk and 20,000 characters means the headings were not split.
  • Pick three chunks, open their url and find the text. If you cannot, fix the chunker before you embed.
  • Ask three questions you know the answer to. The right page should come back in the top results.

What it costs

Crawl costs $1 per 1,000 pages, which is $0.001 a page. A 100-page section costs $0.10, and a 1,000-page section costs $1. Reading the results costs nothing, and neither does a page that fails. A monitor on the same section costs the same per page, on every check. See Pricing.