Crawl

POST /crawl

Combines Map and Extract: traverse a site from a root URL and return extracted content for each page discovered. Use instructions to steer the crawl toward relevant sections.

Request

curl

curl -X POST https://api.crisps.ai/crawl \
  -H "Authorization: Bearer $CRISPS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.crisps.ai",
    "instructions": "API reference pages only",
    "max_depth": 2,
    "limit": 40,
    "extract_depth": "basic",
    "format": "markdown"
  }'

Python

import os, httpx

r = httpx.post(
    "https://api.crisps.ai/crawl",
    headers={"Authorization": f"Bearer {os.environ['CRISPS_API_KEY']}"},
    json={
        "url": "https://docs.crisps.ai",
        "instructions": "API reference pages only",
        "max_depth": 2, "limit": 40, "extract_depth": "basic",
    },
    timeout=300,
)
print(len(r.json()["results"]))

Node.js

const res = await fetch("https://api.crisps.ai/crawl", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.CRISPS_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://docs.crisps.ai",
    instructions: "API reference pages only",
    max_depth: 2, limit: 40, extract_depth: "basic",
  }),
});
console.log((await res.json()).results.length);

Try it

POST /crawl Try it

API base URL API key

Request body { "url": "https://docs.crisps.ai", "instructions": "API reference pages only", "max_depth": 2, "limit": 40, "extract_depth": "basic" }

Send request

Body parameters

Accepts all Map parameters (url, instructions, max_depth, max_breadth, limit, path/domain filters, allow_external), plus:

Parameter	Type	Default	Description
extract_depth	string	basic	basic or advanced — sets per-page extraction cost (1× or 2× per 5 pages).
format	string	markdown	markdown or text.
chunks_per_source	integer	3	Top-N most relevant chunks per page, used with instructions.
include_favicon	bool	false	Include each page’s favicon.

Response

{
  "base_url": "https://docs.crisps.ai",
  "results": [
    { "url": "https://docs.crisps.ai/api/search/", "raw_content": "# Search\n…" }
  ]
}

Each result carries raw_content (and favicon when requested). Billed like extract at the chosen extract_depth — basic = 1 credit per 5 extracted pages, advanced = 2 per 5 (so a 10-page advanced crawl ≈ 4 credits) — reported in the X-Credits-* headers.