Crawl

Crawl a site breadth-first from a start URL and get every page as clean Markdown in one ZIP, with an index of titles, depths and word counts. 1 credit per page; failed pages are refunded.

POST/v1/crawl

curl -X POST https://shotlane.dev/v1/crawl -H "X-API-Key: $SHOTLANE_API_KEY" -H "Content-Type: application/json" \
  -d '{"url":"https://docs.python.org/3/tutorial/","limit":20,"include":"/3/tutorial/"}'

Fields

Field	Default	Description
`url`	—	Where the crawl starts.
`limit`	`50`	Maximum pages (up to 500).
`max_depth`	`2`	Link hops from the start page (0–10).
`include` `exclude`	—	URL patterns (substrings or `` wildcards), e.g. `/docs/` or `/blog/*`.
`include_subdomains`	`false`	Follow links to subdomains too.
`sitemap`	`false`	Also seed the crawl from the site's sitemaps.
`respect_robots`	`true`	Skip pages that robots.txt disallows.
`webhook_url`	—	https URL that receives a signed `crawl.finished` event (webhooks).

Status and results

GET /v1/crawl/{id} returns progress and, when finished, results and a zip_url. Bulk and crawl jobs share one running slot per account. The Python SDK's crawl() waits for the result by default.

PreviousBulk NextMonitors