Scrape, map and report

Structured data from rendered pages: values by CSS selector, every URL of a site, and a report of what search engines, link previews and AI agents get. 1 credit per call.

Scrape values by selector

GET | POST/v1/scrape

Pull values out of a page with CSS selectors, no LLM involved. elements is JSON: {"name": "css selector"} for the first match's text, or {"name": {"selector": "…", "attr": "href", "all": true}}.

Field	Default	Description
`elements`	—	Map of names to selectors. `attr` is `text` (default), `html`, `outer_html`, `href` or `src` (made absolute), or any attribute name; `all: true` returns a list.

curl -X POST https://shotlane.dev/v1/scrape -H "X-API-Key: $SHOTLANE_API_KEY" -H "Content-Type: application/json" \
  -d '{"url":"https://news.ycombinator.com","elements":{"titles":{"selector":".titleline > a","all":true}}}'

{"url": "https://news.ycombinator.com/", "data": {"titles": ["…", "…"]}, "status_code": 200}

All page options work: wait_for_selector, click, cookies, proxy…

Map a site's URLs

GET | POST/v1/map

Every link on the rendered page plus the URLs in its sitemaps (found through robots.txt, including sitemap indexes and .gz files). Useful for finding the right page before reading it.

Parameter	Default	Description
`url`	—	Any page of the site.
`limit`	`500`	Maximum URLs returned (up to 5000).
`search`	—	Only URLs containing this text, e.g. `pricing`.
`include_subdomains`	`false`	Also keep URLs on subdomains.
`sitemap`	`true`	`false` returns only links found on the page.

curl "https://shotlane.dev/v1/map?url=docs.python.org/3/library/&search=asyncio" -H "X-API-Key: $SHOTLANE_API_KEY"

Report: SEO, link previews and AI crawlers

GET | POST/v1/report

What search engines, link previews and AI agents get from one URL, as JSON: robots.txt rules for 13 AI crawlers, llms.txt, how much content exists before JavaScript runs, bot walls, Open Graph and Twitter tags (with the og:image size), JSON-LD types, a Markdown preview and a screenshot link. Each finding comes with a fix in checks, plus a score. Try it in the browser: AI readiness checker, Open Graph checker.

curl "https://shotlane.dev/v1/report?url=vercel.com" -H "X-API-Key: $SHOTLANE_API_KEY"

PreviousDevices and page checks NextBulk