Scrape

Pull values out of a rendered page with CSS selectors and get them as JSON, no LLM involved. 1 credit per call.

Scrape values by selector

Loads the page, JavaScript and all, then reads each selector you name: the first match's text, an attribute, or every match as a list.

GET | POST/v1/scrape

curl https://shotlane.dev/v1/scrape \
  -H "X-API-Key: $SHOTLANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "quotes.toscrape.com",
    "elements": {
      "quotes": {"selector": ".quote .text", "all": true},
      "next": {"selector": "li.next a", "attr": "href"}
    }
  }'

import os
from shotlane import Shotlane

sl = Shotlane(os.environ["SHOTLANE_API_KEY"])
res = sl.scrape(
    "quotes.toscrape.com",
    elements={
        "quotes": {"selector": ".quote .text", "all": True},
        "next": {"selector": "li.next a", "attr": "href"},
    },
)
print(res["data"]["next"])

import { Shotlane } from "./shotlane.mjs";

const sl = new Shotlane(process.env.SHOTLANE_API_KEY);
const res = await sl.scrape("quotes.toscrape.com", {
  quotes: { selector: ".quote .text", all: true },
  next: { selector: "li.next a", attr: "href" },
});
console.log(res.data.next);

Response200 OK

{
  "url": "https://quotes.toscrape.com/",
  "data": {
    "quotes": [
      "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”",
      "“It is our choices, Harry, that show what we truly are, far more than our abilities.”",
      …
    ],
    "next": "https://quotes.toscrape.com/page/2/"
  },
  "status_code": 200
}

Parameters

urlstringRequired

The page to scrape, or send html instead.

elementsobjectRequired

1–50 names, each with a CSS selector: {"title": "h1"} for the first match's text, or {"links": {"selector": "a", "attr": "href", "all": true}}. attr is text (the default), html, outer_html, href or src (made absolute), or any attribute's name; all: true returns every match as a list (up to 1,000).

Plus every page option: wait_for_selector, click, cookies, proxy and the rest.

Returns

JSON: the page's final url, data with a value (or a list) per name, and the page's HTTP status_code. A selector that matches nothing gives null, or an empty list with all.

PreviousDevices and page checks NextMap