Map

Every URL of a site that one page links to or its sitemaps list, to find the right page before reading it. 1 credit per call.

Map a site's URLs

Renders the page you give, collects its links, and adds the URLs in the site's sitemaps (found through robots.txt, including sitemap indexes and .gz files). Only URLs on the same site are kept.

GET | POST/v1/map

curl -G https://shotlane.dev/v1/map \
  -H "X-API-Key: $SHOTLANE_API_KEY" \
  -d url=docs.python.org/3/library/ \
  -d search=asyncio

import os
from shotlane import Shotlane

sl = Shotlane(os.environ["SHOTLANE_API_KEY"])
res = sl.map("docs.python.org/3/library/", search="asyncio")
print(res["links"])

import { Shotlane } from "./shotlane.mjs";

const sl = new Shotlane(process.env.SHOTLANE_API_KEY);
const res = await sl.map("docs.python.org/3/library/", { search: "asyncio" });
console.log(res.links);

Response200 OK

{
  "url": "https://docs.python.org/3/library/",
  "count": 1,
  "links": [
    "https://docs.python.org/3/library/asyncio.html"
  ],
  "sources": {
    "page_links": 295,
    "sitemap_urls": 7
  }
}

Parameters

urlstringRequired

Any page of the site.

searchstring

Only URLs that contain this, e.g. pricing (case doesn't matter).

limitintegerDefault 500

At most this many URLs, up to 5000.

include_subdomainsbooleanDefault false

Keep URLs on the site's subdomains too.

sitemapbooleanDefault true

false returns only the links on the page.

Returns

JSON: the page's final url, count, the links (the page's own first, then the sitemaps', without duplicates or # fragments), and sources: how many links the page had and how many URLs the sitemaps listed. A bot wall isn't charged and says so in X-Shotlane-Blocked.

PreviousScrape NextReports