Guides · 2026-10-17

How to convert any web page to Markdown for LLMs (Python & Node.js)

LLMs read Markdown far better than HTML. A typical article page is 150–500 KB of HTML — scripts, inline styles, navigation, tracking pixels — but only 5–20 KB of actual content. Converting it to clean Markdown cuts your token bill by 10× or more and gives the model clean structure: headings, lists, links and tables. Here's how to do it properly in Python and Node.js, and the three things that trip people up.

Step 1: get the HTML — after JavaScript runs

A plain HTTP request (requests.get, fetch) returns the HTML the server sent. For a growing share of the web — React, Vue, Next.js client components, docs sites, dashboards — that's an empty <div id="root">. You need a real browser to execute the JavaScript first:

# Python
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto(url, wait_until="domcontentloaded")
    page.wait_for_load_state("networkidle", timeout=3000)  # let client-side rendering finish
    html = page.content()
    browser.close()

Waiting for networkidle with a short cap is a good compromise: SPAs finish rendering, but you don't hang forever on sites with analytics beacons. Block images, fonts and media while you're at it — you don't need pixels to read text, and pages load 2–3× faster:

page.route("**/*", lambda r: r.abort() if r.request.resource_type in ("image", "media", "font") else r.continue_())

Step 2: extract the main content

Converting the whole page drags in menus, cookie notices, "related articles" and footers — noise that wastes tokens and confuses retrieval. Use a boilerplate-removal library to isolate the main content. In Python, trafilatura is excellent and can emit Markdown directly:

import trafilatura

md = trafilatura.extract(html, url=url, output_format="markdown",
                         include_links=True, include_tables=True, include_formatting=True)

In Node.js, Mozilla's Readability (the engine behind Firefox Reader View) plus Turndown does the same job:

import { JSDOM } from "jsdom";
import { Readability } from "@mozilla/readability";
import TurndownService from "turndown";

const dom = new JSDOM(html, { url });
const article = new Readability(dom.window.document).parse();
const md = new TurndownService({ headingStyle: "atx", codeBlockStyle: "fenced" }).turndown(article.content);

Step 3: fix the three classic problems

1. Relative links. [docs](/guide/intro) is useless once the text leaves the page. Resolve every href and src against the page URL before converting (Python: urllib.parse.urljoin; Node: new URL(href, url)).

2. Layout tables. Older sites (Hacker News, many forums and intranets) use <table> for layout. Converted naively, you get nested pipe-tables that are unreadable to humans and models alike. Treat tables without <th> header cells, or tables containing other tables, as layout and unwrap them into plain blocks; keep real data tables as Markdown tables.

3. Script and style contents. Many HTML-to-Markdown converters "strip" tags by removing the tag but keeping its text — so your Markdown fills up with minified JavaScript and CSS. Delete <script>, <style>, <noscript>, <svg> and <template> elements entirely before converting.

Step 4: chunk for retrieval (if you're doing RAG)

Split on headings rather than fixed character counts, so each chunk is a coherent section. Prepend the page title and section heading to each chunk — it dramatically improves retrieval for short sections. Keep the source URL in metadata so your app can cite it.

The one-request version

All of the above — browser rendering, resource blocking, main-content extraction, absolute links, layout-table cleanup — is what Shotlane's /v1/markdown endpoint does:

curl "https://shotlane.dev/v1/markdown?url=https://react.dev/learn" -H "X-API-Key: $SHOTLANE_KEY"

Use /v1/extract for JSON with title, description, author, publish date and word count; main_content=false to convert the whole page; or selector to convert one element. If you're building an agent, the MCP server gives it a read_page tool directly. Or just try it in the browser with the free URL to Markdown converter.