Scraping and conversion
Turn a URL, saved HTML or a local file into clean Markdown.
scrape takes a URL or a file and returns Markdown. md converts raw HTML you already have, with no fetch.
webfold scrape <url|file> [base] [--chunk n] [--json] [--action-budget n] [--timeout n]
webfold md <file> [base] [--chunk n] [--json]
webfold fetch <url> [--dump text|html|links] [--quiet] [--timeout N]Options
| Option | Effect |
|---|---|
--json |
Write one JSON object to stdout (for example bytes_markdown, blocks_content) for a saved file, stdin or a URL. A URL scrape with --json fetches the page and runs the same extraction as a file, so it skips the tiered acquisition that a plain URL scrape uses. Progress stays on stderr. |
--chunk n |
Split the output into chunks of about n characters. |
--timeout n |
Outer deadline for a URL scrape. |
--action-budget n |
Cap the number of acquisition actions (initial fetch, specification fetch, fallback navigation). |
--recovery-json |
Emit the acquisition (recovery) record as JSON alongside the output. |
[base] |
Base URL used to resolve relative links when converting a local file. |
How a URL is acquired
A URL scrape runs a tiered acquisition. Each tier reserves one action from the budget before it executes. Redirects, transport retries and robots requests are outside the count.
- Native HTTP fetch and conversion.
- If the response is empty and the page is a documentation shell, a same-origin HTTPS ReDoc specification can be fetched and the requested operation rendered.
- Fallback navigation through the built-in page model, which fetches, converts and builds a DOM. It does not execute JavaScript.
URL fragments are not sent in the initial request, but extraction keeps the full selector so it can find the targeted section. For pages whose content only exists after scripts run, use rendered pages.
What conversion keeps
- Headings, paragraphs, lists, code blocks and links (resolved against the page URL).
- Tables as one block. An all-
thfirst row becomes the header; ragged rows keep empty cells. Spans are not expanded and nested tables flatten into their parent cell. - Embedded hydration data on thin pages: when the initial text has fewer than 50 words, selected script payloads are decoded.