Webfold
Menu

Format · HTML

Saved or fetched HTML to Markdown, with page furniture removed and tables kept.

Yes, Webfold converts HTML from a file, a URL or a rendered page to Markdown in process with no browser: a hot median of 4.6 µs a page on 54 public fixtures, 3.9 times faster than Firecrawl's Rust html-extractor on the conversion step alone, and a main-content accuracy level with Trafilatura (test F1 0.861 against 0.863), not ahead of it.

Figures from: HTML to Markdown conversion benchmark, Main-content extraction quality benchmark.

Webfold reads HTML from a file, a URL or a rendered page and writes Markdown: headings, paragraphs, lists, fenced code, pipe tables, images and links with resolved addresses. Navigation, headers, footers, asides, forms, scripts and styles are dropped by their tags, and a trained scorer can keep only the main-content blocks of one saved page. The converter runs in-process and needs no browser.

Last updated , the newest measurement or check on record for this page.

Get startedAll formats

See it work

How do I try it from wherever I work?

1webfold md saved-page.html https://example.com/ --json

Input

What input does HTML reading accept?

Saved HTML files

`webfold md <file> [base]` and `webfold scrape <file>` read a local file with `std::fs::read_to_string`, so the file must be valid UTF-8. The optional base is the address that relative links resolve against.

URLs

`webfold scrape <url>` fetches through the native backend, follows up to 10 redirects and decodes the body as lossy UTF-8. `--timeout` sets the acquisition deadline in seconds, default 20. A blocked or empty response can escalate to the built-in headless engine.

Rendered pages

`webfold scrape <url> --render` converts the DOM after scripts have run. `--browser` names a CDP endpoint for a Chromium you run, and `--wait-for`, `--wait-selector`, `--scroll-document` and `--expand-disclosures` shape the render.

Library input

`webfold::to_blocks(html, base)` takes a `&str` and returns blocks. The caller chooses the encoding; the function does no charset detection.

WARC response records

`webfold warc` and `webfold replay` take the payload of each `response` record, skip the HTTP head and run the same converter. `replay` also decodes gzip and deflate content coding first; other codings are passed through with a note.

Crawl directories

`webfold crawl <dir|url>` converts a set of pages and removes blocks that recur across them. That is a separate configuration from single-page selection.

Output

What comes out?

Markdown

Kept blocks are joined in source order, separated by blank lines, and trimmed. Headings use `#`, list items use `- ` or `1. `, code is fenced, strong and emphasis use `**` and `*`, links are `[text](href)` and images are `![alt](src)`. Duplicate blocks are not removed on this path.

Tables

A rectangular table becomes a pipe table, with the first row as the header when every cell in it is a header cell. A table that is not rectangular is written row by row without padding. Spans are not expanded.

Block and link lists

The library returns `Block { md, sig, links }` values, and `links()` returns the link targets of all blocks in order.

JSON and chunks

`md` and file `scrape` accept `--json`, which prints one object with source, base, bytes_html, blocks_total, blocks_content, bytes_markdown, elapsed_ms and markdown. `--chunk n` splits the Markdown into chunks, and `--jsonl` writes the chunks as JSON lines.

Text from thin pages

When the body has fewer than 50 words, text can come from embedded Next.js, Nuxt, JSON-LD, SvelteKit, React Server Components, Remix and Angular payloads, and from some CMS, booking, photo and video payloads.

How it works

What does the code do, step by step?

  1. 1

    Scan tags

    A byte scanner walks the source once. Tag tokens borrow the input, the bodies of script and style are skipped, comments are dropped, and entities are decoded from a fixed table when text is emitted.

  2. 2

    Drop by element

    Elements such as script, style, noscript, svg, head, nav, header, footer, aside, form, button, iframe, template, dialog, select and option are contained with their subtree. A header inside main or article and a figure are kept, and an aside is kept only inside main. A stray close tag cannot release a hidden or dropped container.

  3. 3

    Hide by attribute and style

    Elements with the hidden attribute, type=hidden, a fixed set of ids and class tokens for consent and ad chrome, or inline display or visibility that hides them are excluded with their subtree. aria-hidden subtrees are excluded by default. The body element's own inline style does not hide the page: a body with display:none, a common anti-flash pattern, no longer drops everything, and only its descendants are judged by their inline style.

  4. 4

    Emit blocks

    Text is collected into blocks. Headings, paragraphs, lists, code, links, images and emphasis become inline Markdown, and table cells are gathered by the table module and flushed as one table block.

  5. 5

    Fall back to embedded data

    If the body has fewer than 50 words or no blocks, the hydration payloads found during the scan are unpacked and their text is added. Payloads inside hidden subtrees are never used.

  6. 6

    Select main content (saved files only)

    For `md` and `scrape <file>`, each block is scored by a two-stage gradient-boosted model over 296 features. Blocks at or above 0.35 are kept, or 0.5 with --precision; if no block reaches the threshold the whole page is returned. The URL path and crawl do not run this step.

  7. 7

    Render

    Kept blocks are joined into one Markdown string. The dropped count is non-zero only when a crawl template is supplied.

Measured

What did we measure, and how?

HTML to Markdown speed against Firecrawl's Rust html-extractor

Run of Five runs, then a CPU-pinned re-run on 2026-09-30

How fast does each tool convert the same saved HTML pages to Markdown, and do both pass the same quality assertions?

Corpus. The 54 public fixtures of Firecrawl's Rust html-extractor (github.com/firecrawl/html-extractor at commit 0a2b7432a20e, crates/html-extractor/tests/fixtures); 52 carry quality assertions (211 expected lines and forbidden phrases). The fixture licence is not recorded in the result files.

ToolVersionRole
Webfoldgit 60ac8ce1789b for the re-run (blockquote-prefix and fenced-code fixes); earlier runs at older revisionsSubject, in process.
Firecrawl html-extractorfirecrawl/html-extractor, commit 0a2b7432a20e (Rust)Comparison, in process, same binary and same bytes.
MetricHow it is computed
HotDirect Rust conversion of preloaded HTML: 2 warm-ups, 7 samples of 50 conversions per fixture; the figure is the lower median of the 54 per-fixture medians.
Fresh processA fresh runner process per conversion including file read, warm operating-system cache: 7 samples per fixture, same aggregation.
Asserted fixtures passedFixtures where every expected line is present and no forbidden phrase appears. This is permissive assertion parity, not byte-identical Markdown or a fidelity score.
MeasurementWebfoldRust html-extractor
Hot conversion, median of five runs4.601 µs17.829 µs
Fresh-process conversion, median of five runs0.372 ms4.021 ms
CPU-pinned re-run, 2026-09-30, hot (two runs)4.127 to 4.251 µs17.506 to 17.889 µs
CPU-pinned re-run, 2026-09-30, fresh process (two runs)0.360 to 0.362 ms4.028 to 4.036 ms
1# webfold-bench is private until the public release of Webfold. Run this from a checkout of it.
2git clone https://github.com/firecrawl/html-extractor
3cd webfold-bench/bench/structured-h2h
4CARGO_NET_GIT_FETCH_WITH_CLI=true cargo build --release
5taskset -c 8 ./target/release/webfold-structured-h2h --corpus ../../../html-extractor/crates/html-extractor/tests/fixtures --out structured-parity.json --warmup 2 --samples 7 --iterations 50

Reproduce: webfold-bench/bench/STRUCTURED-CONVERSION.md (revisions, hashes and dependency overrides: webfold-bench/bench/results/parity-refresh-2026-09-30.json). Figures read from webfold-bench/bench/results/html-rust-cold-hot-2026-09-24.json, parity-refresh-2026-09-30.json and bench/STRUCTURED-CONVERSION.md.

Limits of this measurement

  • Conversion step only: not the hosted Firecrawl service, networking or browser rendering. Intel Core i9-14900KF; the re-run was pinned to one logical CPU, the earlier runs were not.
  • The fixtures are development fixtures that fixes were made against. The assertion score is not a measure of complete Markdown fidelity: a review found six malformed tables and one extra blank line that the assertions missed.
  • Timings predate the current repair work and no confidence interval is claimed. The tracked runner manifest pins an older Webfold, so reproducing the final figures needs the revision and overrides recorded in the evidence bundle.
  • An earlier run on 2026-09-24, before the fixes, measured 5.0 times hot and 7.4 times fresh with Webfold at 0.955 on the assertions.

One row or limit identical to the HTML to Markdown conversion benchmark is stated there only, not repeated here.

HTML conversion benchmark →

Main-content extraction on WCXB, the held-out test split

Run of 2026-10-14 and 2026-10-15 (Rust re-run 2026-11-04)

How much of a page's main content does webfold md recover, and how does that compare with the published and re-run baselines?

Corpus. WCXB, the Web Content eXtraction Benchmark (Foley, 2026, CC-BY-4.0): 2,008 human-reviewed pages from 1,613 domains in seven page types, scored here on the 511 held-out test pages. Dataset commit c039d5ee9f5a. The benchmark's author also wrote rs-trafilatura.

ToolVersionRole
webfold mdwebfold md with the trained main-content selector (docs/WCXB-MAIN-SELECT.md in the webfold repository), one process per page; the revision of the scored build is not recorded in the README (a Rust re-run on 2026-11-04 reproduced 0.8608)Subject.
Trafilatura2.3.1Re-run by us on the same 511 pages.
rs-trafilatura 0.2.2, Trafilatura (version not stated), dom-smoothie, Readabilityas published in the WCXB READMEPublished baselines, not re-run.
MetricHow it is computed
F1Word-level F1: lower-cased \w+ token bags compared with each page's plain-text main content, averaged over pages, by the benchmark's own scorer. Our Markdown output is projected to plain text first (links keep their label, markup is dropped); that projection is our assumption.
SystemTest F1Status
rs-trafilatura 0.2.20.893published
Trafilatura 2.3.10.863re-run here
Webfold, with the trained selector0.861ours (precision 0.834, recall 0.931)
Trafilatura0.833published in the WCXB README
Webfold, before the selector0.815ours
dom-smoothie0.808published
superdocuments HTML module, revision 786e8bd0.797ours, no main-content selection
Readability0.726published
1git clone https://github.com/Murrough-Foley/web-content-extraction-benchmark
2# webfold-bench is private until the public release of Webfold. Run this from a checkout of it.
3cd webfold-bench
4cargo run --release -p webfold-wcxb -- run ../web-content-extraction-benchmark test <webfold binary> out/
5cargo run --release -p webfold-wcxb -- score ../web-content-extraction-benchmark test out/test.text.json

Reproduce: webfold-bench/tests/harness/wcxb/README.md; results and method: webfold-bench/bench/wcxb/README.md. Figures read from webfold-bench/bench/wcxb/README.md and webfold-bench/external-eval/results/wcxb_test.json.

Limits of this measurement

  • The selector was fitted on the development split only and the test split was scored once. The Rust scorer reimplements the benchmark's evaluate.py and reproduces the recorded figures to four decimals; it was not run side by side with the original.
  • On accuracy Webfold is level with Trafilatura 2.3.1, not ahead of it, and rs-trafilatura is ahead. Webfold's case is speed and memory. Precision (0.834) is the weaker side.
  • A single page at a time: Webfold's template recurrence across a site's pages is absent in this benchmark, so it says nothing about multi-page crawl quality. Firecrawl's extractor has not been scored on WCXB.
  • Baselines other than Trafilatura 2.3.1 are the benchmark author's published numbers, which we did not reproduce (the 0.833 published for Trafilatura does not reproduce).

Main-content extraction quality benchmark →

Evidence

How do we know each claim?

Each row says how we know. Read from the code means the behaviour is in the implementation at the path shown. Covered by a test names a test in the repository; this page does not claim it was run for this release. Measured points to a benchmark with its protocol and limits.

ClaimHow we knowWhereDetail
The scrape and md commands are two branches of one handler. md reads a file without fetching and takes an optional base; scrape goes to the network only for http or https sources.Read from the codewebfold/src/main.rs:417-442; webfold/src/main.rs:424-430md reads with std::fs::read_to_string and takes the base from argument 3. The URL branch calls acquisition_cli::scrape and returns before the JSON printer.
Main-content selection uses 296 features per block, a two-stage model and a default keep threshold of 0.35.Read from the codewebfold/src/html_extract/main_select/mod.rs:23; :372-398; webfold/src/html_extract/main_select/model.rs:1-3NF = NUMERIC + 2*BINS + LEX_BINS = 40 + 128 + 128. DEFAULT_THRESHOLD = 0.35 (line 383). select_main returns the whole page when no block reaches the threshold (lines 387-398). model.rs is generated by tools/main_select/gen_model.py.
Selection runs only for the file branch of md and scrape and for the search route. The URL scrape path and the library scrape call to_blocks and render directly.Read from the codewebfold/src/main.rs:462-469; webfold/src/cli/route.rs:27-31; webfold/src/lib.rs:404-406; webfold/src/lib.rs:474-476A search for select_main call sites finds only main.rs and cli/route.rs. lib.rs scrape and scrape_tiered_inner call to_blocks and render(&blocks, None, 1) with no selector.
Elements are contained by a fixed drop list, and figure and header inside main or article are transparent.Read from the codewebfold/src/html_extract.rs:86-146; webfold/src/html_extract/content.rs:157-200DropTag::from_tag maps script, style, noscript, svg, head, nav, header, footer, aside, form, button, iframe, template, dialog, select, option and figure. content.rs marks figure and header-in-main as transparent and treats aside as soft unless it sits inside main.
Hidden content is detected by attributes, a fixed id and class list, and inline display or visibility.Read from the codewebfold/src/html_extract/content.rs:234-340; webfold/src/html_extract/content.rs:342Rules: the hidden attribute, type=hidden, ids such as wayback-toolbar and __VIEWSTATE, class tokens for ad and cookie-consent chrome (including the cookieconsent- prefix), and hidden_style for inline display and visibility.
Embedded payloads are read only on thin pages, using a named list of hydration kinds.Read from the codewebfold/src/html_extract.rs:945-947; webfold/src/html_extract.rs:257-269; webfold/src/html_extract.rs:865-902Unpacking runs when there are no blocks or fewer than 50 words. Kinds: NextData, NextAppRouterRsc, Nuxt3Data, SvelteKit, RemixContext, AngularState, JsonLd, PageContentHtml and StructuredHtml (the last two need the native feature, which is in the default build). tests/parser_review.rs:89 hidden_scripts_do_not_supply_hydration_content checks the hidden-script rule.
Entity decoding handles 13 named entities and numeric references. Other named entities are not decoded.Read from the codewebfold/src/html_extract.rs:1045-1100Named: amp, lt, gt, quot, apos, nbsp, mdash, ndash, hellip, rsquo, lsquo, ldquo, rdquo. An unrecognised name leaves the ampersand in the output as a literal character.
Tables become pipe tables only when the grid fits a size budget; otherwise rows are written unpadded. Spans are not expanded.Read from the codewebfold/src/html_extract/table.rs:93-140; ARCHITECTURE.md:8-20markdown() checks grid_slots against 2 * source_cells + width. ARCHITECTURE.md describes the same rule and says spans are not expanded.
Table, nested table and span behaviour has named tests.Covered by a testwebfold/tests/corpus_review.rs:10, 64, 168, 200Tests column_headers_and_rows_form_one_contiguous_table_block, excessive_rectangular_padding_keeps_unpadded_rows_and_links, nested_tables_keep_content_without_claiming_nested_grid_fidelity and spans_retain_source_cells_without_inventing_expanded_values. Read, not run.
Fenced code, language classes, images, entity decoding and script stripping have named tests.Covered by a testwebfold/tests/firecrawl_compat.rs:144, 232, 352, 433, 474test_fenced_code_blocks, test_images, test_pre_code_language_classes, test_html_entity_decoding and test_script_tags_completely_stripped. The file header says these were ported from a Firecrawl suite whose source is not in this checkout. Read, not run.
Hot and fresh-process conversion times against the Rust html-extractor on 54 fixtures.Measuredbenchmark slug html-conversion; webfold/README.md:70README records 4.601 µs (Webfold) against 17.829 µs for the Rust html-extractor, hot, and 0.371781 ms against 4.021488 ms, fresh process, with 52 of 52 asserted fixtures passing on both. It is a historical five-run measurement on one host. The manifest it cites (benchmarks/review/h2h/manifest.json) is not in this checkout, so the figures are cited, not re-derived.
Main-content F1 on the WCXB held-out test split is 0.861, with precision 0.834 and recall 0.931. Before selection it was 0.815.Measuredbenchmark slug extraction-quality; webfold/docs/WCXB-MAIN-SELECT.md (results table)Test split: F1 0.861, precision 0.834, recall 0.931. md before the selector: 0.815, 0.743, 0.968. Development 5-fold cross-validation: 0.812. The doc says the selector was trained on the development split only and the test split was scored once. The WCXB dataset is not in this checkout, so the figures are cited from the doc and the benchmark record.
A charset helper in the headless engine, with unit tests, decodes a fetched page in its declared charset. The to_blocks function does not use it.Read from the codewebfold/src/headless.rs:1307-1322; webfold/src/headless.rs:2600-2625document_encoding checks a byte order mark, then the Content-Type charset, then a meta prescan of the first 1024 bytes, then falls back to UTF-8. The tests in document_encoding_tests cover these cases. decode_document is called when a URL is scraped (webfold/src/lib.rs, tiered scrape), so a Shift_JIS or GBK page is decoded instead of turning into replacement characters. File input and to_blocks still take a &str.

What are the limits?

Know the limits

  • Plain conversion does not run JavaScript. The URL path escalates to the headless engine only when the response is classed Blocked or Empty (webfold/src/acquisition/coordinator.rs:35-49), so a page whose text is built only by scripts can still come back thin. Use --render with a browser you control for those pages.
  • Encodings: md and scrape <file> need valid UTF-8 (webfold/src/main.rs; webfold/src/cli/document.rs). A scraped URL is decoded in the charset the response declares (byte order mark, Content-Type, then a meta tag; webfold/src/lib.rs). The WARC harness still decodes with lossy UTF-8, so bad bytes there become replacement characters (webfold/src/cli/warc.rs).
  • Tables: spans are not expanded, nested tables keep their text without a nested grid, and a non-rectangular table is written row by row (webfold/src/html_extract/table.rs:1-3; tests/corpus_review.rs:168).
  • Styling and entities: only inline display and visibility are read. External stylesheets are not parsed, and the one page-specific CSS rule emulated is a Storybook state rule (webfold/src/html_extract/content.rs:31-36). Only 13 named entities are decoded; others, such as &copy;, stay as literal text (webfold/src/html_extract.rs:1045-1100).
  • Links and images are not sanitised. javascript:, data: and protocol-relative targets are written as they appear (webfold/src/html_extract/url.rs:1-6). Treat the Markdown as untrusted text.
  • Depth: a source with more than 256 nested block elements gives no blocks from the production converter, because depth overflow discards partial output. Inline formatting (such as strong and em) does not count toward the bound, and nested emphasis of the same kind collapses, so a page with hundreds of unclosed strong tags is still read (webfold/src/html_extract/extraction_state.rs; tests/firecrawl_scrape_v1_regressions.rs).
  • Selection is for one saved page and is not a page-type classifier. The code has no page-type enum; classify_response (src/stealth.rs:688) detects block pages and challenges. The WCXB doc names forum (0.76), product (0.75) and collection (0.73) as the weakest held-out types (docs/WCXB-MAIN-SELECT.md, Limits).
  • Benchmark framing: the html-conversion comparator is Firecrawl's Rust html-extractor, which is a different project from the crates.io package of the same name (html-extractor). Firecrawl's extractor has not been scored on WCXB. The WCXB figures are a single held-out split scored once, not a crawl configuration.

Compared

How does it compare with other tools?

Compared withWhat the evidence saysSource
Trafilatura 2.3.1 (Python, Apache-2.0)On the WCXB held-out test split Webfold scores 0.861, and the figure for Trafilatura in the benchmark's README is 0.833. A re-run of Trafilatura 2.3.1 on the same 511 pages scored 0.8632, above Webfold's, so this result does not show Webfold ahead.WCXB README; webfold-bench/external-eval/results/wcxb_test.json
rs-trafilatura 0.2.2The published held-out figure is 0.893, above Webfold's 0.861. The WCXB repository and rs-trafilatura share an owner, so the baseline figures come from the benchmark's author and were not re-run here.WCXB README; rs-trafilatura README
pandoc 3.13Pandoc's manual lists html as an input and markdown as an output, so the output shape is comparable. Fidelity was scored against Pandoc 3.1.3's own reading of 42 HTML files: 0.834 for superdocuments before the Pandoc-compatibility work, 0.990 after it, 0.968 for MarkItDown 0.1.9. These scores are for superdocuments main. Part of the work behind them is in the revision Webfold pins and the rest, the Pandoc layout, came after it, so the Webfold command line has not been scored. Thirty of the 42 files were written by Pandoc's HTML writer, which favours any reader that copies Pandoc; the 12 hand-written files score 0.966. No speed comparison on HTML has been run.Pandoc manual

Where should I read next?

HTML to Markdown benchmark →

Main-content extraction benchmark →

Scrape →

Scraping guide →

Security reference →

FAQ

Frequently asked questions

Does it run JavaScript?

Not for plain conversion. A URL moves to the built-in headless engine only when the response looks blocked or empty. --render drives a browser, and --browser takes a CDP endpoint for a Chromium you run.

Why does a page in windows-1252 fail with md?

md and scrape on a file read the file as UTF-8 and report a read error when it is not. URL fetches use lossy UTF-8 instead, so bad bytes become replacement characters. Re-save a file as UTF-8 for an exact result.

Can I keep only the main article?

For a saved file, yes. The default selector keeps blocks scoring at least 0.35, --precision raises that to 0.5, and --all-blocks keeps every block. The URL path does not use the selector.

Are tables kept?

Yes, as Markdown pipe tables when the grid is rectangular. Spans are not expanded, and a non-rectangular table is written row by row without padding.

Does --json work with a URL?

Yes, for a plain scrape <url> --json. It fetches the page, then prints one JSON object with the status, final URL, strategy, byte and block counts and the Markdown. A PDF or Office response is converted. With --render, --recovery-json, --chunk, --jsonl, --all-blocks or --precision the command takes another route.

Is the extracted text safe to pass to a model?

Extraction removes script, style, hidden and chrome elements, but visible text is kept as written. Prompt-injection text on a page survives extraction, so treat the output as untrusted.

Read HTML with Webfold.

Run the command-line tool today, and join the waitlist for the hosted API.

Join the waitlistRead the quickstart