Performance
What Webfold does, and how fast it does it.
Webfold converts a web page to Markdown in a median 4.6 µs in process (the conversion step only, on 54 public pages) and turns a 7,000-page live site into Markdown in about 16 seconds on four cores over the network; each job below lists its own speed, throughput and memory, with the scope of every figure.
9 jobs. For each: what goes in, what comes out, and the speed, throughput and memory measured on that job alone. How it compares with other tools comes after, on the compare pages.
At a glance
What are the jobs, and the headline number for each?
| Job | Speed | Throughput |
|---|---|---|
| Turn a web page into clean Markdown | 4.6 µs a page, in process | About 217,000 pages a second per core, conversion only |
| Crawl a whole site into a folder of Markdown | 7,000 live pages in about 16 s | 427 to 443 pages a second on 4 cores, over the network |
| Read web archives (WARC) at volume | Not a latency job | Up to 26,675 pages a second on 8 threads |
| Convert Word, Excel, PowerPoint, OpenDocument and ebooks to Markdown | 27.7 µs a document, in process | 203,854 documents a second on 64 workers |
| Read PDFs, with OCR when the page is a picture | 5 to 12 ms a text file; 0.6 to 1.0 s a scanned page | Not measured at scale |
| Read LaTeX papers and BibTeX bibliographies | About 4 ms a LaTeX file | Not measured at scale |
| Run a page's scripts and drive it like a browser | 8 ms to start, 70.7 ms a scripted task | 106.6 tasks a second in 30 MB |
| Parse search results and search a folder | 1.4 ms a results page | 49.3 MB a second parsing; 0.42 s to index 554 files |
| How good is the extraction? | F1 0.861 on 511 pages | Level with Trafilatura (0.863 re-run here), behind rs-trafilatura (0.893 published) |
Job
Turn a web page into clean Markdown.
Takes the HTML of a page, from a file or a URL, and returns Markdown with the headings, links, tables and code kept and the navigation, banners and footers that repeat across a site folded away. No browser is started and no page tree is built.
| Measure | Webfold | Scope |
|---|---|---|
| Convert one page, in process | 4.6 µs median | 54 public pages, hot, conversion step only. One core converts about 217,000 pages a second at that rate (1 ÷ 4.601 µs). |
| Convert one page, fresh process | 0.37 ms median | Includes starting the process and reading the file. |
| Fetch and convert one live page | 90 ms wall, 14.9 MB peak memory | One scrape on a 16-core host, including the network. Directional. |
Job
Crawl a whole site into a folder of Markdown.
Takes a sitemap or a list of URLs, fetches every page (robots.txt is always honoured), converts each one and writes one Markdown file per page, with a manifest and a failure ledger so an interrupted run can resume.
| Measure | Webfold | Scope |
|---|---|---|
| A live site, 7,000 pages | 15.8 to 16.4 s (427 to 443 pages a second) | MDN, 96 requests at a time, one 4-core machine, 0 failures on two different samples. 1.27 GB of HTML became about 35 MB of Markdown. |
| The same crawl against a local server | 7,718 to 8,703 pages a second | 7,000 and 10,000 pages on loopback: Webfold's own cost, without the network. |
| A local crawl, no network | 90.6 pages a second | 5 repeats of 50 pages; 27 MB sampled memory. |
Job
Read web archives (WARC) at volume.
Captures pages to WARC, replays them, and extracts clean text from archives such as Common Crawl, with the exact bytes kept for audit.
| Measure | Webfold | Scope |
|---|---|---|
| Seed path, 8 threads | 26,675 pages a second | 84,749-page Common Crawl subset (15.9 GB); 4.8 GB peak memory. |
| Raw WARC, 8 threads | 14,238 pages a second | 15.2 GB peak memory. |
| Seed path, 1 thread | 2,943 pages a second | 35 MB peak memory. |
Job
Convert Word, Excel, PowerPoint, OpenDocument and ebooks to Markdown.
Reads DOCX, PPTX, XLSX, XLSM, XLSB, ODT, ODS, ODP, EPUB, RTF and CSV in process, and the text of legacy DOC files,, with no Office installation and no external converter, and returns Markdown.
| Measure | Webfold | Scope |
|---|---|---|
| One document, in process | 27.7 µs median | 45 public fixtures, hot. |
| Throughput, 1 worker | 17,670 documents a second | 17-fixture pool of small CSV, XLSX and DOCX files. |
| Throughput, 64 workers | 203,854 documents a second | The same pool; 21.0 MiB peak memory for the whole run. |
| Command-line calls, 16 workers | 2,173 a second | One process per document, so process start is included. |
Job
Read PDFs, with OCR when the page is a picture.
Reads the text layer of a PDF in process. When a page is a scan it hands the page to Tesseract and returns the recognised text.
| Measure | Webfold | Scope |
|---|---|---|
| A file with a text layer | 5 to 12 ms | Native parser, one process per file with process start included. Eight test PDFs. |
| A scanned page, with OCR | 0.6 to 1.0 s | Dominated by Tesseract. The three of eight test PDFs that need it. |
| Accuracy on eight test PDFs | Character error rate 0.000 to 0.141 | Fuzzy scan 0.023, mixed native and scanned 0.119, two columns 0.141 (the best a correct read scores). |
| Against liteparse on 200 PDFs | Overall score 0.760 against 0.883 | Tables 0.376 against 0.818. Headings 0.574 against 0.824. Webfold loses on both. |
Job
Read LaTeX papers and BibTeX bibliographies.
Reads LaTeX source with its tables, figures, captions, labels and cross-references, a paper's whole project (\input, \include and the .bib files next to it), BibTeX or BibLaTeX bibliographies, MathML, and arXiv e-print downloads as they come from arxiv.org, and returns Markdown or citation data.
| Measure | Webfold | Scope |
|---|---|---|
| A LaTeX file to Markdown | 3.97 ms median | 16 hand-made files, command line, process start included. |
| A BibTeX file to citation data | 2.4 to 6.8 ms | 5 of 5 files read, command line. |
| A BibTeX file, in process | 0.08 to 0.24 ms | The same five files. |
| LaTeX fidelity to Pandoc's reading | 0.968 mean over 118 documents (0.857 before) | Superdocuments main, newer than the revision Webfold pins. Pandoc 3.1.3 is the reference; 10 of 118 documents are identical. Algorithm pseudocode is kept as source, and a reference list is appended from the .bib, both unlike Pandoc. |
| BibTeX to citation data against Pandoc | 179 of 179 entries identical (0 before) | 25 files, Pandoc 3.1.3 reading them as BibTeX or BibLaTeX; same superdocuments change. |
Job
Run a page's scripts and drive it like a browser.
Runs the page's scripts in its own engine and serves the Chrome DevTools Protocol, so Playwright, Puppeteer, Selenium and other clients can open, click, fill and read pages that build themselves with JavaScript, without starting Chrome. It passes the full independent benchmark for agent browsers.
| Measure | Webfold | Scope |
|---|---|---|
| Start | 8 ms | Cold start until it answers. |
| One scripted task, warm | 70.7 ms median (74.1 ms at p95) | Open a page, run the script that builds 200 rows and calls an API, check six features, read the result, close. The page has a 60 ms timer, so about 60 ms is the floor. |
| Throughput | 106.6 tasks a second | 8 workers, one connection each; 0 of 363 tasks failed. |
| Memory and CPU | 30 MB resident, 0.8 CPU-seconds | For the whole run. |
| Single protocol calls | 0.2 to 0.6 ms to evaluate; 18.8 ms to navigate an 8,000-row page | Medians of 30 calls, run of 4 November. |
| Compatibility with agent tools | 1,928 of 1,928 tasks | Lexbench, the independent agent-browser benchmark: 13 drivers plus raw CDP, and 188 web-platform tasks. Chrome passes 1,926 in the published run. |
| Page JavaScript | 68 of 68 expressions, and 300 of 300 more, match Chrome | Ordinary page script over raw CDP, compared with Chromium on the same machine: the 68 original expressions, then 300 more on the language, built-ins, async code and the DOM. BigInt works (all 388 BigInt expressions match), typed arrays share real bytes (all 277 typed-array expressions match) and Float16Array is in (all 60 match). Generators run lazily and async code runs in Chrome's order (206 of 217 match). Form controls keep their state apart from the markup (197 of 250 match). SVG case is kept (198 of 198). These sets are model-written samples checked against Chromium, not conformance suites. Not yet in the engine: canvas, WebGL, web workers, WebAssembly. |
Job
Parse search results and search a folder.
Normalises the result pages of six search engines into one shape, and indexes and searches a folder of files with ranked passages.
| Measure | Webfold | Scope |
|---|---|---|
| Parse a results page | 1.4 ms mean, 49.3 MB a second | 19 saved pages from six engines: 25.65 ms in total, all 19 exact. |
| Index a folder | 0.42 s | 554 files. xerj took 46.6 s. |
| Query a folder | 4.0 ms median | A new process per query. A running serve process answers in 0.7 ms. |
| Right file ranked first | 94.6% of 240 queries | Plain-words queries cut from the folder, lexical ranking only. xerj 80.8%. |
Job
How good is the extraction?.
Speed means little if the text is wrong. On the public WCXB benchmark, 511 held-out pages scored with the benchmark's own tool, Webfold's main-content extraction scores an F1 of 0.861.
| Measure | Webfold | Scope |
|---|---|---|
| Main-content F1 | 0.861 | WCXB test set, 511 pages. Recall 0.931, precision 0.834. |
| Best page types | Documentation 0.927, articles 0.920 | Per-type scores on the same set. |
| Other systems on the same benchmark | rs-trafilatura 0.893 (published), Trafilatura 2.3.1 0.863 (re-run here), Trafilatura 0.833 (benchmark README), Readability 0.726 (published) | The Trafilatura figure in the README does not reproduce: the benchmark paper has 0.841 and our re-run of the current version gives 0.863. |
How should these numbers be read?
Each figure is a median or a total from a named run on a named machine, and the scope column says what was and was not included. Process start, network time and memory are counted where the scope says they are.
Most runs are on one shared 4-core machine, so read the ratios and the pass counts, not the last digit. A conversion time is not an end-to-end time: a page fetched over the network costs the network too.
These are speed and footprint figures. Where a job also has a quality measure, it is shown beside the speed, and the benchmark page lists what was not measured.
Run it on your own files.
The harnesses are public so you can repeat any of this on your own traffic.