Benchmark · Directional
PDF extraction against other extractors
On 200 public PDFs, Webfold scores 0.760 against liteparse's 0.883. We lose to liteparse on tables. It leads on headings too. Stray line breaks fell from 28.68 to 2.35 a page. The samples are small.
Webfold's PDF parser against the CPU extractors. Six public test sets. It trails liteparse.
What question does this benchmark ask?
Does Webfold's PDF parser keep reading order? Does it keep tables? Does it keep headings? Where does it rank?
How was it run?
- Six test sets: a seeded sample of 144 PDFs from olmOCR-bench (ODC-BY), the 200 PDFs of opendataloader-bench with ground-truth Markdown, 60 Federal Register of Legislation PDFs (48 with a Word reference), the 14 PDFs of the py-pdf read corpus, the Noisy OCR Dataset (12 pages in 3 noise conditions) and Webfold's 8 test fixtures.
- Tools: liteparse 2.18.0 (Markdown and text, with and without OCR), PyMuPDF and pymupdf4llm 2.1, pdfminer.six 20260107, pdfplumber 0.11.10, pypdf 7.0.1, pypdfium2 5.15.2, Poppler pdftotext 24.02.0, MarkItDown 0.1.9 and Tesseract 5.3.4.
- Webfold is measured before the PDF parser change (superdocuments 2aaec131, the revision Webfold pinned then) and after it. The change is now in superdocuments main.
- Scorers: olmOCR-bench's share of tests passed, opendataloader-bench's reading order (NID), table (TEDS) and heading (MHS) scores, a stray line-break count we defined, and character error rate. Each benchmark's own scorer was used where it has one.
- The text rows use the native text layer only, with no OCR. The OCR rows are marked.
What are the results?
| Measurement | Webfold | Webfold before | liteparse, Markdown | Best other tool |
|---|---|---|---|---|
| olmOCR-bench, share of tests passed (144 PDFs) | 31.8% | 28.0% | 45.9% | pymupdf4llm 38.9% |
| olmOCR-bench, table tests (148) | 12.8% (19) | 2.0% (3) | 54.1% (80) | pymupdf4llm 66.9% (99) |
| olmOCR-bench, multi-column tests (89) | 66.3% (59) | 48.3% (43) | 76.4% (68) | pymupdf4llm 67.4% (60) |
| opendataloader-bench, overall (200 PDFs) | 0.760 | 0.727 | 0.883 | pymupdf4llm 0.869 |
| opendataloader-bench, reading order (NID) | 0.874 | 0.860 | 0.911 | pymupdf4llm 0.909 |
| opendataloader-bench, tables (TEDS) | 0.376 | 0.244 | 0.818 | pymupdf4llm 0.773 |
| opendataloader-bench, headings (MHS) | 0.574 | 0.520 | 0.824 | pymupdf4llm 0.783 |
| Stray line breaks per page, 200 PDFs (lower is better) | 2.35 | 28.68 | 2.06 | pymupdf4llm 0.78 |
| Paragraphs run together per page, same PDFs | 0.17 | 0.04 | 1.45 | pymupdf4llm 0.10 |
| Reference words found, same PDFs | 0.83 | 0.83 | 0.88 | pdftotext 0.91 |
| Federal Register, 42 born-digital PDFs: stray breaks, run together, words found | 0.82, 2.53, 0.92 | 19.83, 0.54, 0.94 | 1.27, 4.84, 0.88 | pymupdf4llm 0.01, 1.23, 0.77 |
| Same set with the table fix in superdocuments#50 (library, not in a release) | 0.78, 1.07, 0.87 | not run | not run | not run |
| opendataloader-bench with that fix: overall, reading order, tables, headings | 0.772, 0.880, 0.438, 0.586 | not run | not run | not run |
| olmOCR-bench, scanned sets with OCR, share of tests passed | 47.2% | not run | 35.5% | Tesseract alone 47.1% |
| Noisy OCR Dataset, mean character error rate | 0.068 | not run | 0.307 | Tesseract alone 0.066 |
Webfold's PDF parser is better than it was and it is not the best. Stray line breaks inside paragraphs fell from 28.68 to 2.35 per page. Table tests passed on olmOCR-bench went from 3 of 148 to 19.
liteparse is ahead on tables (TEDS 0.818 against 0.376), headings (MHS 0.824 against 0.574), reading order and overall score. pymupdf4llm is ahead on tables and has fewer stray line breaks. We lose on tables and headings.
On the olmOCR-bench sample Webfold's native text scores 31.8%, behind liteparse, pymupdf4llm, pdftotext and pdfminer. Part of that is the scorer: tests that expect text to be absent pass on empty output, which is why pdftotext scores 94.3% on scanned tiny-text pages.
On scanned pages the OCR route matches Tesseract run alone: 47.2% against 47.1% on the scanned olmOCR-bench sets, and 0.068 against 0.066 mean character error on the Noisy OCR Dataset. liteparse's OCR is behind (35.5%, and 0.307).
We corrected the Federal Register comparison. An earlier version of this page showed 0.16 for runs-together, and that output read only a quarter of the words.
Like for like, the parser before the line-break work ran paragraphs together 0.54 times a page. It had 19.83 stray breaks a page.
The line-break work removed the stray breaks. It ran paragraphs together 2.53 times a page.
Reading borderless Word tables as tables cuts that to 1.07. It is merged in superdocuments. It is not in a Webfold release yet.
What does this not show?
Know the limits
- The samples are small: 144 PDFs, 200, 60, 14 and 36 pages. Intervals are in the retained tables. Ranks within a few points are not separated.
- Tests that expect text to be absent pass on empty output. Read the per-type scores before the overall olmOCR-bench score.
- No layout model, GPU or vision-language model was used. ParseBench, Docling, marker and MinerU were not run.
- The two math sets need LaTeX output. Only liteparse produces any, and only on arXiv math. Webfold scores 0 there.
- The Federal Register PDFs are one publisher's output. Its Word version is a paragraph reference, not a pixel-exact one.
- The OCR rows use a Webfold command-line build whose revision the result files do not record. Speed was measured on a loaded shared machine and is not reported here. The Webfold command line was not rebuilt for the final parser commit.
- liteparse's published scores differ between sources (22.4% and 39.6% on olmOCR-bench). We report only our own runs.
Which other benchmarks are there?
- HTML to Markdown conversion Measured
- Document conversion Measured
- Main-content extraction quality Measured
- Web archive extraction Measured
- Footprint of a scrape or crawl Directional
- PII detection against Presidio Measured
- Localisation extraction Directional
- PDF text extraction with OCR Directional
- LaTeX conversion with BibTeX Directional
- A whole site to Markdown Directional
- Agent-browser compatibility (Lexbench) Measured
- Headless engine over CDP Directional
- Search-results parsing Directional
- Folder search Directional
Check it on your own pages.
Run the command-line tool on your content.