Compare · Microsoft Presidio
terredactal (pii-svc) vs Microsoft Presidio
terredactal is Webfold's PII detector. It ran about 470 times faster than Presidio without its language model. It ran about 1,400 times faster than Presidio's defaults. The 2,000 documents are generated by us. Presidio is ahead on place names in news text.
Presidio is Microsoft's open-source toolkit for finding personal data in text. It is a Python library that pairs patterns with a spaCy language model. terredactal is a Rust library that uses patterns with checksums. Both ran on the same 2,000 generated documents.
Now the comparison
terredactal (pii-svc) against Microsoft Presidio
| Aspect | terredactal (pii-svc) | Microsoft Presidio |
|---|---|---|
| Documents a second (Presidio defaults) | 110,200 | 80 |
| Documents a second (Presidio, language model removed) | 110,200 | 235 |
| Peak memory | 13 MB | 781 MB (135 MB without the language model) |
| Cold start | 16 ms | 2.2 s |
| F1 on the 7 shared types, relaxed span | 0.998 | 0.867 |
| F1 with Presidio's Australian recognizers on (11 types) | 0.997 | 0.924 |
| False alarms on lookalike numbers | 17 | 525 |
| Person names, 244 hand-written sentences (F1, exact span) | 0.960 (optional language-model tier) | 0.857 (spaCy), 0.821 (transformers) |
| Place names, same sentences (F1, exact span) | 0.974 | 0.884 (spaCy), 0.827 (transformers) |
| Speed with the language-model tier on (300 documents) | 1.9 documents a second, 1,167 MB | 52 a second, 1,043 MB (spaCy); 3.4 a second, 1,487 MB (transformers) |
| Names and places in the default tier | Not detected (they need the optional model tier) | Yes, from spaCy |
| Australian company number (ACN), 50 held-out labels | Recall 0.860, precision 1.000 (needs the words "ACN" or "company number" nearby) | Recall 0.900, precision 0.634 (reports bare nine-digit numbers) |
Where is each tool ahead?
Where Microsoft Presidio is ahead
- Names without extra setup. Presidio's spaCy model is in its defaults. terredactal needs its optional model tier. That tier is slower per CPU-second than both Presidio configurations we ran. It does about 2 documents a second in 1.2 GB.
- Many more recognizers out of the box. They cover many countries. It has a large community. It has Microsoft's name behind it.
- Place names in news text. On CoNLL-2003 Presidio's spaCy model has the higher exact-span F1 (0.775 against 0.701).
- The Australian company number when no keyword is near it. Presidio finds more of the bare numbers (recall 0.900 against 0.860). It pays with many false alarms.
Where terredactal (pii-svc) is ahead
- About 470 times the speed with Presidio's language model removed. About 1,400 times its defaults. A sixtieth of the memory.
- Roughly one thirtieth of the false alarms on lookalike numbers. Numbers are checked against their checksums. Context is checked too.
- Starts in 16 ms. It runs in-process or as a small service. Nothing to download. It fits request paths. It fits air-gapped sites.
- Australian identifiers have their check digits built in. That covers the tax file number. It covers the ABN. It covers Medicare.
How do I move from Microsoft Presidio to terredactal (pii-svc)?
| If you use | In terredactal (pii-svc) |
|---|---|
| presidio AnalyzerEngine().analyze(text) | the terredactal library or service: POST text, get typed spans and the masked text back |
What is the evidence?
Each figure states its dataset. It states its limits.
FAQ
Frequently asked questions
Is this a fair test?
It is a test on a corpus we generated, with identifiers that have valid check digits and hard negatives we chose, so it measures these recognizers on these patterns and says nothing about real-world text. We published the generator, the scorer and the command, and the span bugs the run exposed in terredactal are fixed in the open with before and after tables. Run it on your own data.
Does terredactal find names?
Not in the default tier: names and places need a language model. With the optional model tier on, it found names with an exact-span F1 of 0.960 on 244 hand-written sentences against 0.857 for Presidio's spaCy model, and it was ahead of Presidio's spaCy model on person names on the four public samples where exact spans are meaningful (CoNLL-2003, Gretel and both ai4privacy samples), but behind on places in CoNLL-2003. Presidio's transformers model scored higher on person names on the first 100 Gretel documents (0.712 against 0.689). That tier is slow, about 2 documents a second in 1.2 GB on our test machine, so it is a choice, not the default.
Why is Presidio so much slower?
It is a Python library calling a language model on every document, and loading that model takes seconds and hundreds of megabytes. With the model removed it is 235 documents a second, which is the fairer comparison for pattern-only detection.
Try it on your own pages.
Run the command-line tool. Join the waitlist for the hosted API.