---
title: "Corpusforge"
description: "A corpus-governance pipeline: extract, scan and redact, build a deduplicated corpus and record an AI bill of materials."
---

Corpusforge is one command-line tool for the path from raw sources to a corpus you can govern: **extract, scan and redact, ontology, corpus, adapt, bill of materials**.

## Commands

| Command | Does |
| --- | --- |
| `corpusforge scan <path>` | Find secrets and personal data, offline. Exits 2 if any are found, so it gates CI |
| `corpusforge redact <in> [-o out]` | Replace every detected span with `[ENTITY_TYPE]` |
| `corpusforge extract <args>` | Extract text. PDFs run natively; other formats go through the extractor |
| `corpusforge ontology <args>` | Tag content with the ontology tagger |
| `corpusforge corpus <in> <out> [--target F]` | Chunk, deduplicate and select a coreset |
| `corpusforge adapt <traces> <out.jsonl> --kind K --name N [--input-field F]` | Turn agent trace files into redacted, labelled rows |
| `corpusforge aibom [-o out]` | Generate an AI bill of materials |
| `corpusforge pipeline <src> <out>` | Extract, scan, redact, build the corpus and write the bill of materials, end to end |

```sh
corpusforge pipeline ./sources ./out
corpusforge adapt ./auto-traces ./router-witnesses.jsonl --kind tool_call --name route --input-field prompt --target 0.25
```

The adapter writes rows with `text`, `class`, `label`, `kind`, `name` and `trace_id`. Repeated inputs with conflicting labels are dropped instead of guessed.

## Identity terms

`scan` and `redact` also look for names you supply. Set `PII_IDENTITY_FILE` (one `term<TAB>ENTITY_TYPE` per line) or `PII_IDENTITY_TERMS`.

## Build

Corpusforge links the PII library as a path dependency, so check out `pii-svc` beside it:

```text
<workspace>/
  pii-svc/         # provides pii-core
  corpusforge/
```

:::note
Some stages still call external tools: non-PDF extraction, the ontology tagger and the bill of materials. Known gaps include Common Crawl ingestion, a crawl frontier, near-duplicate detection beyond the current method, shard output and language identification.
:::

## Hosted API

Two steps of the pipeline are also available over HTTP. `POST /v1/corpus/redact` redacts up to 100 records. `POST /v1/corpus/build` redacts if asked, splits each record into chunks of at most `chunk_chars` characters (preferring paragraph, line, then word boundaries) with optional overlap, and drops chunks whose normalised text was already seen. The rest of the pipeline, including the bill of materials, stays in the command line tool.

```sh
curl -s https://api.webfold.ai/v1/corpus/build \
  -H "Authorization: Bearer $WEBFOLD_API_KEY" -H "Content-Type: application/json" \
  -d '{"records":[{"id":"a","text":"..."}],"chunk_chars":2000,"overlap":200,"dedup":true,"redact":true}'
```