Crawling and indexing

Crawl sites, fold shared chrome once, build a local index and search it.

Crawl

webfold crawl <dir|url> [--min n] [--max n] [--chunk n]

crawl reads each page once, counts how many pages each block signature appears on, and omits blocks whose frequency crosses the template threshold. The report states how many blocks were shared chrome and how many content blocks were kept.

Index

webfold index crawl   <seed-url> [--max n] [--depth n] [--out file] [--http-cache dir]
webfold index search  <seed-url> <query> [--max n] [--depth n] [--top n]
webfold index map     <seed-url> [--max n] [--no-crawl-fallback]
webfold index resolve <guessed-url> [--max n]

Command	Does
`index crawl`	Crawl from a seed and serialise the extracted pages as JSONL.
`index search`	Crawl from a seed, build the in-memory index, and rank pages for a query.
`index map`	Discover same-origin URLs: sitemap first, then a bounded crawl fallback when the sitemap is thin or missing.
`index resolve`	Guess a topic URL on an unstable host (for example a government site) and return the canonical one.

Ranking uses a term-frequency score: query-term boundary matches normalised by the log of page length. It is a lexical index, not an embedding index.

Offline search over saved HTML

webfold search "rotate wal segment" --corpus ./saved-pages --count 10
webfold folder-search . "rotate wal segment" -n 5

--corpus indexes a directory of saved HTML with no live fetch. folder-search searches a local folder; add --semantic with WEBFOLD_EMBED_BASE_URL set to use an embedding endpoint you run.

Batch and chunk

webfold batch ./dir-of-html-files
webfold chunk page.html --size 4000 --jsonl
webfold sink jsonl < records.ndjson     # also: parquet, iceberg

chunk accepts HTML, Markdown, text, PDF or a URL and emits retrieval-ready chunks. sink writes NDJSON records to JSONL, Parquet or Iceberg.