Use case
Data pipelines
Feed web or document data into the tools you use.
The problem.
Data teams want pages as tables. They want files as records. They want the format their warehouse reads. They do not want a scraper per site.
How Webfold helps.
Step 1
Crawl or batch
Process a site or a folder of saved pages.
webfold batch ./pagesStep 2
Emit records
Output NDJSON, one record per page.
webfold crawl ./pages --chunk 4000Step 3
Sink to storage
Write JSONL, Parquet or Iceberg.
webfold sink iceberg < records.ndjsonProducts used.
Worth knowing.
Note
- Check extraction on a sample of your own sources before running at volume.
Try it on your own data.
Run the command-line tool today. Hosted API access is coming.