Skip to content
Webfold Docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

Rendered pages

Extract pages whose content only appears after JavaScript runs, using a Chromium you control.

Webfold does not launch a browser and does not need one. Fetching, extraction, layout, paint and its own CDP server are all native. For the rare page whose content only exists after its own JavaScript runs, scrape --render connects to a Chromium that you run, over the Chrome DevTools Protocol.

WEBFOLD_BROWSER_CDP_URL=http://127.0.0.1:9222 \
  webfold scrape https://example.com --render

webfold scrape https://example.com --render --browser ws://127.0.0.1:9222/devtools/browser/<id> \
  --recovery-json --screenshot

Without an endpoint the command fails immediately with browser tier unavailable: no CDP endpoint configured.

Choose the cheapest option first

  1. Do not render. Native extraction first.
  2. A browser you already run, started with --remote-debugging-port, as a sidecar or shared service.
  3. Last resort: a container. Run Chromium as a container next to Webfold, with CDP on port 9222, and point Webfold at it. Webfold never starts it.

Configuration

Setting Flag or environment variable Default
CDP endpoint --browser <endpoint> / WEBFOLD_BROWSER_CDP_URL none
Proxy listens on WEBFOLD_BROWSER_PROXY_BIND (host:port) 127.0.0.1:0
Proxy address the browser dials WEBFOLD_BROWSER_PROXY_ADVERTISE (host:port) the bound address
Extra proxy clients (IPs or CIDRs) WEBFOLD_BROWSER_PROXY_PEERS none

The endpoint is ws://host:port/devtools/browser/<id>, or http://host:port, which is resolved through /json/version. wss:// and https:// are not supported in this build, so keep the browser on loopback or a private network.

Readiness and interaction flags

Flag Effect
--wait-for text Wait until the text appears (repeatable).
--wait-selector css Wait until the selector matches (repeatable).
--scroll-document Scroll the document, within the action budget.
--expand-disclosures css Open ordinary disclosure panels (such as FAQ accordions), within the action budget.
--document-selector css Restrict extraction to the observed article regions that match.
--include-link-labels Include visible labels of public HTTP(S) link buttons outside forms.
--include-aria-hidden, --include-form-text Keep ARIA-hidden text or non-control form text when you ask for it.
--screenshot Keep a viewport screenshot with the capture.

What stays enforced

  • Every acquisition creates a fresh browser context, disposed when the connection closes. Nothing is shared with other pages, saved sessions or profiles.
  • The context’s proxy is Webfold’s validating proxy. It resolves each upstream host itself, refuses private, loopback, link-local, metadata, CGNAT and multicast addresses, connects to the address it checked, and bounds connections, bytes and time.
  • A deadline and limits on connections, bytes, DOM nodes, frames and interactions bound the adapter.