The terms, in plain words.

Terms.

Boilerplate

The parts of a page that are not its content: menus, headers, footers, cookie banners, related links and ads.

CDP (Chrome DevTools Protocol)

The protocol Playwright and Puppeteer use to control a browser. Webfold implements a subset so those tools can connect to it. Learn more →

Chunking

Splitting a long text into pieces of a set size for indexing and retrieval. Learn more →

Headless browser

A browser that runs without a visible window, used to load pages and run their scripts from a program. Learn more →

HTML to Markdown

Turning a web page's markup into plain Markdown text, keeping headings, lists, tables and links, and dropping scripts and layout. Learn more →

MCP (Model Context Protocol)

An open protocol that lets an AI assistant call external tools. Webfold ships an MCP server. Learn more →

OCR

Optical character recognition: reading text out of an image, such as a scanned PDF page. Learn more →

Page furniture (chrome)

Another name for boilerplate that repeats across the pages of a site. Webfold counts repeated blocks across a crawl and drops them. Learn more →

Prompt injection

Hidden or hostile text on a page that tries to steer a model that reads it. Extraction does not remove it, so treat fetched text as untrusted. Learn more →

RAG (retrieval-augmented generation)

Giving a language model relevant passages from your own data at question time. It depends on clean, well-chunked source text. Learn more →

robots.txt

A file where a site states which automated visitors may fetch which paths. Polite crawlers read it first.

SERP

A search engine results page. Webfold parses each engine's results into one common shape. Learn more →

Sitemap

A file where a site lists its pages, often with last-changed dates. Faster than crawling when it exists. Learn more →

Token efficiency

Giving a model only the text it needs, so you pay for fewer tokens. Removing page furniture is the first step.

WARC

The standard file format for web archives. It stores each request and response so a page can be examined or replayed later. Learn more →

Web crawler

A program that follows links from a starting page to discover and fetch many pages of a site. Learn more →