---
search:
  tags:
    - Web
    - POST
seo:
  description: >-
    BFS crawl starting from url, staying on the same origin, up to max_pages and
    max_depth.… Reference for the POST /v1/web/crawl endpoint in the Webfold
    API.
sidebar:
  label: Crawl a web site
  badge: POST
title: Crawl a web site
type: openapi-operation
---
BFS crawl starting from `url`, staying on the same origin, up to
`max_pages` and `max_depth`. Each page is stored in the local web index
and returned as Markdown.

`POST /v1/web/crawl`

**Request body** (`application/json`, required)

- `url` (string<uri>, required)
- `max_pages` (integer) — Maximum number of pages to fetch
- `max_depth` (integer) — Maximum link depth from the seed

Request body example:

```json
{
  "url": "https://example.com",
  "max_pages": 10,
  "max_depth": 2
}
```

**Responses**

- `200` — Crawled pages
- `400` — Bad request
- `401` — Missing or invalid bearer token
- `403` — Billing or plan entitlement blocked the request
- `502` — Upstream service (arXiv, YouTube, web page) failed

Response example, 200:

```json
{
  "pages": [
    {
      "url": "http://example.com",
      "title": "string",
      "markdown": "string"
    }
  ]
}
```