Discover URLs on a web site

BFS URL discovery starting from url, staying on the same origin, up to max_pages. Returns a list of discovered URLs without fetching full content.

POST/v1/web/map

Authorization

AuthorizationBearer token (JWT) · headerrequired

Bearer API key issued for your account. Send Authorization: Bearer <key>.

Request body

requiredapplication/json

urlstring<uri>required

max_pagesinteger

Maximum number of URLs to discover

default: 50

Responses

200

Discovered URLs

urlsstring<uri>[]required

400

Bad request

string

401

Missing or invalid bearer token

string

403

Billing or plan entitlement blocked the request

string

502

Upstream service (arXiv, YouTube, web page) failed

string

Try it

Server

Base URLCustom base URLOverrides the selected server when set.

Authorization

Bearer token (JWT)Remember on this deviceInclude my values in samples

Bodyapplication/json

urlstring*max_pagesintegerMaximum number of URLs to discover

Request

curl -X POST 'https://api.webfold.ai/v1/web/map' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "https://example.com",
  "max_pages": 50
}'

import requests

response = requests.post(
    "https://api.webfold.ai/v1/web/map",
    headers={
        "Authorization": "Bearer YOUR_TOKEN",
        "Accept": "application/json",
        "Content-Type": "application/json"
    },
    data="{\n  \"url\": \"https://example.com\",\n  \"max_pages\": 50\n}",
)

const response = await fetch("https://api.webfold.ai/v1/web/map", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_TOKEN",
    "Accept": "application/json",
    "Content-Type": "application/json"
  },
  body: "{\n  \"url\": \"https://example.com\",\n  \"max_pages\": 50\n}"
});

Response

{
  "urls": [
    "http://example.com"
  ]
}

"string"

"string"

"string"

"string"