Product · Map
Every URL on a site, before you fetch a single page.
Webfold discovers a site's pages from its sitemap first. If the sitemap is thin or missing, it falls back to a bounded crawl. You get the list. Then you decide what to fetch.
See it work
Try it from wherever you work.
1webfold index map https://example.com --max 50
2webfold index resolve https://agency.example/guess-a-topic
What you get
What Map does.
Sitemap first
Reads sitemaps and sitemap indexes, including last-modified, change frequency and priority when present.
Crawl fallback
When the sitemap is thin or absent, it falls back to a bounded crawl of same-origin links.
Resolve a guess
`index resolve` takes a guessed topic address on an unstable host, such as a government site, and finds the canonical one.
No page content fetched
Mapping returns URLs only, so it is cheap to run before a real crawl.
How it works
Three steps.
- 1
Read the sitemap
Fetch and parse sitemap files.
- 2
Fall back
Crawl same-origin links if the sitemap is thin.
- 3
Return URLs
A flat list you can filter before fetching.
Limits up front.
Know the limits
- Only same-origin URLs are discovered.
- A site with no sitemap and heavy script navigation may map poorly.
Keep reading.
FAQ
Questions.
Does it fetch page content?
No. Map returns addresses only.
Use Map with Webfold.
Run the command-line tool today. Join the waitlist for the hosted API.