01 / Async by design
Start now, collect later.
Enqueue a crawl and keep your application responsive while the worker processes each page.
Product / Crawl
Traverse whole sites with control.
Start an asynchronous crawl, monitor its progress, and collect normalized results across a site.
curl -X POST https://api.beecrawl.dev/crawl \
-H "X-Web-Extract-Api-Key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","limit":100,"maxDepth":2}'Features // Product / Crawl
Turn a starting URL into a controlled, observable collection of pages.
02 / Crawl controls
Set the boundary.
Choose page limits, discovery depth, subdomain behavior, and query parameter handling.
03 / Observable results
Know what happened.
Poll status for progress, completed pages, failures, and the next slice of results.
How it works //
From request
to ready data.
01
Enqueue a starting URL.
02
Discover and process pages.
03
Poll status and page through results.
{
"requestId": "req_01H...",
"status": "ready",
"data": "useful web content",
"metadata": { "provider": "beecrawl" }
}FAQ //
Everything you need
to get moving.
Is Crawl synchronous?
No. Crawl returns a job ID so you can monitor progress and retrieve results after processing begins.
How do I control crawl size?
Use limit and maxDepth, plus the subdomain and query parameter options, to keep the crawl within a known boundary.