Crawl
Crawl a site starting from one or more seed URLs, following links up to a configurable depth.
Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Body
Starting URLs to crawl from
1Cap on total pages crawled
x <= 50Max link depth from seeds
0 <= x <= 5Restrict crawl to seed domains
Output formats per page (default ["markdown"])
1html, markdown, json Structured extraction config, requires json output
- Option 1
- Option 2
- Option 3
- Option 4
Max crawl duration, in ms
x <= 300000Skip the cheap direct/stealth tiers and go straight to the strongest (proxy/Camoufox) access tier for every page crawled. Requires a paid plan.
Response
Crawl completed inline.
Common envelope every workflow endpoint (crawl, map, interact, search) returns: request/step metadata plus a kind-specific result.
crawl, map, interact, search inline completed, partial, failed Kind-specific payload -- see the endpoint's example.