For the complete Lumen Crawler documentation index, see llms.txt. A markdown version of any docs page is available by appending .md to its URL. Docs MCP: /docs/mcp.
Crawls API
Create, poll, and read crawls
List crawls for the API key tenant, or for the account when a console JWT is used.
Authorization
BearerAuth Crawl key (lc_live_…) for /crawls, or Cognito JWT for /account and /billing.
In: header
Query Parameters
1 <= value <= 10020Response Body
application/json
application/problem+json
curl -X GET "https://example.com/crawls"{ "items": [ { "id": "string", "status": "queued", "created_at": "2019-08-24T14:15:22Z", "started_at": "string", "finished_at": "string", "elapsed_ms": 0, "counts": { "property1": 0, "property2": 0 }, "config": { "url": "string", "allowed_domains": [ "string" ], "max_pages": 0, "follow_doors": true, "cleanup": true }, "cleanup": { "status": "skipped", "model": "string", "pages_cleaned": 0, "error": "string" }, "stop_reason": "string" } ], "next_cursor": "string"}Accept a crawl. Returns 202. Requires Idempotency-Key. follow_doors true is 422.
Authorization
BearerAuth Crawl key (lc_live_…) for /crawls, or Cognito JWT for /account and /billing.
In: header
Header Parameters
Request Body
application/json
TypeScript Definitions
Use the request body type in TypeScript.
Response Body
application/json
application/problem+json
application/problem+json
application/problem+json
application/problem+json
application/problem+json
curl -X POST "https://example.com/crawls" \ -H "Idempotency-Key: string" \ -H "Authorization: string" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "max_pages": 50, "cleanup": true }'{ "id": "string", "status": "queued", "created_at": "2019-08-24T14:15:22Z", "started_at": "string", "finished_at": "string", "elapsed_ms": 0, "counts": { "property1": 0, "property2": 0 }, "config": { "url": "string", "allowed_domains": [ "string" ], "max_pages": 0, "follow_doors": true, "cleanup": true }, "cleanup": { "status": "skipped", "model": "string", "pages_cleaned": 0, "error": "string" }, "stop_reason": "string"}Return crawl status, counts, config, cleanup, and stop_reason.
Authorization
BearerAuth Crawl key (lc_live_…) for /crawls, or Cognito JWT for /account and /billing.
In: header
Path Parameters
Response Body
application/json
application/problem+json
curl -X GET "https://example.com/crawls/string"{ "id": "string", "status": "queued", "created_at": "2019-08-24T14:15:22Z", "started_at": "string", "finished_at": "string", "elapsed_ms": 0, "counts": { "property1": 0, "property2": 0 }, "config": { "url": "string", "allowed_domains": [ "string" ], "max_pages": 0, "follow_doors": true, "cleanup": true }, "cleanup": { "status": "skipped", "model": "string", "pages_cleaned": 0, "error": "string" }, "stop_reason": "string"}List page records. Pass include=markdown,cleaned for bodies. cursor is the last urlHash.
Authorization
BearerAuth Crawl key (lc_live_…) for /crawls, or Cognito JWT for /account and /billing.
In: header
Path Parameters
Query Parameters
Comma-separated fields to include (markdown, cleaned)
1 <= value <= 10050Response Body
application/json
curl -X GET "https://example.com/crawls/string/pages"{ "items": [ { "url": "string", "canonical_url": "string", "title": "string", "status_code": 0, "ok": true, "fetched_at": "string", "content_hash": "string", "extraction": "string", "markdown": "string", "cleaned": {} } ], "next_cursor": "string"}Return the in-universe node/edge graph. Doors are omitted.
Authorization
BearerAuth Crawl key (lc_live_…) for /crawls, or Cognito JWT for /account and /billing.
In: header
Path Parameters
Response Body
application/json
curl -X GET "https://example.com/crawls/string/map"{ "nodes": [ {} ], "edges": [ {} ]}List off-universe URLs discovered but not fetched. No bodies.
Authorization
BearerAuth Crawl key (lc_live_…) for /crawls, or Cognito JWT for /account and /billing.
In: header
Path Parameters
Response Body
application/json
curl -X GET "https://example.com/crawls/string/doors"{ "items": [ {} ], "next_cursor": "string"}Return the site-level cleaned JSON rollup. 409 while cleanup runs; 404 if skipped.
Authorization
BearerAuth Crawl key (lc_live_…) for /crawls, or Cognito JWT for /account and /billing.
In: header
Path Parameters
Response Body
application/json
application/problem+json
application/problem+json
curl -X GET "https://example.com/crawls/string/cleaned"{}Cancel a queued or running crawl. Finished crawls return 409.
Authorization
BearerAuth Crawl key (lc_live_…) for /crawls, or Cognito JWT for /account and /billing.
In: header
Path Parameters
Response Body
application/json
application/problem+json
curl -X POST "https://example.com/crawls/string/cancel"{ "id": "string", "status": "queued", "created_at": "2019-08-24T14:15:22Z", "started_at": "string", "finished_at": "string", "elapsed_ms": 0, "counts": { "property1": 0, "property2": 0 }, "config": { "url": "string", "allowed_domains": [ "string" ], "max_pages": 0, "follow_doors": true, "cleanup": true }, "cleanup": { "status": "skipped", "model": "string", "pages_cleaned": 0, "error": "string" }, "stop_reason": "string"}Last updated on