Lumen Crawler
Reference

For the complete Lumen Crawler documentation index, see llms.txt. A markdown version of any docs page is available by appending .md to its URL. Docs MCP: /docs/mcp.

Errors

RFC 7807 problem+json types returned by the crawl API.

Errors use Content-Type: application/problem+json:

{
  "type": "https://lumen.invalid/errors/quota_exceeded",
  "title": "Quota exceeded",
  "status": 402,
  "detail": "max_pages does not fit remaining included pages"
}

type URLs are identifiers, not human docs pages. Match on the code suffix.

CodeHTTPWhen
unauthorized401Missing or invalid Bearer or JWT
unregistered_key401Production key not in the ledger
quota_exceeded402max_pages does not fit remaining included pages
plan_inactive402Account past_due or canceled
invalid_url400Bad JSON, unparseable URL, or bad include
invalid_domain400allowed_domains is not eTLD+1
ssrf_blocked400Private, metadata, or non-https in production
missing_idempotency_key400POST create without the header
follow_doors_unsupported422follow_doors: true
idempotency_conflict409Same key, different body
conflict409Cancel of a finished crawl
cleanup_not_ready409GET cleaned while cleanup is queued or running
cleanup_unavailable404Cleanup skipped, failed, or missing
not_found404Unknown crawl, key, or billing customer

Foreign crawl ids return 404, never 403.

For agents

Treat 402 as a quota or plan problem, not a retryable 5xx. Treat 409 cleanup_not_ready as wait. Treat 409 idempotency_conflict as a client bug: new body needs a new Idempotency-Key.

Edit on GitHub

Last updated on