# Lumen Crawler

> Task API for agents. Create a crawl, poll until it finishes, then read Markdown, a site map, doors, and optional cleaned JSON.

**When to use Lumen Crawler:** Use it when an agent or application needs Playwright-rendered pages from a public website as clean Markdown or typed JSON, with robots and SSRF fail-closed.

**When not to use Lumen Crawler:** Do not use it to crawl private networks, authenticated targets, or off-universe hosts in the same job. `follow_doors` is unsupported. There are no webhooks or search.

**Preferred call pattern:** Create a key at the console. POST /v1/crawls with Bearer `lc_live_…` and a unique Idempotency-Key. Expect HTTP 202. Poll GET /v1/crawls/{id} until status is completed, cancelled, or failed. Then GET pages with include=markdown. On 402, check usage or raise the plan; do not retry the same max_pages. Off-universe links are doors — start a new crawl to enter them.

## Start here

- [Lumen Crawler documentation](https://crawler-docs.lumen-labs.ai/docs/v2.md): Overview and product guarantees
- [Quickstart](https://crawler-docs.lumen-labs.ai/docs/v2/quickstart.md): Create a key, start a crawl, poll, read Markdown
- [Installation](https://crawler-docs.lumen-labs.ai/docs/v2/installation.md): Console account and API keys
- [Lumen Crawler OpenAPI 3.1 specification](https://crawler-docs.lumen-labs.ai/docs/openapi.json): Machine-readable REST contract
- [Console](https://crawlapp.lumen-labs.ai): Usage, keys, billing
- [Crawl API](https://crawler.lumen-labs.ai): Production host

## Task-shaped entry points

- [Create and poll](https://crawler-docs.lumen-labs.ai/docs/v2/create-and-poll.md): POST 202, status, cancel, idempotency
- [Pages and map](https://crawler-docs.lumen-labs.ai/docs/v2/pages-and-map.md): Markdown bodies and the in-universe graph
- [Doors and universe](https://crawler-docs.lumen-labs.ai/docs/v2/doors.md): Off-site links are listed, never fetched
- [Cleanup JSON](https://crawler-docs.lumen-labs.ai/docs/v2/cleanup.md): Typed page and site rollups
- [FAQ](https://crawler-docs.lumen-labs.ai/docs/v2/faq.md): Keys, 402 quota, robots, doors, retention

## Agent interfaces

- [Copy-paste prompts](https://crawler-docs.lumen-labs.ai/docs/v2/prompts.md): Cursor MCP JSON and a paste-ready agent prompt
- [Docs MCP](https://crawler-docs.lumen-labs.ai/docs/v2/docs-mcp.md): Unauthenticated Streamable HTTP MCP for this documentation
- [Docs MCP discovery](https://crawler-docs.lumen-labs.ai/docs/.well-known/mcp.json): Machine-readable Docs MCP endpoint metadata
- [Docs skill](https://crawler-docs.lumen-labs.ai/docs/skill.md): Short skill for coding agents
- [Error catalog](https://crawler-docs.lumen-labs.ai/docs/v2/errors.md): RFC 7807 problem types including 402 quota_exceeded

## Area indexes

- [This agent index](https://crawler-docs.lumen-labs.ai/docs/llms.txt): Product-level discovery and usage guidance
- [V2 docs](https://crawler-docs.lumen-labs.ai/docs/llms/v2.txt): Current Lumen Crawler documentation
- [Full V2 markdown feed](https://crawler-docs.lumen-labs.ai/docs/llms-full.txt): Consolidated public documentation for indexing