Lumen Crawler
Start

For the complete Lumen Crawler documentation index, see llms.txt. A markdown version of any docs page is available by appending .md to its URL. Docs MCP: /docs/mcp.

Lumen Crawler

Browser-rendered crawls. Clean Markdown. Structured JSON. A task API for agents.

Lumen Crawler is a task API: you point it at a URL, it renders pages in headless Chromium, and you pull Markdown, a site map, doors, and optional cleaned JSON.

Hosts

SurfaceURL
Crawl APIhttps://crawler.lumen-labs.ai
Consolehttps://crawlapp.lumen-labs.ai
Docsthis site
Docs MCP/docs/mcp

What you get

  • Playwright loads each page the way a browser does, so client-side content is present.
  • Robots rules for LumenCrawler fail closed. Private and metadata targets are blocked at fetch time.
  • Main content per page as Markdown, plus an in-universe map.
  • Typed JSON on request after cleanup finishes.

The loop

  1. Create an API key in the console. The secret is shown once.
  2. POST /v1/crawls with Authorization: Bearer lc_live_… and an Idempotency-Key. The API returns 202.
  3. Poll GET /v1/crawls/{id} until status is completed, cancelled, or failed.
  4. Read GET /v1/crawls/{id}/pages?include=markdown.

For agents

Create with a unique Idempotency-Key. Poll until a terminal status. Off-universe links are doors — start a new crawl to enter them. On 402, do not retry the same max_pages.

Start

Edit on GitHub

Last updated on

Quickstart

Next Page

On this page

What you getThe loopStart