Crawl4AI

Crawl4AI turns the web into clean, LLM ready Markdown for retrieval augmented generation, agents and data pipelines. It was written because the available web-to-Markdown services wanted an account, an API token and a fee, and still under-delivered.

The library is Python and asynchronous, driving a browser pool with caching and few hops. Markdown generation is pluggable: heuristic filtering strips noise, BM25 pulls the core content, and page links become a numbered reference list with clean citations. There is also a crwl command line tool and a Docker server for the same work.

Beyond Markdown, the crawler is built for extraction under control.

• Structured extraction: schema-based CSS and XPath selectors, or LLM-driven extraction with open source and proprietary models alike.
• Chunking strategies: topic, regex and sentence-level chunking, with cosine similarity to surface the passages a query needs.
• Browser integration: managed and remote browsers over the Chrome DevTools Protocol, persistent profiles, sessions and authenticated proxies.
• Adaptive crawling: learns site patterns and explores only what matters, with deep crawl strategies and a prefetch mode for URL discovery.
• Crash recovery: resume state and state-change callbacks keep long deep crawls restartable.

Install with pip, then run the setup and doctor commands; Playwright's Chromium can be installed by hand if the browser misbehaves. Recent releases have concentrated on hardening the Docker API server, which now enables authentication by default and binds to loopback unless given a token.