Crawl4AI – The Future Of Asynchronous Web Crawling For AI

Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications.

Try It Now!

✨ Play around with this

✨ Visit our Documentation Website

Features

• 🆓 Completely free and open-source
• 🚀 Blazing fast performance, outperforming many paid services
• 🤖 LLM-friendly output formats (JSON, cleaned HTML, markdown)
• 🌍 Supports crawling multiple URLs simultaneously
• 🎨 Extracts and returns all media tags (Images, Audio, and Video)
• 🔗 Extracts all external and internal links
• 📚 Extracts metadata from the page
• 🔄 Custom hooks for authentication, headers, and page modifications before crawling
• 🕵️ User-agent customization
• 🖼️ Takes screenshots of the page
• 📜 Executes multiple custom JavaScripts before crawling
• 📊 Generates structured output without LLM using JsonCssExtractionStrategy
• 📚 Various chunking strategies: topic-based, regex, sentence, and more
• 🧠 Advanced extraction strategies: cosine clustering, LLM, and more
• 🎯 CSS selector support for precise data extraction
• 📝 Passes instructions/keywords to refine extraction
• 🔒 Proxy support for enhanced privacy and access
• 🔄 Session management for complex multi-page crawling scenarios
• 🌐 Asynchronous architecture for improved performance and scalability

Installation

Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.

Using Pip

Choose the installation option that best fits your needs:

Basic Installation

For basic web crawling and scraping tasks:

pip install crawl4ai

By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.

👉 Note: When you install Crawl4AI, the setup script should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:

1. Through the command line:

playwright install

For more information click here.