TECH

Crawl4AI – The Future Of Asynchronous Web Crawling For AI

Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications.

Looking for the synchronous version? Check out README.sync.md. You can also access the previous version in the branch V0.2.76.

Try It Now!

✨ Play around with this

✨ Visit our Documentation Website

Features

🆓 Completely free and open-source
🚀 Blazing fast performance, outperforming many paid services
🤖 LLM-friendly output formats (JSON, cleaned HTML, markdown)
🌍 Supports crawling multiple URLs simultaneously
🎨 Extracts and returns all media tags (Images, Audio, and Video)
🔗 Extracts all external and internal links
📚 Extracts metadata from the page
🔄 Custom hooks for authentication, headers, and page modifications before crawling
🕵️ User-agent customization
🖼️ Takes screenshots of the page
📜 Executes multiple custom JavaScripts before crawling
📊 Generates structured output without LLM using JsonCssExtractionStrategy
📚 Various chunking strategies: topic-based, regex, sentence, and more
🧠 Advanced extraction strategies: cosine clustering, LLM, and more
🎯 CSS selector support for precise data extraction
📝 Passes instructions/keywords to refine extraction
🔒 Proxy support for enhanced privacy and access
🔄 Session management for complex multi-page crawling scenarios
🌐 Asynchronous architecture for improved performance and scalability

Installation

Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.

Using Pip

Choose the installation option that best fits your needs:

Basic Installation

For basic web crawling and scraping tasks:

pip install crawl4ai

By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.

👉 Note: When you install Crawl4AI, the setup script should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:

Through the command line:

playwright install

For more information click here.

Varshini

Varshini is a Cyber Security expert in Threat Analysis, Vulnerability Assessment, and Research. Passionate about staying ahead of emerging Threats and Technologies.