Crawl4AI Tools - Web Scraping and Content Extraction

The source defines three Crawl4aiTools agents. It runs the default pruning agent and the explicit enable_crawl agent; the raw no-pruning agent is configured but not executed.

The source uses deprecated gpt-4o in all three agent configurations and defines agent_raw without executing it. Update the model IDs before running.

crawl4ai_tools.py

"""
Crawl4AI Tools - Web Scraping and Content Extraction

This example demonstrates how to use Crawl4aiTools for web crawling and content extraction.
Crawl4aiTools has a single enable_ flag: enable_crawl.

Run: `uv pip install crawl4ai` to install the dependencies
"""

from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.tools.crawl4ai import Crawl4aiTools

# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------


# Example 1: All functions enabled with pruning (default behavior)
agent_full = Agent(
    model=OpenAIChat(id="gpt-4o"),
    tools=[
        Crawl4aiTools(use_pruning=True)
    ],  # All functions enabled with content pruning
    description="You are a comprehensive web research assistant with all crawling capabilities.",
    instructions=[
        "Use Crawl4AI tools to extract information from web pages",
        "Provide detailed summaries and analysis of web content",
        "Handle various content types including articles, documentation, and repositories",
        "Use content pruning to focus on main content and reduce noise",
    ],
    markdown=True,
)

# Example 2: Enable specific crawling functions
agent_basic = Agent(
    model=OpenAIChat(id="gpt-4o"),
    tools=[
        Crawl4aiTools(
            use_pruning=True,
            enable_crawl=True,
        )
    ],
    description="You are a basic web content extractor focused on page content only.",
    instructions=[
        "Extract and summarize main content from web pages",
        "Focus on text content analysis and summarization",
        "Provide clean, well-structured content summaries",
    ],
    markdown=True,
)

# Example 3: Without pruning
agent_raw = Agent(
    model=OpenAIChat(id="gpt-4o"),
    tools=[Crawl4aiTools(use_pruning=False)],
    description="You are a web intelligence agent that captures full page content.",
    instructions=[
        "Perform comprehensive web analysis using Crawl4AI",
        "Capture full page content without pruning",
        "Provide detailed insights about web pages",
    ],
    markdown=True,
)

# Example usage

# ---------------------------------------------------------------------------
# Run Agent
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    print("=== Comprehensive Web Analysis Example ===")
    agent_full.print_response(
        "Give me a detailed summary of the Agno project from https://github.com/agno-agi/agno and what are its main features?"
    )

    print("\n=== Basic Content Extraction Example ===")
    agent_basic.print_response(
        "Extract the main content and history from https://en.wikipedia.org/wiki/Python_(programming_language)"
    )

Run the Example

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

uv venv --python 3.12
.venv\Scripts\activate

Install dependencies

uv pip install -U agno crawl4ai openai

Export your OpenAI API key

export OPENAI_API_KEY="your_openai_api_key_here"

$Env:OPENAI_API_KEY="your_openai_api_key_here"

Set up Crawl4AI

Install the browser dependencies required by Crawl4AI:

crawl4ai-setup

Update the models

Replace all three OpenAIChat(id="gpt-4o") constructors with OpenAIChat(id="gpt-5.4-mini") in the saved file.

Run the example

Save the code above as crawl4ai_tools.py, then run:

python crawl4ai_tools.py

Full source: cookbook/91_tools/crawl4ai_tools.py

Examples

Primitives

Context

Models

Tools

More

Crawl4AI Tools - Web Scraping and Content Extraction

Run the Example

​Run the Example

Run the Example