Firecrawl /search: Search + Scrape Websites in One API Call (2026)

Web scraping in 2026 often involves two painful steps: discovering relevant pages and then reliably extracting clean content from them. Traditional approaches with requests + BeautifulSoup fail on JavaScript-heavy sites, while full browser automation like Playwright or Selenium adds complexity, cost, and maintenance overhead.

Firecrawl’s new /search endpoint solves this by merging search discovery with powerful scraping capabilities in one call. It returns ready-to-use results with full page content in LLM-friendly formats like markdown.

In this guide, you’ll learn exactly how the feature works, see production-ready code examples, best practices for scaling, and real-world applications. Whether you’re building AI agents or running large-scale data projects, this approach dramatically simplifies your workflow.

Why Search + Scrape in One Call Matters for Modern Web Scraping

Modern websites use heavy JavaScript, dynamic loading, and anti-bot protections that make classic scraping brittle. Manually searching Google (or alternatives) and then scraping each URL separately introduces rate limits, proxy management, and parsing headaches.

Firecrawl /search addresses this by handling both discovery and extraction server-side. You provide a natural language query, and it returns top results complete with scraped content. This is especially powerful for:

• AI agents needing fresh web context
• Competitive analysis and market research
• Lead generation pipelines
• SEO content and backlink discovery

As of 2026, this unified approach reduces engineering time while delivering higher-quality, structured data.

Tools and Setup for Firecrawl Search

Getting started requires minimal setup. You’ll need:

• A Firecrawl API key (free tier available for testing)
• The official Python SDK (firecrawl-py)
• Optional: integrations for Zapier, n8n, or MCP for agentic workflows

Install the SDK:

pip install firecrawl-py

Set your API key via environment variable for security:

export FIRECRAWL_API_KEY="fc-your-key-here"

Ethical note: Always respect robots.txt, website Terms of Service, and applicable laws like GDPR/CCPA. Firecrawl is designed for legitimate data collection—use responsibly and avoid overloading sites.

Related: Our complete guide to ethical web scraping and rate limiting

Step-by-Step Implementation

Here’s how to use the /search endpoint in Python:

from firecrawl import FirecrawlApp
import os

app = FirecrawlApp(api_key=os.getenv("FIRECRAWL_API_KEY"))

result = app.search(
    query="best hosted Postgres databases",
    params={
        "limit": 10,
        "scrapeOptions": {
            "formats": ["markdown", "html", "links", "screenshot"],
        }
    }
)

for page in result["data"]:
    print(f"Title: {page['title']}")
    print(f"URL: {page['url']}")
    print(page["markdown"][:500] + "...")

The scrapeOptions parameter lets you reuse all familiar Firecrawl scraping features directly in search results. This includes custom headers, JavaScript execution waits, and content formatting.

Handling Common Scraping Challenges

Even with a powerful API, real-world scraping hits issues:

• Dynamic content — Firecrawl’s scraping engine handles JavaScript rendering reliably.
• Anti-bot detection — The service uses enterprise-grade evasion (rotating infrastructure, realistic fingerprints).
• Pagination and depth — Combine search with Firecrawl’s crawl features for broader discovery.
• Data freshness — Search pulls current web results rather than stale indexes.

Pro tip: For high-volume use, implement exponential backoff and monitor usage quotas. Test with small limit values first.

Production Best Practices and Scaling Tips

For production-grade scrapers:

1. Error handling — Wrap calls in try/except and retry on transient failures.
2. Rate limiting — Respect API limits and add delays between batches.
3. Data cleaning — Post-process markdown for your specific LLM or database needs.
4. Caching — Store results locally or in a vector DB to avoid redundant calls.
5. Monitoring — Log success rates and response times.

Example with basic resilience:

import time

def robust_search(query, max_retries=3):
    for attempt in range(max_retries):
        try:
            return app.search(query=query, params={"limit": 10})
        except Exception:
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)

This keeps your pipelines stable at scale.

Real-World Use Cases and Example Projects

The video demonstrates several powerful applications:

• Competitive analysis: Search “best hosted Postgres” and instantly get markdown comparisons.
• AI agent research: Let agents fetch latest docs (e.g., OpenAI image generation models) and implement features dynamically.
• Automated slideshow/report generation: Pull data and feed it into presentation or analysis tools.
• Deep research apps: Fire Search demo compares products like iPhone 16 Pro vs. Samsung Galaxy S25 with citations.

The open-source Fire Search template is available in Firecrawl’s library for self-hosting similar research agents.

Related: Building production AI agents with web data

Comparison with Alternatives

Firecrawl shines when you want simplicity without sacrificing quality. For ultra-custom browser needs, combine it with Playwright via custom scrape options.

Conclusion

Firecrawl’s /search endpoint represents a major leap for web data extraction in 2026: one call for discovery and rich content. Key takeaways include seamless integration of search and scrape, excellent LLM-ready outputs, strong support for agents, and reduced operational complexity.

Try the playground, test the Python examples above, and build something powerful. Share your results in the comments or on X—we love seeing creative applications.

Last updated: 2026. Explore more battle-tested scraping tutorials on EasyWebData.com.