The best open-source Data Scraping tools, curated: 31 in the registry, including PGX (Postgres for agents), OpenSERP, proxy-scraper. Every tool is open-source; each entry lists license, self-hostability, and links.
PGX (Postgres for agents)· tool, 2026-10-22 AGPL-3.0 Rust runtime that holds thousands of isolated, disposable Postgres databases in one process, so you can give every AI agent its own throwaway database instead of sharing one instance or paying to provision a managed DB per agent — self-hostable on Mac and Linux with no Docker.
OpenSERP· tool, 2026-09-28 Self-hosted, open-source SERP API and CLI (MIT) that returns structured search results from Google, Bing, Yandex, Baidu, DuckDuckGo, and Ecosia via a real rendering browser — no API keys or per-search fees. Runs as a localhost HTTP API or a terminal command, with JSON, Markdown, plain-text, and NdJSON output. An open alternative to paid SERP APIs.
proxy-scraper· tool, 2026-09-26 proxy-scraper is a self-hostable proxy scraper and verifier that pulls candidate proxies from 700+ sources, then runs honeypot, injected-script and TLS checks, and can serve a rotating proxy plus an MCP server. Open source: yes (MIT); self-hostable via pipx, pip or Docker. 5 stars, created 2026-09-24.
browser-use web-ui· tool, 2026-09-26 A Gradio-based web interface for running browser-use AI agents in your browser without the command line. Supports most browser-use functionality, custom browser and persistent sessions, and multiple LLM backends — Google, OpenAI, Azure OpenAI, Anthropic, DeepSeek, and Ollama. MIT-licensed.
Sifthound· tool, 2026-09-25 Sifthound is an MIT-licensed, self-hosted web search / extract / crawl / map API for AI agents, exposing a Tavily-compatible interface plus an MCP server. It is brand-new and unproven — created 2026-09-23, last pushed 2026-09-25, 0 stars — so treat it as a fresh drop, not an adopted tool. The value is that it is self-hostable: an owned endpoint you run rather than a paid search API you call.
Firecrawl launches Alexandria, one-connection data access for LLM agents· news, 2026-09-23 Alexandria is a hosted Firecrawl offering that gives LLM agents single-connection access to data providers, specialized indexes and datasets for extraction, reachable via MCP, CLI or API. It is built on top of Firecrawl — whose core engine is open source — but Alexandria itself is a hosted data-access product, not open source, and carries no SPDX license. It offers free options plus provider-specific pricing; Firecrawl's '+21% answer quality vs built-in web tools' is a vendor claim.
DeepScrape· tool, 2026-09-14 Open-source web scraper (MIT, TypeScript) that turns pages into agent-readable data: Playwright automation plus a fit-markdown extractor (pruning content filters) for clean Markdown, and an LLM-extraction path (GPT-4o) that returns structured JSON to a schema. Ships a hardened one-command Docker deployment (managed-Redis ready, non-root). A smaller, self-hostable entry in the URL-to-Markdown category alongside Firecrawl, Crawl4AI and browser-use.
Google routes search-result links through /goto redirects — a direct hit on SERP scraping· news, 2026-09-13 Google confirmed (Aug 26, 2026) that organic search-result links now route through a google.com/goto?url=... redirect instead of the destination URL, after testing since July. The destination is encoded, HEAD requests on the redirect are blocked, and each result needs its own request — pushing a five-page ranking report from one fetch toward 500-1,000 requests. Nothing changes for a human clicking; the visible URL still renders as text. Read widely as an anti-scraping move.
Browser Harness· tool, 2026-09-03 Pipeline inventory: Stage 5 strong acquisition fallback (fit 8/10) and Stage 2 partial enrichment extractor (fit 6/10). Self-healing harness connects LLMs to a real Chrome browser via CDP/Playwright and writes reusable helpers at runtime; complements Agent Reach/Crawl4AI/Crawlee/Lightpanda; ~16.7k stars. Expected to reduce interactive public-web extraction gaps by roughly 40-60%, but does not supply proprietary firmographic/contact data. CAVEATS: target-site ToS and…
browser-use· tool, 2026-09-03 Flagship 'make websites accessible for AI agents'; ~108k stars; CDP/Playwright; complements Crawl4AI/Browser Harness/Lightpanda; CAVEAT target-site ToS + agent runtime actions
cdp-use· tool, 2026-09-03 Pure Chrome DevTools Protocol, type-safe in Python; browser-automation primitive; ~308 stars
Crawl4AI· tool, 2026-09-03 OSS alt to Browserbase; ~77347 stars; via openalternative.co
Darts· tool, 2026-09-03 ~9487 stars; via ossinsight.io
dlt· tool, 2026-09-03 Pure OSI Python EL library; schema inference
PyOD· tool, 2026-09-03 ~9953 stars; via ossinsight.io
ScrapeGraphAI· tool, 2026-09-03 OSS alt to Diffbot; ~29238 stars; via opensource.builders
Scrapy· tool, 2026-09-03 Mature; Zyte-backed; no built-in JS render
spaCy· tool, 2026-09-03 Production NLP; Explosion-maintained
Stanza· tool, 2026-09-03 SOTA neural accuracy; 60+ languages
STUMPY· tool, 2026-09-03 ~4140 stars; via ossinsight.io
Firecrawl anydoc· tool, 2026-09-03 Open-source document-to-Markdown converter from the Firecrawl team (Rust with multi-language bindings): Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV and PDF into clean GitHub-Flavoured Markdown — an ingest layer for agent and RAG pipelines.
Obscura· tool, 2026-08-31 Open-source headless browser for AI agents and web scraping, built in Rust: runs V8 JavaScript, speaks Chrome DevTools Protocol, drop-in for Puppeteer/Playwright, ~30MB, with stealth/anti-fingerprinting and MCP support — no Chromium.
Maxun· tool, 2026-08-31 Open-source, no-code web-data platform: extract structured data (recorded actions or natural language), scrape pages to Markdown/HTML with screenshots, crawl whole sites, and run automated searches — self-hostable, with an SDK and CLI.
Chrome DevTools MCP· tool, 2026-08-25 MCP server that lets an agent inspect and control a live Chrome browser — performance traces, network requests, console errors, rendered pages, and screenshots.
Docling· tool, 2026-08-25 Document-processing toolkit with advanced PDF/DOCX/PPTX/XLSX understanding (tables, complex layouts) and generative-AI integrations — for when simple PDF-to-text loses structure.
Firecrawl· tool, 2026-08-25 The context API to search, scrape, and crawl the web at scale and turn sites into clean, LLM-ready data. Core is AGPL-3.0; SDKs/UI are MIT.
Firecrawl MCP Server· tool, 2026-08-25 Official Firecrawl MCP server that brings web search, scraping, crawling, and structured extraction to MCP-compatible agents as clean, agent-ready context.
MarkItDown· tool, 2026-08-25 Microsoft utility that converts PDFs, Word, PowerPoint, Excel, HTML, images and more into clean Markdown for LLM pipelines — structure-preserving.
Playwright MCP· tool, 2026-08-25 Microsoft MCP server that lets agents automate a browser via structured accessibility snapshots (not screenshots) — for rendered-page checks, forms, funnels, and landing-page QA.
oc (only-cli)· tool, 2026-08-25 Open-source CLI that fetches a web page and returns a compact numbered view instead of raw HTML — a cheap web-read layer for scraping/enrichment agents. Installs as a CLI and as an agent skill (npx skills add).