Most projects either provide raw HTML scraping tools or fully managed data-as-a-service; this project sits in between by turning web pages into clean, structured JSON tailored for AI agents and downstream analytics. It pairs a graph-style pipeline abstraction with LLM reasoning so you describe what you want extracted and the pipeline composes the extraction logic.
What Sets It Apart
- Graph-driven pipelines: prebuilt graph workflows (SmartScraper, SearchGraph, ScriptCreator, SpeechGraph, multi-page variants) let you express extraction goals instead of low-level selectors—so you spend time defining intent, not brittle parsing rules.
- LLM-first extraction: integrates with OpenAI, Gemini, Groq, Azure, MiniMax and local models via Ollama—so you can use cloud LLMs or run models locally depending on privacy and cost needs.
- Dual distribution model: an open-source Python library for full control and a managed cloud API that handles JS rendering, anti-bot, proxies, scaling and scheduled monitors—so teams can choose self-hosting or a low-friction production path.
- SDK & integrations: Python and Node SDKs plus integrations with LangChain, LlamaIndex and low-code platforms—so it fits into existing ML/agent stacks with minimal glue code.
- Practical telemetry & tooling: includes telemetry opt-out and focused pipelines for script generation, audio output and multi-page search aggregation—so common scraping workflows are covered without custom engineering.
Who It's For and Trade-offs
Great fit if you need curated, LLM-enhanced extraction for AI applications (RAG, agent inputs, analytics) and want either full control (self-hosted, bring-your-own LLM) or a managed service that handles rendering and anti-bot. Look elsewhere if you only need simple HTML parsing (BeautifulSoup/Playwright) or require strict, deterministic parsers for legal/archival use—LLM-driven extraction can be more flexible but less deterministic. The project is production-focused but requires you to choose and manage LLM costs and browser configuration when self-hosting.
