Overview
CocoIndex is a data transformation framework purpose-built for AI workflows. Its core engine is implemented in Rust for performance, while developers interact with it through a concise Python API. The framework is designed around a dataflow programming model: transformations are declared as pure functions that produce new fields from existing fields, enabling full observability and automatic data lineage.
Key features
- High performance core (Rust) with Python bindings for developer ergonomics.
- Declarative dataflow: define how to transform data, not how to mutate state.
- Incremental processing: minimal recomputation when source data or transformation logic changes; reuses cached results where possible to keep targets in sync with sources.
- Built-in sources, targets and transformation functions (local files, S3, Postgres, Qdrant, LanceDB, graph DBs, embedding/vision helpers, etc.).
- Export/collect primitives to write results to vector DBs, relational DBs, graph DBs, files, or custom targets.
- Developer velocity: short concise flow definitions (example flows fit in ~100 lines of Python) and many example projects (text embedding, PDF parsing, multimodal indexing, knowledge-graph extraction, FastAPI server, etc.).
Typical use cases
- Building/updating semantic search indexes (embeddings -> vector DB) with incremental updates.
- Extracting structured knowledge from documents and constructing knowledge graphs for context engineering.
- Multimodal indexing (text + images + metadata) for retrieval and LLM augmentation.
- Production data pipelines that must keep downstream stores synchronized with changing sources with minimal recomputation.
Developer experience & examples
CocoIndex exposes a FlowBuilder/DataScope model in Python: you add sources, declare transformations, collect results and export them. The README and docs include quickstart guides and many examples (text embedding, code embedding, PDF processing, S3/Azure/Google Drive sources, Qdrant/LanceDB exports, FastAPI example, and more). Installation is via pip (pip install -U cocoindex) and Postgres is used for incremental processing (as described in docs).
Architecture & integrations
The framework’s architecture emphasizes composable building blocks (sources/transform functions/collectors/targets). This makes it easy to swap storage backends or vector indexes with minimal code changes. It also integrates with common embedding models (e.g., sentence-transformers) and vector stores, and provides hooks for LLM-based extraction and image-captioning pipelines.
Community, license & maturity
CocoIndex is open-source (Apache 2.0) with documentation, examples and a Discord community. The project is presented as production-ready with CI and release automation. Its GitHub repo hosts examples and guides to contribute and extend connectors and transforms.
