Running a model locally is the easy part; the hard part is everything an application needs around it — reading your documents, citing sources, calling tools — without that data ever touching the cloud. This is the layer most "private AI" stacks are missing: it sits above an OpenAI-compatible inference server and exposes a single Claude-style messages API, turning a raw local model into something you can build a product on rather than just chat with.
What Sets It Apart
- An application layer, not an inference engine — it pairs with runtimes like Ollama or vLLM instead of replacing them, so you keep your own choice of model and hardware and avoid lock-in to one runtime.
- Retrieval with inline citations is built in — answers point back to the source document, which matters when "the model made it up" is a compliance problem rather than a nuisance.
- Tools and MCP ship out of the box — web search, web fetch, code execution, custom tools, and an MCP connector mean agents can act, not just answer.
- Fully local by design — file ingestion, embeddings, and orchestration all run in your environment under Apache-2.0, with nothing sent to a third-party API.
Who It's For
Great fit if you operate in a regulated or air-gapped setting — healthcare, legal, finance — and need document-grounded AI you can self-host and audit. Look elsewhere if you just want a quick local chatbot, where a runtime's built-in UI is simpler, or if you'd rather not run and maintain inference infrastructure: the project assumes you bring the GPU and the ops. It's also the open-source engine behind Zylon, the team's commercial on-prem platform, so a managed path exists if self-hosting becomes a burden.
