Most self-hosted LLM setups break down when models must call tools reliably: malformed tool calls, wrong formats, and fragile text-vs-tool decisions turn simple workflows into engineering overhead. Forge addresses that gap by applying a lightweight, opt-in reliability stack around a single agentic loop so local models behave predictably when invoking tools.
What Sets It Apart
- Practical guardrails focused on tool-calling: rescue parsing (extracts tool calls from Mistral/Qwen/JSON-in-fences), response validation, and retry nudges that correct malformed outputs before they hit your client. This reduces brittle integration failures without changing your agent harness.
- Two usage surfaces: a drop-in OpenAI-compatible proxy that transparently fortifies existing clients, and a WorkflowRunner for structured multi-step workflows (with SlotWorker for queued GPU access). The proxy injects a synthetic
respondtool and hides retries so clients see normal chat-completion responses. - Measured improvements: the project reports raising an 8B local model from single-digit reliability to 84% on its 26-scenario eval suite and improving a Sonnet baseline from 85% to 98% on the same workload (reported from the repo evals). Supports llama-server, Ollama, Llamafile and API backends.
Who it's for — and tradeoffs
Great fit if you run or want to prototype self-hosted LLMs that must call external tools reliably (local inference on ~8B models, multi-step workflows, or adding guardrails to existing agent harnesses). Look elsewhere if you need multi-agent orchestration, cross-agent graphs, or a full application-level coding harness: forge intentionally focuses on a single agent loop and middleware-level reliability. Also note proxy mode is single-shot per request (session state and advanced step-enforcement live in WorkflowRunner).
