Why this matters
Long coding sessions and high-volume tool outputs quickly exhaust an LLM context window or blow up token costs. This project moves compression into a transparent proxy that sits between any agent and its upstream model: the model itself calls injected tools to decide what to compress, the proxy executes those compress/decompress operations server-side, and compressed ranges remain individually restorable. The result is sustained single-session workflows measured in months and billions of cumulative input tokens without enlarging the model window.
What Sets It Apart
- Model-driven, incremental compression: the proxy injects four tools (compress, decompress, search_context, acp_status) and the model chooses when and what to fold. That keeps only recently active content raw while older ranges are replaced by compact summaries.
- Reversible, prefix-cache friendly fold layout: summaries are written in small ranges and can be decompressed on demand. Because the proxy only replaces mid-history suffix ranges, the KV-cache prefix stays intact, preserving cache hit rates and stable attention behavior.
- Measured economics at scale: repository materials and a linked preprint report production usage and long-running experiments showing large token-savings (order-of-magnitude reductions in billed tokens for heavy tool-output sessions) and continuous sessions that avoid window violations on large-window models.
- Universal, zero-per-agent adapter design: any client that can set a base URL can be routed through the proxy; the project provides native plugin modes, a launcher, and a simple /bili/ base-URL rewrite to cover diverse agent deployments.
Who It's For / Trade-offs
Great fit if you run long-lived or tool-heavy coding sessions and need to: 1) keep a single interactive session alive for days-to-months, 2) cut provider billing from repeated re-reads of large tool outputs, or 3) avoid adding external retrieval layers while preserving the ability to restore exact past content. Expect to operate an extra local service (the proxy) and to tune compression cadence for your workload (growth-nudge tokens, preservation rules). It is not a substitute for external document stores when those are already the canonical source of truth; it is designed to manage conversational history rather than serve as a general-purpose vector DB.
Where It Fits
Use as a middle layer between coding agents (CLI/IDE agents, hosted coding assistants) and OpenAI/Anthropic/compatible model endpoints to extend session lifetime and reduce token billing. Works especially well for agents that emit lots of low-density tool output (logs, diffs, test results) where incremental summaries deliver large compression ratios.
Practical notes
Installation options include a native plugin mode for supported clients, a launcher that starts both proxy and client, or a simple base-URL rewrite. The repo documents operational tooling for cache inspection, fold economics, and per-provider tuning. The design emphasizes safe reversibility and observability (cache reports, /acp status endpoints) rather than opaque global summarization.
