Real-time conversational speech systems need memories that both stream as the user speaks and capture emotional and persona cues — not just facts. VoiceMem answers this by splitting memory into two cooperating tracks: a schema-driven left brain for precise fact retrieval and a right brain that stores affective and persona signals, all wired into a streaming ingestion and retrieval pipeline designed for sub-VAD latency.
Key Findings
- Accuracy: The left-brain schema/entity design yields much higher effective retrieval under a Top-5 injection budget, outperforming classical wide-index approaches while keeping injected context small. This design reduces expensive context bloat during generation.
- Emotional & Personal Modeling: The right brain uses short- and long-horizon affect attribution plus dual-node persona modeling to capture who the user is and how they feel, improving persona benchmarks and aggregate scores versus prior systems.
- Real-time & Cost: A streaming I/O and hierarchical storage strategy (compression, layered retrieval, speculative prefetch) yields median retrieval times reported around 134 ms and keeps injected memory tokens low, enabling memory-aware SLMs without adding perceptible conversational delay.
Who it's for and trade-offs
Great fit if you build or evaluate real-time speech agents (duplex SLMs, voice assistants, on-device agents) that must personalize across turns while respecting strict latency budgets. It helps when you need both accurate factual recall and ongoing emotion/persona modeling. Look elsewhere if you only need simple session-scoped context or if you cannot afford the extra pipeline components (ASR, voice fingerprinting, affect extractors) or have strict privacy regimes that forbid storing user-sensitive values without additional safeguards.
How it works (brief)
VoiceMem ingests audio streams, runs ASR and audio-native perception (speaker ID, emotion, scene), extracts embeddings, and writes structured nodes into two cooperatively queried stores. Queries are routed and ranked, then only the Top-K memories are injected into the model context. The architecture is backend-agnostic so retrieval DBs and perception modules can be swapped per deployment needs.