AIAny
Icon for item

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Reconstructs historical experience into latent memory tokens and weaves short- and long-term latent memories directly into vision-language-action reasoning to improve long-horizon robotic manipulation. Uses a four-part pipeline (curator, seeker, condenser, weaver) so memory participates natively in multimodal action formation.

Introduction

Robotic manipulation tasks with long horizons fail when models assume Markovian observations and lack a fluid way to consult past episodes. The core insight of this paper is that memory must live in the same continuous latent space as multimodal reasoning: by representing, retrieving, and consuming history as latent tokens, memory can directly shape vision-language-action (VLA) decision-making within a bounded context.

Key Findings
  • Latent-native memory: Rather than appending raw observations or policy-side context, historical experience is reconstructed into compact latent memory tokens that sit in the same embedding stream used for multimodal reasoning, enabling direct interaction between memory and current perception/instruction.
  • Dual vault design: A curator separates experience into complementary short-term and long-term vaults; a seeker performs multimodal queries over both vaults to retrieve context-relevant evidence; a condenser converts retrieved evidence into compact latent tokens; a weaver injects those tokens into the continuous embedding sequence used for action generation.
  • Empirical gains: Experiments on SimplerEnv and LIBERO show improved performance on temporally dependent, long-horizon manipulation tasks compared with prior memory-augmented VLA baselines, indicating better use of historical cues for action sequencing.
Who it's for and tradeoffs

Great fit if you work on embodied agents or robot learning that require temporal crediting across long episodes and you need memory to influence multimodal reasoning directly. Look elsewhere if your tasks are near-Markovian, memory budget is extremely tight, or minimal-latency inference is mandatory—latent reconstruction and retrieval add compute and storage overhead and require consistent latent alignment across modalities.

Where it fits

This approach sits between vanilla VLA models (which are largely memoryless) and heavy episodic replay systems: it keeps memory concise by design and makes historical evidence first-class in the model’s reasoning pipeline, rather than an external auxiliary context.

Method notes

The four coordinated components (curator, seeker, condenser, weaver) emphasize modularity: vaults organize experience, multimodal queries prioritize relevance, condensers compress for bounded contexts, and weavers ensure memory tokens are consumed alongside live observations and instructions. The paper evaluates this architecture under standard embodied-robot benchmarks, reporting measurable improvements but noting added system complexity and resource costs.

Information

  • Websitearxiv.org
  • AuthorsHongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang, Xinlei Yu, Rui Yan, Wenguan Wang, Xiangbo Shu, Shuicheng Yan
  • Published date2026/07/08

More Items

Distills a diffusion teacher’s conditional image distribution by matching exponentially tilted student means to teacher posterior means. It supports flexible sampling budgets on the trained timestep grid while preserving generation diversity.

Introduces a multimodal lens for tracing how vision-language models represent visual concepts and selectively access them in response to questions. Counterfactual tests examine how those concepts affect later reasoning; the work does not claim models are conscious.

Tests multimodal models’ spatial reasoning through screen-based clicks and drags. Its programmatically generated CAPTCHA tasks also form a training environment, with fine-tuning gains reported across seven out-of-distribution benchmarks.