AIAny
Icon for item

Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory

Provides a reflexive agentic framework for long-horizon video understanding that replaces costly iterative reasoning with dual contextual states: a consolidated global multimodal script and parametric latent states for fast retrieval and response, improving speed and memory efficiency.

Introduction

Most long-video agents consume or iteratively re-reason over large contexts, which drives latency and high GPU cost. The core insight behind this work is that many iterative reasoning steps primarily compensate for missing global context and misaligned retrieval; by maintaining a compact global multimodal script and generating a parametric latent state conditioned on it, an agent can decide, retrieve, and respond in a single pass with much lower overhead.

Key Findings
  • Latent reflex pathway replaces detective-style loops: Light-Omni learns compact latent states for generation, memory, and reaction so the model can decide whether to answer or retrieve without repeated forward/backward reasoning — this reduces end-to-end latency and contextual blowup.
  • Global multimodal script + hierarchical consolidation: the system continuously summarizes episodic memory into a fixed-size global context that preserves recent details while compressing older events, enabling semantically aligned retrieval at scale.
  • Empirical trade-offs are concrete: compared to a prior M3-Agent baseline, Light-Omni reports an average +2.4% accuracy, a 12.1× speedup, and 2.6× improvement in GPU memory efficiency; evaluated on long-video benchmarks (e.g., MME-Video Long, LVBench, HippoVlog) with competitive scores.
  • Modular adapters for practicality: training proceeds in stages (generation, memory, reaction) and the released code vendors patched model/runtime pieces (Qwen2.5-Omni template and ms-swift) so reproducibility requires those tailored components.
Who It's For and Tradeoffs

Great fit if you need a long-video interactive agent that must maintain user- or session-specific memory across vision, audio, and text while keeping inference latency and memory footprint low. It is especially useful when full-context iterative reasoning is too costly (real-time demos, streaming agents, or memory-augmented MLLM pipelines).

Look elsewhere if your use case relies on heavy symbolic deduction that requires many explicit reasoning turns, if you cannot accept dependency on patched runtime/model templates (Qwen2.5-Omni + ms-swift patches), or if you need a tiny on-device footprint — Light-Omni optimizes retrieval/response efficiency but still assumes GPU-backed multimodal encoders and adapter training.

Where It Fits

Positioned between full-context iterative agents and simple RAG pipelines: it sacrifices exhaustive multi-turn search for a learned reflex that is semantically aligned to long-term memory, aiming to make video agents scale with information need rather than raw video length.

Information

  • Websitearxiv.org
  • AuthorsChang Nie, Jiaju Wei, Junlan Feng, Chaoyou Fu, Caifeng Shan
  • Published date2026/07/06

More Items

Enables interactive world models to follow user-defined rules and preserve them across save-and-restore. An executable 3D rule graph tracks rules, dependencies, and object states, while constraining video generation to show their effects.

Finds that agent evolution often stalls not for lack of better prompts or skills, but because noisy evaluations fail to identify real improvements. The study tests seven methods across six models.

Scales verifiable gameplay data by having a Builder and visual and coding players co-develop web games. Progress-weighted training improves agent performance on both its own benchmark and an external game benchmark.