Everyday personal assistants must recall detailed past events from continuous visual and audio inputs, but maintaining an efficient, retrievable long-term multimodal memory on constrained devices is challenging. This work argues that a streaming, hierarchical memory with dynamic routing can provide accurate, grounded answers about past experiences while staying light enough for deployment on phones and AI glasses.
Key Findings
- Hierarchical memory tiers (current, short-term, long-term): stores and indexes egocentric visual/audio streams at multiple temporal scales, so queries are routed to the appropriate timescale instead of searching an undifferentiated archive.
- Dynamic retrieval routing: at query time the system decides which memory tier(s) to consult, reducing latency and irrelevant matches — so what: faster, more precise retrieval on-device.
- Multimodal grounding for answers: retrieved visual and audio evidence is used to generate responses that reference concrete sensory inputs (object sightings, conversations, activities), enabling tasks like object finding, conversation recall, life summarization and routine discovery.
- Lightweight streaming design: engineered for continuous capture and incremental indexing with resource constraints in mind, so what: feasible deployment on smartphones and wearable AI glasses rather than requiring server-side logging of all data.
Who It's For and Tradeoffs
Great fit if you need an on-device personal memory assistant that continuously records first-person visual and audio streams and returns evidence-grounded answers (e.g., personal journaling, object-finding, behavior summaries). Look elsewhere if you require strong privacy guarantees via formal verification, extremely high-fidelity long-term archival (petabyte-scale), or advanced cross-user aggregation — the design prioritizes resource efficiency and per-user streaming utility over large centralized corpora.
How It Works (brief)
The system aligns egocentric video and audio on a shared timeline, incrementally encodes segments into compact multimodal representations, and maintains indices at each memory tier. A lightweight router uses the query and temporal cues to select tiers and candidate evidence, and a generator composes answers grounded in returned multimodal snippets. The project also provides a demo pipeline intended for smartphone and AI-glasses prototypes.