Realistic personal assistants must remember and adapt to a user's evolving experiences over months, but most memory benchmarks reduce personalization to short dialogs or isolated facts. MobileMem flips that assumption: it treats a year of mobile usage as the unit of experience, asking memory systems to aggregate fragmented signals across apps, modalities, and time so agents can answer long-horizon questions and update beliefs as users change.
Key Findings
- Benchmark design: MobileMem supplies year-scale user trajectories in both text and multimodal (omni) splits, pairing chronologically ordered app events, conversations, and screenshots with long-horizon QA that require multi-hop retrieval and temporal integration. This emphasizes experience-level memory rather than single-shot retrieval.
- Synthesis & realism: A knowledge-grounded experience synthesis pipeline (used to produce coherent lifelong trajectories from fragmented sessions) enables temporally consistent scenarios where attributes, habits, and preferences must be inferred from distributed traces across apps.
- Evaluation protocol: Systems process sessions in chronological order, incrementally build or update memory, and answer retained questions from the final memory state—making failures traceable to what the memory retrieved vs. model reasoning.
- Research implications: Experiments reveal gaps in current memory systems, especially in updating changing preferences and aggregating sparse cross-app evidence, highlighting memory design as the primary bottleneck for long-horizon personalization.
Who it's for and tradeoffs
Great fit if you research or build personalized assistants, retrieval/memory architectures for agents, or multimodal long-horizon benchmarks and need realistic mobile behavior traces and targeted probes (e.g., temporal QA, implicit preference inference). Look elsewhere if you need purely conversational benchmarks, very large-scale public user logs (privacy constraints limit raw real-world data release), or turnkey memory systems—the contribution is a benchmark and synthesis framework, not a drop-in production memory service.