Most memory evaluations assume the facts a model needs will resemble the query that triggers them. That assumption hides a failure mode: important user facts can be implicitly related to a query (e.g., a tree-nut allergy and an almond-flour macaron) yet share no retriever-visible cue. InMind reframes memory evaluation to expose that implicit-association blind spot and measure where retrieval pipelines actually fail.
Key Findings
- InMind consists of 125 expert-verified tasks across ten life domains, with 113 tasks grounded in citable public sources. Each task includes paired controls designed to disentangle three failure explanations: the fact was never stored, the model lacks the bridging world knowledge, or the fact was stored but not surfaced by the retrieval interface.
- When the decisive memory is placed directly in context before the query, the backbone model answers 84.0% of indirect queries. When retrieval must surface the same memory, six representative memory systems (vector, graph, and agentic) achieve at most 14.4% on the answer-blind target recall metric despite being able to recall the facts on demand at up to 100% in other probes.
- Increasing embedding dimensionality (×8) improves recall across systems but does not close the gap: the retrieval-interface failure remains the dominant bottleneck. A minimal diagnostic probe that keeps memory visible before the query recovers most of the gap, locating the failure in the query-conditioned routing/selection interface rather than in storage or world knowledge alone.
What Sets It Apart
- Paired-control design: separates storage, knowledge, and retrieval-interface failures instead of conflating them in a single accuracy number — so you can tell whether a system truly forgot a fact or simply failed to route it at query time.
- Realistic, expert-curated tasks: domain coverage and citable grounding make failures actionable for system designers who must decide what to index, surface, or keep visible.
- Diagnostic-first framing: instead of proposing a single new memory architecture, the paper supplies evaluation tooling and metrics that make routing the measurable, optimizable axis.
Who It's For and Tradeoffs
Great fit if you design or evaluate long-term memory modules for agents, RAG systems, or LLM-based assistants and need to know whether failures stem from storage, knowledge gaps, or retrieval routing. Look elsewhere if your focus is purely on raw embedding quality benchmarks or on application-level end-to-end metrics without interest in causal diagnosis. InMind trades broad scale (125 tasks, focused domains) for diagnostic clarity; it reveals interface-level routing failures but does not itself prescribe a single architectural fix.