AIAny
AI Agent Papers2026
Icon for item

SubtleMemory: A Benchmark for Fine-Grained Relational Memory Discrimination in Long-Horizon AI Agents

Benchmark that measures an agent's ability to discriminate fine-grained relational structure in long-term memories. It embeds relation-controlled memory variants into realistic user–agent histories and tests downstream recovery and reasoning, highlighting where current memory systems fail.

Visit Website

Introduction

Long-running AI assistants build up large, overlapping collections of memories whose usefulness depends on the relations between items (complementary, nuanced, or contradictory), not just isolated recall. SubtleMemory shows why this relational discrimination matters: agents that retrieve plausible items can still fail downstream when they can't recover correct inter-item relations embedded across long histories.

Key Findings
  • Constructs and scale: SubtleMemory builds 1,090 relation-controlled memory-variant sets and 1,522 evaluation instances spread across 10 long user–agent histories — focused probes rather than broad memorization tests. This design isolates relational errors from mere retrieval failures.
  • Diagnostic power: The benchmark distinguishes memory preservation, retrieval, and downstream reasoning failures with dedicated protocols, revealing different failure modes across systems and enabling targeted improvements.
  • Empirical result: Evaluations across standalone memory systems and Claw-style agents (native and plugin modules) show consistently weak performance on fine-grained relational discrimination, even when raw retrieval appears adequate.
  • Practical implication: Improving long-term assistant behavior requires module-level changes (how memories are stored and linked) and reasoning-stage designs that explicitly model inter-memory relations rather than only scoring item relevance.
Who It's For and Trade-offs

Great fit if you are developing or evaluating long-term memory modules, retrieval-augmented agents, or reasoning layers that must resolve nuanced conflicts across distributed memories. The benchmark is especially useful for diagnostic comparisons and for driving designs that model relational structure.

Look elsewhere if your priority is large-scale open-domain memorization, raw recall benchmarks, or tasks that do not depend on inter-item relations — SubtleMemory intentionally focuses on relational discrimination and uses a limited set of long histories to keep probes interpretable. Its controlled variants trade off ecological breadth for diagnostic clarity.

Where It Fits

This benchmark complements long-term memory and retrieval evaluations by shifting the question from "can the agent find relevant items?" to "can the agent recover and reason over the correct relationships among retrieved items?" Use it alongside broader memory datasets to get a fuller picture of an agent's long-horizon competence.

Back

Information

  • Websitearxiv.org
  • AuthorsWenxuan Wang, Haoyu Sun, Fukuan Hou, Mingyang Song, Weinan Zhang, Yu Cheng, Yang Yang
  • Published date2026/06/04

Categories

Tags

More Items

Large Language Model Papers2026
Icon for item

Language Models Can Control Their Own Attention

Namgyu Ho, Huzama Ahmad +4KAIST AI, Google DeepMind

Introduces Declarative Attention (DA), a zero-shot protocol that has LMs declare which parts of long context to attend to during chain-of-thought, letting the runtime build dynamic attention masks and skip most KV-cache reads. Produces large token savings (up to ~52% on Gemma-4-31B) with modest accuracy loss.

llmlong-horizonvllmgemmaqwen+5
Natural Language Processing Papers2026
Icon for item

It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning

Runpeng Dai, Kaili Huang +2University of North Carolina at Chapel Hill, Apple

Generates compact keyword sets for both queries and items with LLMs and matches them directly via an inverted index. Uses supervised fine-tuning to align keyword spaces, then alternates GRPO-based reinforcement learning on query- and item-side generators to co-evolve representations and maximize retrieval F1 while staying compatible with keyword-based infrastructure.

retrievalLLMRLsftbenchmark+2
AI Agent Papers2026
Icon for item

EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction

Yuling Shi, Zhensu Sun +4Shanghai Jiao Tong University, Shanghai, China, Singapore Management University, Singapore +2

Predicts an LLM agent's final success or failure from partial execution traces and halts runs when outcomes are confident to save per-task compute. Uses LightGBM success/failure classifiers on behavioral, textual, and reference features; cuts 13–26% steps and up to 44% input tokens on benchmarks.

evaluationbenchmarksai-agentllmagent-skills