AIAny
Icon for item

SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue

Stores speaker-labeled verbatim messages alongside derived person- and group-level states to improve attribution and long-term memory in multi-party dialogues. Proposes a dual-track memory and a speaker-conditioned Writer trained with SpeakerLevenshtein and GRPO; yields consistent gains on multiple dialogue memory benchmarks.

Introduction

Why this matters

Multi-party conversations mix who said what with information about relationships, shared facts, and evolving states. General-purpose memory systems often blur speaker attribution and lose relations that are crucial for queries like “who promised X?” or “what does the group agree on now?”. The core insight of this work is that keeping a verbatim record plus a structured, provenance-linked set of person- and group-scoped states lets a system reconstruct facts and relations more reliably from interleaved histories.

Key Findings
  • Dual-track design: a verbatim track preserves exact messages with speaker/time metadata, while a derived track stores PERSON (Core/Profile) and GROUP (Interaction/Insight) views. So what: this separation keeps precise wording and high-level state jointly available for retrieval.
  • Evidence-anchored query pipeline (Anchor–Separate–Resolve–Compose) organizes evidence by entity, event, and time, which reduces attribution errors when answering who/when/whom questions.
  • Training the Writer with SpeakerLevenshtein and speaker-conditioned GRPO improves structured update accuracy; in evaluations the system raises benchmark binary accuracies (GroupMemBench, SocialMemBench, EverMemBench) compared with mainstream frameworks.
Who it's for and tradeoffs

Great fit if you build dialogue systems that must track identities, commitments, and group beliefs across long, interleaved conversations (e.g., meeting assistants, group chat summarizers, multi-agent coordinators). Look elsewhere if your application is single‑turn or two‑party short chats where simple retrieval suffices—dual-track storage and structured updates add complexity and storage overhead.

How it works (brief)

The system links every derived record to supporting verbatim messages, separating source (who provided the info) from owner (whom it concerns). Person-level records capture stable identity facts, stances, and recurring behavior; group-level records capture interactions, decisions, and norms. At query time, the pipeline retrieves verbatim and structured evidence, aligns by speaker/entity/time, resolves conflicts, and composes a grounded answer.

Practical takeaway

If your failure modes include misattributing utterances or losing relational clues across time, a dual-track, provenance-aware memory with speaker-aware update policies is a pragmatic architectural choice—it trades storage and update complexity for better factual attribution and state reconstruction.

Information

  • Websitearxiv.org
  • OrganizationsState Key Lab of CAD&CG, Zhejiang University
  • AuthorsHaobo Zheng, Tan Tang, Yan Chen, Weijie Wang, Yingcai Wu
  • Published date2026/09/22

More Items

Builds tool-composition environments from real execution traces and generates verifiable long-horizon agent tasks. Results show substantial gains in models’ ability to coordinate tools across environments.

Proves that JEPAs across modalities can learn representations that differ only by a rotation when positive pairs describe the same world states. These “good views” achieve cross-modal alignment with far fewer samples than next-token prediction.

Estimates how committed tokens improve predictions at remaining masked positions, then uses those contributions to refine reinforcement-learning signals for diffusion language models. On LLaDA-8B-Instruct, it raises MATH500 and GSM8K scores by up to 9.0 and 5.3 points.