AIAny
Icon for item

Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning

Supervises audio reasoning by generating per-sample, audio-grounded rubrics that evolve with model rollouts and serve as reinforcement-learning rewards, improving perception and adaptive multi-step reasoning while avoiding reward saturation.

Introduction

Most audio QA systems either supervise only final answers or rely on fixed, hand-crafted process metrics that quickly saturate. AudioRubrics reframes reward design: it synthesizes fine-grained, waveform-anchored rubrics per sample and continually regenerates and reweights criteria conditioned on the model's own rollouts, producing a moving target that focuses training on current weaknesses instead of letting static rewards plateau.

Key Findings
  • Outperforms a broad set of open-source and training-based baselines across three audio reasoning benchmarks, indicating better generalization for both perception-heavy and multi-step tasks — so what: more reliable end-to-end audio understanding without extra labeled reasoning traces.
  • Converges to a stable reasoning length that avoids degenerate collapse or unchecked growth — so what: produces concise yet sufficient reasoning traces, improving interpretability and training stability.
  • Gains scale with the capability of the rubric generator and judge — so what: investing in stronger rubric modules yields proportional policy improvements, making the approach modular and extensible.
  • Anchoring supervision in acoustic evidence improves raw audio perception as well as reasoning — so what: the rubric mechanism serves dual roles of perceptual grounding and process supervision.
Who it's for and tradeoffs

Great fit if you are a researcher or practitioner building RL-based or process-supervised audio reasoning systems, need audio-grounded explainability, or want adaptive rewards that keep training signal informative as models improve. Look elsewhere if you only need simple answer supervision (outcome-only labels), cannot afford the additional compute or infrastructure for rubric-generator/judge components, or lack access to sufficiently rich waveform data for rubric synthesis.

Where it fits

Practically, AudioRubrics sits between outcome-only reward schemes and static process metrics: it keeps the verifiability and interpretability of process supervision while adapting criteria per sample and per training phase, making it a candidate for multimodal reasoning pipelines that require grounded intermediate evidence.

Information

  • Websitearxiv.org
  • AuthorsFangxu Yu, Tao Feng, Dehai Min, Zinan Lin, Weijia Xu, Michael Xu, Philip S. Yu, Ge Liu, Tianyi Zhou
  • Published date2026/08/03

More Items

Analyzes why supervised fine-tuning (SFT) causes severe task conflicts under multi-stage multi-task training while reinforcement learning (RL) enables stable coexistence, attributing the effect to sparse, near-orthogonal RL parameter updates and proposing Parallel-RL to decouple multi-task training.

Analyzes how to build effective training environment distributions for multimodal agents and proposes Ability-aware Environment Selection (AES) and Hierarchical Difficulty Curriculum (HDC) to improve diversity and difficulty scheduling, yielding large relative gains in experiments.

Retrieves short speech segments from MEG recordings with a compact interpretable neural decoder trained against wav2vec 2.0 embeddings, and maps decoder weights to cortical source space to reveal which acoustic and linguistic features drive retrieval.