AIAny
Icon for item

Adaptive Reward Routing: Dynamic Multi-Reward Optimization for Joint Audio-Video Diffusion via Forward-Process RL

Proposes a forward-process RL method that dynamically localizes gradient updates and adapts multi-reward coordination for joint audio–video diffusion models. Uses bidirectional cross-attention responses for token/layer routing and preference-preserving reweighting to balance competing objectives, improving modality quality, alignment, and synchronization.

Introduction

Most post-training methods for joint audio–video generation assume fixed ways to route reward-driven updates and fixed reward weights, but cross-modal functions and reward conflicts evolve during fine-tuning. The core insight of this paper is that adapting both WHERE updates act (token/layer localization) and HOW rewards are combined (modality-aware reweighting anchored by user preferences) yields more stable and effective multi-objective RL for audio–video diffusion models.

Key Findings
  • Cross-Modal Influence-Guided Routing: bidirectional cross-attention responses serve as a lightweight proxy to identify which tokens and cross-attention layers are functionally important; token-aware loss reweighting and layer scaling concentrate updates along influential cross-modal pathways so gradient flow is preserved where it matters.
  • Preference-Preserving Modality-Aware Reweighting: after a warm-up, the method computes branch-specific reward-gradient interactions and applies them as residual corrections to predefined preference weights, resolving emergent conflicts without discarding user priors or letting strong rewards suppress weaker but necessary objectives.
  • Empirical gains: when applied to LTX-2 backbones, the combined approach improves video/audio quality, text–audio alignment, and audio–video synchronization versus strong RL baselines; ablations show token-level routing, layer routing, and residual reweighting each contribute.
Who it's for and trade-offs

Great fit if you work on multimodal generative models that couple streams via cross-attention and need to fine-tune for several quality and alignment metrics at once (e.g., audio/video quality, cross-modal semantic alignment, temporal sync). Look elsewhere if you need a plug-and-play single-reward tuning recipe or if your model has no explicit cross-modal attention structure—this approach relies on access to cross-attention responses and gradient interactions and adds complexity compared with fixed-weight scalar aggregation.

Where it fits

This sits between scalar weighted-sum reward tuning and gradient-space multi-reward methods: it localizes updates along cross-modal interaction paths like routing-based methods, while keeping user-defined preference priors and only applying learned residual corrections to reward weights, aiming to balance principled adaptation with predictable priorities.

Mechanism notes

The routing uses layer- and token-level aggregations of cross-attention response norms to derive token weights and layer scales; reweighting estimates conflicts within modality branches and applies residual corrections after warm-up to adapt over training. Reported probes show the attention-response proxy correlates strongly with true functional influence and that static routing becomes stale as fine-tuning progresses.

Information

  • Websitearxiv.org
  • OrganizationsMMLab@HKUST, The Hong Kong University of Science and Technology, Tencent Video, The University of Hong Kong
  • AuthorsSonglin Yang, Xiaotong Zhao, Jiacheng Zhang, Zhe Wang, Toyota Li, Eric Liu, Alan Zhao, Anyi Rao
  • Published date2026/09/29

Categories

More Items

Turns LLMs’ physical knowledge into video training data using simulator constraints, physical verification, and iterative correction. It also repairs physical errors in generated videos and releases 100K videos spanning 2,500 physical events.

Continuously perceives live audio and video while deciding whether to respond, interrupt, or keep listening. It couples interaction reasoning with audiovisual generation for real-time, full-duplex exchanges.

Generates driving video and a bird’s-eye view from one camera image, with optional ego-trajectory control. The paired views help model surrounding traffic reactions without requiring vehicle-state inputs or an HD map at inference.