Why this matters
Bringing real recordings into a token-first music generator was the missing piece for fine-tuning and covering: this release supplies the audio→semantic-token encoder YuE2 never shipped and pairs it with a decoder LoRA so tokens derived from real productions actually decode back to realistic audio. That closes the loop for tokenizing your catalog, LoRA-tuning YuE2’s autoregressive planner, and producing new songs or covers that sound like the originals.
What Sets It Apart
- End-to-end real-audio workflow rather than just a token format: an 8-layer transformer head (d=512) maps MERT-v2-FullSong layer-20 features to 32,768 YuE2 semantic codes at 25 Hz, and a jointly trained NAR LoRA (rank-32) adapts the decoder to render real-production latents. This avoids the mismatch where tokens from a separate encoder would not decode faithfully.
- Measured fidelity and practical numbers: the v4 head reports ~16.1% held-out exact top-1 token match on YuE2-generated songs (near-miss codes render nearly identically), and NAR round-trips give high perceived fidelity in ear tests. Training used ~4,765 YuE2-generated songs, then adapted to real audio; later v5/v8/v9 joint rounds improved token accuracy and audio loss tradeoffs.
- Tooling and formats for integration: ships fp32/.bf16/.safetensors variants plus a ComfyUI-native LoRA layout, scripts to prepare MERT features, demix vocals, align lyrics, tokenize, and render with YuE2 pipelines. The release also provides guidance on which VAE merges and LoRA checkpoints to pair for best decoding.
Who it's for and tradeoffs
Great fit if you need to: tokenize an artist catalog for LoRA training, generate covers or new songs that match production fingerprint, or iterate on artist-specific AR LoRAs using real recordings. It assumes familiarity with the YuE2 ecosystem (m-a-p/YuE2-3B, MERT-v2-FullSong, YuE2 VAE) and non-commercial CC-BY-NC licensing for the weights.
Look elsewhere if you need a plug-and-play consumer app: using the toolkit requires the YuE2 base models, some data preparation (stems, lyrics, style captions), and a capable GPU (authors report ~24 GB recommended; many stages fit in 14–18 GB with checkpointing). Also, token-choice errors (occasional note mismatches) remain a head-accuracy limitation and may require retraining or later head versions to reduce.
