For years, "segment anything" lived only in the image world: point at an object, get a mask, no per-class retraining. Audio never had that. Pulling a single dog bark out of a street scene usually meant training a model dedicated to that one source type. SAM-Audio ports the promptable foundation-model idea to sound — you describe or point at what you want, and it lifts that out of the mixture.
What Sets It Apart
- Three ways to name a target with one model: a natural-language text prompt, a visual cue (a mask on a video frame), or a temporal span — no retraining per sound category.
- Every separation returns two tracks, the isolated target and the residual (everything else), so the output is composable: subtract it, recombine it, or feed it downstream.
- It rides on the Perception-Encoder Audio-Visual (PE-AV) encoder, which is what lets a single backbone reason jointly over audio and the video frames used as visual prompts.
- Optional span prediction can detect when a sound occurs on its own, and re-ranking via CLAP, Judge, and ImageBind lets the model score its own separations.
Who It's For and the Trade-offs
Great fit if you build audio post-production, dataset-cleaning, or video-understanding pipelines and want one promptable separator instead of a zoo of source-specific models. The "-tv" variants trade some generality for target correctness and stronger visual prompting, so you pick a point on that curve. Look elsewhere if you need a permissive license: the weights ship under Meta's custom SAM License, not Apache or MIT. Validate on your own audio first — fidelity varies across speech, music, and general sound effects.
