Real-world speech mixtures are harder than the usual two-speaker benchmark: distance, reverberation, changing speaker counts, and sudden noise all affect what reaches a microphone. The central insight here is that better streaming extraction need not come from repeatedly running a small separator; a single multi-scale pass can improve separation while keeping computation low.
Key Findings
- RealSSA broadens the evaluation setting. Its simulated scenes vary near-field and far-field interference, room acoustics, and noise, while a separate real-recorded set tests performance beyond simulation. This helps expose failures that fixed two-speaker mixtures can hide.
- Falcon uses a single encoder–decoder separation pass. Its multi-scale time-frequency modeling avoids the repeated separator cycles used by some lightweight causal systems, reducing redundant computation.
- Training targets leakage without adding runtime cost. An encoder-space speaker contrastive loss pulls the estimate toward the target and away from near-field interferers. The paper reports 0.5 million parameters and 51.3 ms GPU latency, and reductions of 91.9% in parameters, 20.2× in computation, and 11.1× in GPU latency versus causal AV-TFGridNet.
Method and Scope
The system combines mixture audio with video of the target speaker’s mouth, providing a visual cue for selecting whose voice to extract. The reported evaluations cover RealSSA and established two-speaker mixtures; results are strongest evidence for the tested settings, not a guarantee for every microphone, camera, or live deployment.
Who It’s For
A good fit for researchers and engineers exploring low-latency, video-guided speech separation, especially when edge-device cost and realistic interference matter. Look elsewhere if the task has no usable target-speaker video, requires text-only speaker selection, or needs evidence on deployment conditions not represented in the reported tests.
