Combining modalities in one parser is not just about accepting more input types: it lets the model align what a video shows with what its soundtrack says while perception is happening. That design choice also gives all tasks a common structured output, making the results easier for downstream systems to consume consistently.
Key Findings
- A single shared encoder handles visual and audio inputs, with video frames and audio chunks aligned in a common time coordinate system. This places cross-modal matching inside perception rather than leaving it entirely to the text decoder.
- OmniSchema standardizes outputs across ten parsing tasks, from document layout and tables to charts, geometry, audio, and video. A common schema makes results easier to validate and reuse across different input types.
- The 5B-parameter model reports 96.96 overall on OmniDocBench v1.6 and 75.08 average on OmniParsingBench, leading the open-weight models listed in the paper’s comparison. Its broad coverage is particularly relevant when a pipeline needs more than OCR or image captioning alone.
- OmniSchema-Aware On-Policy Distillation (OSAD) separates structural tokens from content tokens in its training objective and refreshes the reference model between rounds. This ties post-training to the same schema the parser must produce at inference time.
Who It Suits—and the Trade-off
A good fit for teams building document or media-understanding pipelines that need structured, spatially or temporally grounded outputs across several modalities. It is less compelling when the task is narrowly scoped and a specialized single-modality model already meets accuracy, latency, or deployment constraints; the reported benchmark scores do not by themselves establish performance on every domain or production workload.
Training and Architecture
The encoder starts from a pretrained language model, with modality-specific input stems and projection heads surrounding shared perception layers. For video, fusion layers allow frames to attend to audio from the same moment. OSAD trains on the model’s own admitted parses, using a reference-conditioned copy to provide token-level targets; the trained weights then become the next round’s reference.