Every new vision-language model ships with its own eval scripts, data formats, and idiosyncratic notion of "accuracy," which turns a head-to-head comparison into an afternoon of plumbing rather than research. The premise here is that evaluating a multimodal model should be one command, not a per-paper engineering project — choose a model, choose a benchmark, and get a number directly comparable to everyone else's.
What Sets It Apart
- One interface, many subjects. A single CLI covers 220+ large multimodal models and 80+ image and video benchmarks, so swapping either side of an experiment is a config change, not a new codebase.
- Generation-based scoring. Models are judged on the text they actually generate rather than on logit tricks, matching how they are used in practice and keeping closed-API models on equal footing with open weights.
- LLM-assisted answer extraction. Free-form responses are parsed by an LLM before scoring, which tolerates verbose or "thinking" outputs that naive string matching would mark wrong.
- Scales when you need it. Multi-node distributed inference through LMDeploy and vLLM keeps large models and long benchmark suites tractable.
Who It's For
Great fit if you publish or track LVLM results and want reproducible, apples-to-apples numbers, or if you are integrating a new model and would rather write one generate_inner() method than a whole harness. Look elsewhere if you need training, fine-tuning, or RLHF tooling — this is strictly an evaluation layer — or if your task is unimodal text, where its sibling OpenCompass is the better fit.
