Multimodal context learning is important because many real tasks require models to learn new, task-specific knowledge from images, tables, or web pages rather than relying solely on pretraining. This paper exposes a large gap: when context itself is multimodal, current models struggle not only to retrieve supporting regions but to apply and internalize new information needed to solve the task.
Key Findings
- Benchmark scope and construction: CLBench-V aggregates converted public datasets and newly constructed tasks using automated construction and filtering to produce 3,443 instances across domains such as science, finance, long-document understanding, spatial reasoning, and web-based visual QA. This design isolates three failure modes: context grounding, new information application, and new knowledge learning.
- Empirical outcome: Evaluations on six recent multimodal models yield a best overall score of 0.2847, indicating substantial headroom. InternVL3.5-30B-A3B leads on context grounding and new knowledge acquisition, while Qwen3.5-Plus leads on new information application, highlighting model-specific strengths.
- Dataset and analysis utility: The benchmark includes 3,443 instances and 31,607 verification rubrics, enabling fine-grained diagnosis (judge reliability, context length, image count) and representative failure-case analysis to guide model improvements.
Who it's for + Tradeoffs
Great fit if you need a diagnostic benchmark to measure and debug how multimodal agents learn from task-specific visual+textual context—especially for research on grounding, RAG-style retrieval with images/tables, and domain adaptation in science or finance. Look elsewhere if you only need pass/fail accuracy on isolated visual QA or short-context retrieval: CLBench-V focuses on tasks where new, out-of-pretraining knowledge must be extracted, applied, and internalized, and its automated construction can introduce dataset noise or domain artifacts that require careful interpretation. The benchmark favors diagnostic depth over turnkey evaluation metrics and is most useful when combined with targeted model probes and localized retrieval strategies.
