Why this matters
When standard RGB renders away sensor evidence (low light, HDR clipping, or exposure artifacts), models that can use measurement-domain inputs (raw/camera-measurement channels) may recover factual cues that RGB-native models miss. MeasL-Bench provides a controlled, capability-driven held-out set to quantify that gap and probe hallucination risk when visual evidence is degraded.
What Sets It Apart
- Paired RAW ↔ RGB design: each RAW (measurement-domain) record has a matched RGB counterpart so evaluators can measure how much grounding improves when sensor evidence is preserved — not just overall VQA skill, but evidence recovery under degraded rendering.
- Capability taxonomy and slicing: 13 capability labels (e.g., HDR Evidence Recovery, Low-Illumination Evidence Recovery, Spatial Relation Understanding) let you report targeted slices instead of a single aggregate score, helping diagnose specific failure modes.
- Realistic image assets and scale for evaluation: 2,183 test rows and ~3,812 local image files enable controlled experiments without reliance on external image hosting; format and layout are optimized for direct inclusion in PRSIMVL-style eval pipelines.
- Evaluation-first orientation: the release includes an explicit protocol for matched inference, lexical metrics (BLEU/ROUGE-L) and optional LLM-as-judge evaluation so results are comparable across methods.
Who It's For & Trade-offs
Great fit if you are researching measurement-grounded vision–language methods, studying hallucination when RGB evidence is insufficient, or building pipelines that ingest raw/camera measurement channels. It’s also useful for reproducible, slice-level comparisons between measurement-aware and RGB-native VLMs.
Look elsewhere if you need very large-scale training data (MeasL-Bench is a held-out benchmark with ~2k examples, not a pretraining corpus), require fully public commercial licensing for downstream products (released under CC-BY-NC-4.0), or cannot accommodate raw/measurement image formats and the tooling to process them. The dataset prioritizes controlled evaluation fidelity over volume.
