Wearable data are long-horizon, noisy, and multimodal, yet most LLM benchmarks focus on text or short-horizon numeric tasks. WearableQA fills that gap by turning real users' longitudinal wearable records into a diagnostic testbed that stresses numerical, temporal, and cross-signal reasoning in realistic, device-noise conditions. The benchmark preserves authentic distributions (up to 500 days per user) and pairs physiological interpretation with computational operations over time series.
Key Findings
- Scale and realism: 4,084 ten-option multiple-choice questions built from wearable time series, blood biomarkers, and demographics of 200 users (up to 500 days per user). This setup preserves device noise and inter-individual variability so performance reflects real-world difficulty.
- Dimensioned taxonomy: 16 question types organized along two axes—data vs health reasoning (computation over measurements vs physiological interpretation) and single- vs cross-signal reasoning (individual-signal analysis vs multimodal integration). This makes failure modes diagnostically informative.
- Dual-grounding for reliability: questions are constructed using literature-grounded physiological relationships and statistically validated population patterns, reducing spurious or dataset-specific artifacts while retaining real distributions.
- Model differentiation: evaluation across 14 LLMs yields accuracies from about 19.6% to 72.9% (10% chance baseline), with most models below 60%, indicating the task remains challenging and useful for benchmarking progress.
Who it's for and tradeoffs
Great fit if you are developing or evaluating LLMs, agentic toolchains, or tool-augmented systems that must reason about longitudinal physiological and behavioral signals, and you need a realistic, diagnostic benchmark to probe numeric, temporal, and multimodal reasoning. It’s also useful for researchers studying robustness to device noise and inter-individual variability.
Look elsewhere if you need labeled clinical outcomes, causal inference across interventions, or regulatory-grade clinical validation—the benchmark targets reasoning ability over observational wearable records rather than providing guidelines for deployment in clinical care. The multiple-choice format favors scalable automatic evaluation but may not capture every nuance of open-ended clinical explanation.