Arithmetic mistakes remain a clear, measurable weak spot for many language models; isolating elementary calculation in a compact, structured benchmark makes those failure modes easier to quantify and compare. ArithMark 3.0 packages short continuation-style word problems so that base models can be scored directly by log-likelihood without requiring instruction-following or chain-of-thought outputs.
What Sets It Apart
- Continuation-style format tuned for base-model log-likelihood scoring: prompts end mid-sentence with four possible continuations, so evaluation can use raw continuation probabilities rather than generation heuristics. This avoids conflating the model's verbalization skill with its underlying numerical computation.
- Small but well-balanced suite: 1,000 examples, 4 answer choices each, exactly balanced correct-answer positions, and 47 templates spanning 17 elementary topics (addition, subtraction, multiplication, division, geometry, fractions, time, money, two-step problems, etc.). The dataset maps topics to grade bands (Grades 1–5) and labels each example with difficulty (easy/medium).
- Lightweight and reproducible baseline tooling: includes a bencharithmark-3.py script for standard scoring workflows and a published leaderboard of 78 baseline models to contextualize results.
Who It's For and Tradeoffs
Great fit if you need a compact, targeted quantitative measure of a model's elementary arithmetic ability across diverse short-word-problem templates — useful for model comparison, ablation studies, and tracking regressions after finetuning. The dataset is also convenient when you want evaluation that minimizes prompt-engineering variance by using continuation log-likelihood.
Look elsewhere if your goal is advanced symbolic math, open-ended explanation quality, multilingual arithmetic, or tasks that require long chains of reasoning or generated step-by-step solutions: ArithMark 3.0 intentionally omits chain-of-thought evaluation and broader reasoning benchmarks. Also note the synthetic, English-only nature and primary suitability for base-model scoring; instruction-tuned or explanation-focused evaluations may need complementary datasets.
