Most standard MT benchmarks and automatic metrics either become saturated or reward-hacked as models improve, leaving practitioners blind to concrete, repeatable failure modes. The Last Translation Benchmark (LTB) tackles this by collecting human-authored hard cases and attaching explicit, machine-checkable verification rules so each example yields objective pass/fail outcomes rather than a single score.
What Sets It Apart
- Example-level fail rules: Contributors specify precise verification rules (e.g., lexical choices, gender agreement, omission of entities) so evaluations reveal the exact nature of mistakes rather than an aggregate metric.
- Multimodal hard cases: Inputs include text, images, audio, and video to expose failures that arise from multimodal grounding or modality-specific ambiguities.
- Peer-reviewed, evolving dataset: LTBv1 (3,456 accepted submissions) is curated and reviewed; the dataset remains live and accepts new contributions, enabling a growing leaderboard and longitudinal tracking.
- Reproducible fail detection: Because rules are explicit and automatable, the benchmark supports repeatable regression testing and targeted stress tests for MT systems.
Who it's for and trade-offs
Great fit if you need to diagnose and monitor concrete translation errors (researchers building model audits, teams validating translation safety or fairness, or benchmark designers wanting fail-oriented tests). Look elsewhere if you only need broad BLEU-style comparisons or very large-scale coverage: LTB emphasizes challenging, curated cases over exhaustive parallel corpora, so it is complementary to large-scale benchmarks rather than a replacement.
Where it fits
Use LTB alongside traditional metrics and large test sets: apply LTB to pinpoint weaknesses revealed by aggregate metrics, or to design focused fine-tuning and evaluation pipelines that target repeatable error classes.
How it works
Contributors submit candidate inputs with a verification rule and optional media. Reviewers accept high-quality submissions; accepted examples include source/target language metadata, linguistic annotations, and past translations annotated with verification outcomes. The dataset exposes a simple JSON and a public leaderboard so teams can run models against the same fail rules and track regressions over time.
