A plausible-looking image can still fail the task if it combines the wrong details from its references. LongBanana makes that gap measurable: the challenge is not just image quality, but whether a system can retrieve scattered evidence, bind it to the right entities, and honor constraints across the whole scene.
Key Findings
- The benchmark’s 90 tasks use about 32 reference images each and more than 20 constraints, turning dense evidence integration into a repeatable test rather than a vague prompt-following judgment.
- Its 4,670 expert-specified checks assess content, relationships, and global constraints separately. This makes failures diagnosable—for example, a missing object can be distinguished from a correct object assigned the wrong appearance or relation.
- Expert review includes a double-labeled subset with 89.4% exact agreement and a 0.83 quadratic weighted kappa, providing a check on the consistency of the evaluation labels.
- LongBanana-Gen tests agent workflows as well as direct generation. The reported results suggest that an agent harness can improve on direct generation, while the benefits of a ReAct–Plan–Exec–Reflect workflow depend on the harness.
Who It’s For—and the Trade-Off
Great fit for researchers comparing multi-reference image generators or studying how agents retrieve, plan around, and repair visual evidence. The graph-grounded checks make results easier to diagnose than a single holistic score. Look elsewhere if you need a general image-quality benchmark: this one targets constraint-heavy synthesis from many references, and its curated tasks cannot represent every real-world image-generation setting.