Many professional tasks are visual in ways that plain-text benchmarks miss: a key measurement may sit in a schematic, a map, or a chart, while the final answer must be delivered as a usable report or spreadsheet. This showcase makes that gap inspectable with real, multi-file workspaces rather than isolated image questions.
What Sets It Apart
- Each task combines multiple source files, including visual artifacts whose meaning depends on layout and domain context. This tests whether an agent can perceive and cross-reference evidence, not just extract text.
- Tasks call for tool use and standalone deliverables such as spreadsheets, slide decks, and reports. Success therefore includes turning analysis into professional work products.
- The public release covers 10 roles across 10 domains, while the broader evaluation set contains 200 tasks. The showcase lets researchers inspect the task format, but it is not a substitute for access to the private scored set.
- Expert rubrics emphasize factual accuracy (75%) alongside visual design (25%), making presentation quality part of the evaluation rather than an afterthought.
Who It’s For—and the Trade-off
A good fit for researchers building or evaluating agents that work with visual documents, technical diagrams, and multi-file tasks. It is less suitable for measuring general language ability or comparing systems on a large, fully public test set: only 10 examples are released for inspection, and results on them may not represent performance on the private benchmark.
