FinanceGym targets a practical but underexplored layer of agent competence: executing precise UI actions inside finance software given multimodal context. The core insight is that competent financial agents need reliable visual grounding, state tracking across multi-step workflows, and exact action selection — not just text-based reasoning — and FinanceGym supplies human-demonstrated trajectories to evaluate that execution layer.
What Sets It Apart
- Human-demonstrated, step-aligned GUI trajectories: examples pair operator narration and tool calls with synchronized screenshots and precise next-action labels, so evaluations reflect realistic operator behavior rather than synthetic heuristics.
- Multimodal, execution-focused targets: the benchmark expects concrete computer_use operations (click, drag, scroll, typing) in a nominal 1000×1000 coordinate space, enabling reproducible grounding and action scoring across models.
- Rich interaction types and context: episodes include multi-step dialogues, prior tool calls, scrolls and drag sequences, and image references, which stress context tracking and multi-action sequencing rather than single-step classification.
- Evaluation-ready packaging: a public test split with an evaluator that reconstructs OpenAI-style multimodal messages and scores predicted actions against JSON ground-truth, encouraging comparable, repeatable evaluations.
Who it's for — and tradeoffs
Great fit if you research or develop multimodal agents that must interpret GUIs and issue precise UI operations (visual grounding, tool-use policy, long-horizon state tracking). It’s designed as an evaluation benchmark rather than general-purpose training data: the public release is a test split (training data by request), and the dataset intentionally excludes real PII or production credentials. Look elsewhere if you need production-grade financial content, regulatory-compliance tooling, or datasets with live transactional data; FinanceGym measures execution fidelity, not advisory quality or portfolio performance.
