Most mobile-agent benchmarks focus either on surface-level GUI control or offline API matching, missing the middle ground where planning, tool selection, and runtime verification matter. MobilePA-Bench addresses that gap by pairing an interactive, stateful function-calling sandbox with evidence-based checks to evaluate whether an agent actually effected the intended side effects in realistic mobile workflows.
Key Findings
- Benchmark design: Runs live app databases and returns structured feedback across 13 functional domains and 212 realistic mobile tools, enabling deterministic verification of task effects rather than heuristic screen-score judgments. This makes success measurable by side effects (database entries, tool outputs) rather than only visual cues.
- Evaluation dimensions: Moves beyond single-action control to measure Sub-agent Collaboration (task decomposition and delegation), Memory Usage (recalling stored preferences and histories), and Skill Usage (invoking composite pre-packaged skills vs. planning primitive steps). These axes expose planning and orchestration weaknesses not visible in GUI-only tests.
- Empirical result: State-of-the-art LLMs suffer sharp performance drops under strict tool ordering, permission limits, and unexpected runtime errors, indicating current models are brittle when faced with realistic mobile constraints. An interactive function-calling sandbox enables more diagnostic evaluations and supports agentic reinforcement learning research.
Who it's for and trade-offs
Great fit if you are a researcher or practitioner building mobile agents that must integrate tool use, long-horizon planning, and multi-agent decomposition, and if you need deterministic, evidence-based evaluation. Look elsewhere if your goal is purely GUI-level action prediction, lightweight synthetic evaluation, or benchmarking against proprietary closed-app ecosystems—the sandbox requires engineering to emulate app backends and may not capture every idiosyncrasy of closed commercial apps. The benchmark is diagnostic and research-oriented rather than a turnkey production validation for every commercial app.