A GUI agent can have all the right information and still make the wrong move: what matters at one step may become noise at the next. This paper’s central insight is that multimodal perception must adapt to the current decision, rather than treating every channel as equally relevant throughout an interaction. Otherwise, one mistaken action can pollute later context and push the agent off course.
Key Findings
- Across four omni-modal backbones, the adapter improves Type Match by an average of 33.2 points and raises Exact Match to 3× the frozen-backbone baseline. This suggests selective perception can improve both action choice and its execution details without retraining the underlying model.
- The relevance of modalities changes over time: the dominant channel switches between adjacent steps in 45.3% of cases, and 90.1% of steps contain at least one interfering modality. Uniformly exposing every signal is therefore a source of errors, not merely wasted context.
- Training uses action supervision alone—no modality-relevance labels or backbone fine-tuning. That lowers the annotation and adaptation burden, while keeping the intervention outside the model’s native parameter space.
How It Works
FocusGUI compares observation channels with a step-specific decision reference, compresses the most useful evidence into a compact latent representation, then injects it into the frozen decoder’s embeddings. The design is deliberately restrained: it guides attention without rewriting the backbone. This is especially relevant for long interaction sequences, where errors can compound as each action changes the next observation.
Who It’s For—and the Trade-off
A good fit for researchers building GUI agents that must combine screen content with transient audio, video, history, or device-state cues, especially when backbone fine-tuning or relevance labels are impractical. It is less compelling for screenshot-only tasks or settings where a suitable omni-modal backbone is unavailable; the reported gains also depend on the paper’s evaluation setup and should not be assumed to transfer unchanged to every agent or interface.