Phone interactions expose a practical gap for agents: completing real mobile workflows requires choosing between GUI touches, device-side commands, and host tools — and proving the intended side effect actually happened. Phone Harness treats the phone as a mixed-action surface and makes those choices executable and auditable so an agent can finish end-to-end mobile tasks, not just predict UI events.
What Sets It Apart
- Mixed-action routing: the harness unifies three execution surfaces — GUI delegation (tap/scroll via iPhone Mirroring or injected events), device CLI (adb input/screencap on Android), and host-side tools (OCR, helpers, cloud APIs) — letting an agent pick the most reliable mechanism for each step. This reduces brittle purely-vision approaches and makes results verifiable.
- Cross-platform day-to-day helpers: the same high-level helpers work across a personal iPhone, desktop-driven Android, rented cloud Android, and a cloud-hosted iPhone. That consistency simplifies agent skill code and test coverage for mixed-device workflows.
- Verifiable traces and fallback modes: captures (screenshots, accessibility trees, OCR) and deterministic host actions produce auditable logs that can be checked by verifiers or benchmarks; accessibility trees are used when available and Vision OCR is a fallback.
- No phone modification: operates without jailbreaking or installing phone-side agents; iPhone control uses macOS Mirroring and HID-level events, Android uses adb, and cloud phones are driven over hosted APIs.
Who It's For and Trade-offs
Great fit if you need an agent to perform and prove mobile workflows (e.g., complete a purchase, configure device settings, or exercise multi-step app flows) and want a consistent developer surface across real and rented devices. It’s also useful for researchers building benchmarks for embodied or phone-use agents because it provides auditable traces.
Look elsewhere if you require multi-touch gestures, camera/FaceID-driven flows, DRM-protected video interactions, or fully headless control with no manual setup: unlocking, initial pairing, or granting OS permissions remains a user step. OCR sees text better than unlabeled icons, so unlabeled UIs may need vision-capable models or screenshots for reliable targeting.
Where It Fits
Phone Harness occupies a middle ground between pure GUI-only controllers and full device instrumentation: it’s more reliable than blind visual tapping because it leverages accessibility and CLI surfaces where available, but it remains less invasive than installing a persistent device agent or modifying OS internals.
Practical implications
Expect straightforward agent development using the provided helpers and skill files, easier cross-device testing via the cloud phones, and trace-backed verification for benchmarks or automation suites. Operational limits (no multi-touch, some manual pairing, OCR limitations) should be factored into task design and evaluation metrics.
