Embodied visual tracking demands both reliable target identification from a natural-language cue and closed-loop trajectory reasoning under onboard vision. Many vision-language-action (VLA) approaches fold identification into abstract spatial latents, making supervision and image-space validation difficult. ReferTrack takes the opposite tack: make the referring decision explicit in image space first, then condition waypoint generation on that grounded choice. This simple decomposition yields clearer supervision, preserves appearance/motion cues, and improves identification-heavy tracking performance.
Key Findings
- Explicit refer-then-track decomposition: the model first picks the referred target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on that image-grounded selection — this reduces failure modes caused by latent ambiguity in end-to-end VLA pipelines.
- Temporal-viewpoint-bbox indicator (TVBI) tokens: a sliding-window queue of previously selected bounding boxes is encoded and injected into the visual history, retaining geometric and motion cues across timesteps so the tracker keeps sight of target motion patterns.
- Data and co-training: co-training on a custom Refer‑QA dataset sharpens identification under ambiguity and distraction, improving robustness when multiple similar objects are present.
- Empirical results: on EVT‑Bench ReferTrack achieves 89.4% / 73.3% / 74.1% success on the single-target, distracted, and ambiguity splits, matching or outperforming several multi-camera baselines on identification-heavy tasks. The method also transfers to real legged and humanoid robots in sim-to-real tests.
Who it's for and tradeoffs
Great fit if you build embodied agents that must follow language-specified targets using a single forward camera and you care about reliable identification under clutter or ambiguity. The architecture is especially relevant when image-space interpretability and debugging of the identification step matter. Look elsewhere if your system already has multi-view sensing, explicit external trackers, or constraints that preclude running a bbox-indexing frontend; ReferTrack focuses on image-grounded referring plus waypoint decoding rather than multi-sensor fusion or low-level locomotion control.
How it works (brief)
ReferTrack indexes candidate detections into a short-term memory and runs a referring module to select the target bbox. The selected bbox stream over time is summarized via TVBI tokens injected into the visual history encoder. A waypoint decoder then generates future motion waypoints conditioned on the grounded selection and visual history. Co-training with Refer‑QA improves the referring module's accuracy under semantic ambiguity. The authors release code and report both simulation benchmarks and robot deployments demonstrating sim-to-real transfer.
