Bridging large pretrained vision-language models (VLMs) with practical driving systems matters because VLMs encode broad visual and language knowledge, but do not natively expose explicit 3D scene structure or trajectory outputs required for autonomous driving. Qwen-Drive-1.0 tests whether a single pretrained VLM can serve as a shared backbone for 3D perception, driving VQA, and motion planning by attaching inspectable, task-specific modules rather than altering the VLM itself.
Key Findings
- External BEV perception head: Fuses vision-encoder features and VLM outputs to produce bird's-eye-view (BEV) representations that jointly predict 3D object detection, semantic occupancy, and BEV map segmentation. Achieves competitive 3D-perception metrics (e.g., ~43.95 mAP and 60.99 mIoU on nuScenes; ~43.45 mAP and 71.27 mIoU on OpenScene).
- Planning Expert and trajectories: Conditions on cached VLM keys/values and uses flow matching to generate future ego trajectories. Shows strong motion-planning performance (Predictive Driver Model Score 90.7 on NAVSIM; Rater Feedback Score 7.91 on Waymo Open Dataset end-to-end test split) and promising closed-loop results in simulation.
- Staged training recipe: Jointly trains perception, language, and planning objectives by combining driving-specific supervision with general-purpose vision-language data, preserving broad instruction-following and multimodal understanding while adding driving competence.
Who it’s for and tradeoffs
Great fit if you study multimodal transfer to embodied driving tasks, want an inspectable BEV probe on top of a pretrained VLM, or aim to unify perception, VQA, and planning in one model. Look elsewhere if you need a purely optimized, standalone state-of-the-art detector/planner—Qwen-Drive prioritizes integration with a VLM backbone and research on unified representations over squeezing every last point on specialist leaderboards. The approach also depends on availability of multi-view driving data and careful dataset unification for trajectories.
Where it fits
Positions itself between pure vision-based driving stacks and end-to-end LLM-based agents: it retains the pretrained VLM's autoregressive decoder and attaches decoupled, inspectable modules (BEV head, Planning Expert), enabling both textual VQA outputs and structured trajectory predictions from shared representations.
Method highlights
- Backbone: uses Qwen3.5-4B as the shared VLM backbone without architectural changes.
- Perception head: explicit BEV construction that serves as a probe into 3D signals encoded by the VLM.
- Planning: flow-matching for trajectory generation and unified waypoint representation across datasets to support joint training and evaluation.
Taken together, Qwen-Drive-1.0 is an exploratory but empirically grounded proof that pretrained VLM representations can be extended to explicit 3D perception and trajectory planning for driving while largely preserving general multimodal capabilities.