VLM Evaluation Harness

Visual Action ModelsQwen2.5-VLSim to Real

Created a closed loop evaluation environment for vision language models operating through physical consequences. The model navigates an obstacle course from camera observations and onboard state, while the evaluation harness records perception, planning, control, and outcomes to expose exactly where behavior succeeds or fails.

Perfect perception success: 30/30
Control rate success: 13/30 → 30/30
Dead end stall time: 69% → 4%
Dead end success: 17/30 → 28/30

Decision

What the model sees

672 × 672 RGB

What it perceives

Qwen2.5-VL boxes

What the car can do

Physics prediction: solid committed, dashed free

Steer: +0.60
Throttle: 1.00
Gear: Forward

Frame tSimulation advancesFrame t + 1

01Problem

Improving vision language models applied to robots requires knowing not only when an agent fails, but which part of the system caused the failure.

A single task success rate collapses perception, planning, control, and recovery into one number, making it difficult to determine what actually needs to be improved when a robot fails a task.
Failures can originate anywhere in the robotic system, from sensing and perception to planning, control, or vehicle dynamic interference. Without an evaluation system that isolates each stage, it is difficult to determine which component actually needs to be improved.
Evaluating these systems requires decision level traces so that we can more easily connect what the specific failure mode was.

02Approach

Built a closed loop evaluation harness that separates perception, planning, vehicle dynamics, and recovery so each stage can be diagnosed.

Qwen2.5-VL receives a 672 × 672 chase camera frame and returns target and obstacle bounding boxes, keeping visual perception isolated from downstream planning and control logic.
A physics aware planner evaluates candidate steering, throttle, and braking commands by predicting the vehicle's future motion with steering lag, momentum, and stopping distance instead of assuming instantaneous control.
The vehicle dashboard exposes states such as velocity, steering, and recent motion, while a motion trace detects stalled behavior and enables recovery through reversing.

03Result

Controlled evaluations exposed specific failure sources and showed how changing the correct part of the harness directly improved closed loop behavior.

With perfect detections feeding the same planner, the system solved 30/30 courses, establishing that the planning and control stack could succeed when perception was removed as a source of error.
On dead end courses, adding vehicle state reduced time spent stationary from 69% to 4% and increased oracle success from 17/30 to 28/30, isolating motion awareness as the missing capability.
Control rate produced another isolated failure mode with the model solving 9/20 courses at 1 Hz versus 20/20 at 10 Hz, showing that policy quality alone does not determine performance.