Dhruv Datta
[ designing geometry ]0%

Vision Action Autonomous Robot

PythonPyTorchOpenCVROS2C#Unity

Created a Vision Action Model designed to autonomously navigate a simulated robot around a racetrack. A CNN learns steering behavior through imitation learning and controls the robot through ROS2, with Unity serving as the physics based simulation environment.

CNN parameters
418k
Demonstration frames
4.5k
Held-out test frames
900
Steering actions
3
System
CAMERARGB · 160 × 120ROS 2 TOPICsensor_msgs/ImageCNN POLICY3 CONV STAGES · 128 HIDDEN · 3 ACTIONSCONTROLLERPUBLISHES /cmd_velUNITY ROBOTMOVES, NEXT FRAMEEVERY MOVE CHANGES THE NEXT CAMERA FRAMELEFTSTRAIGHTRIGHT0.99geometry_msgs/Twistlinear.x2.00angular.z−2.50CAMERARGB · 160 × 120ROS 2 TOPICsensor_msgs/ImageCNN POLICY3 CONV STAGES · 128 HIDDEN · 3 ACTIONSCONTROLLERPUBLISHES /cmd_velUNITY ROBOTMOVES, NEXT FRAMEEVERY MOVE CHANGES THE NEXT CAMERA FRAMELEFTSTRAIGHTRIGHT0.99geometry_msgs/Twistlinear.x2.00angular.z−2.50

01Problem

Recreate the core vision to action loop behind Physical AI systems in a robotics simulation.

  • Build the complete closed loop pipeline from onboard camera observations to learned steering decisions and physical robot movement.
  • Keep simulation, policy, and control separated so the architecture resembles the modular sensor to policy to control stack used in real robotics systems.
  • Used Unity as a robotics env while keeping perception, policy, and control independent through ROS 2. This mirrors a sim to real workflow and lets the same stack move to Isaac Sim or real hardware without changing the architecture.

02Approach

Train a behavior cloning policy that learns steering directly from human driving demonstrations.

  • Collected 4.5k labeled camera frames paired with LEFT, STRAIGHT, and RIGHT actions, with 900 frames held out in consecutive blocks to reduce leakage between similar video frames.
  • Processed each observation from 160 × 120 to 64 × 48 RGB, normalized pixel values, and used the identical OpenCV pipeline during training and live deployment.
  • Trained a 418k parameter CNN with 3 convolution stages, a 128 unit hidden layer, weighted cross entropy, and Adam at 0.001 to handle the uneven steering distribution.

03Result

Deployed the learned policy inside the system and successfully drove the robot autonomously through the track.

  • Achieved 69.5% balanced accuracy across 900 held out frames, with each steering class contributing equally despite the uneven dataset.
  • The accuracy score measures individual frame predictions, not autonomous driving performance. A wrong prediction on one frame does not necessarily cause a driving failure, and a correct prediction does not guarantee stable behavior over time.
  • The more meaningful result was closed loop performance which was extremely successful in testing as the robot was able to drive around the course from various starting conditions.