[01] Project
A CNN that learned to drive from my own demonstrations. It steers a simulated robot around a track in Unity using only its camera, with ROS 2 connecting every part.
Drive with WASD. Let go and the policy takes over.
The goal was a realistic closed loop, like the ones used in physical AI. The camera publishes a frame over ROS 2, a CNN predicts left, straight or right, and a controller turns that into a velocity command for the robot in Unity. The robot moves, a new frame comes in, and the loop repeats.
I drove the track myself to record about 4.5k labeled frames. Each frame is resized and normalized with OpenCV the same way in training and in the live loop. The network has three convolution layers and two dense layers, and a weighted loss keeps the rarer right turns from being ignored.
The model reached 69.5% balanced accuracy on held-out frames. The controller votes over the last five predictions to smooth out single bad frames, and the robot drives the track well from a range of starting points. Next steps are new tracks, random starts and recovery data.