Autonomous Driving Stack on a Real GEM e4
Built and validated an end-to-end autonomous driving stack integrating perception, behavioral planning, local planning, and closed-loop control on a real GEM e4 platform.
A physical Polaris GEM e4 has to follow lanes, avoid static obstacles, and stop for stop signs — in Gazebo first, then on the real vehicle. The failure modes that matter are the ones simulators hide: lighting, calibration drift, latency, and a planner that is kinodynamically feasible but still clips a cone.
The engineering problem was to assemble a modular stack — camera/LiDAR perception, a body-frame traffic model, a behavioral finite-state machine, a local Dynamic Window planner, and closed-loop tracking — and keep the whole loop inside a 10 Hz control cycle on real hardware.
Manual annotation of on-vehicle rosbags would not scale. Perception had to be trained from auto-labeled data, then fused into one scene representation the planner could actually consume.
- Trained a ResNet-34 U-Net lane segmentor (0.79 Dice) and a YOLOv8s multi-class detector (0.995 mAP@0.5 on stop signs; 0.896 on boxes) from rosbags auto-labeled with GroundingDINO and classical-CV pseudo-labels — no manual annotation.
- Fused camera and LiDAR outputs into a single body-frame traffic model used by both the behavioral layer and the local planner.
- Implemented an FSM (lane following, collision avoidance, decelerate-to-stop, stop) plus a Dynamic Window planner that samples kinodynamically feasible rollouts and scores them on goal, speed, lane adherence, and static/dynamic clearance.
- Closed the loop with Stanley lateral tracking and PID speed control at 10 Hz, and validated the stack in Gazebo and on the real GEM e4 across lane-following, obstacle, pedestrian, and stop-sign scenarios.
Perception
U-Net semantic segmentation for lane markings; YOLOv8s for vehicles, pedestrians, boxes, and stop signs. Training data came from an automated rosbag pipeline rather than hand labels, so the detector and segmentor could be retrained as new bags came in.
Localization / State Estimation
Camera and LiDAR detections are transformed into the vehicle body frame and fused into one traffic model. Downstream modules never consume raw image or cloud coordinates — they consume a consistent ego-centric scene.
Planning
A finite-state machine selects the behavior. Inside each mode, DWA samples hundreds of constant-curvature rollouts per cycle and scores them on goal progress, speed, lane adherence, and clearance — the same cost the hero visualization on the homepage is drawing.
Control
The winning path is tracked with Stanley (lateral) and PID (longitudinal) at 10 Hz. Cross-track error on the real vehicle was 0.15 m, with 100% scenario success and 0 collisions across the evaluated cases.
ML / Data
GroundingDINO plus classical-CV pseudo-labeling turned unlabeled rosbags into a training set. That was the only way to iterate perception at the pace of on-vehicle collection.
A planner that is collision-free in Gazebo still needs a body-frame scene that is consistent under real lighting and latency. The expensive part was not any one module — it was keeping perception, fusion, FSM, DWA, and control inside the same 10 Hz budget.
Auto-labeling is a systems problem: GroundingDINO misses and classical-CV false positives both make it into the traffic model unless you treat the label pipeline as part of the stack, not a one-off dataset job.
Discrete behaviors (stop vs. follow vs. avoid) belong in an FSM; continuous collision geometry belongs in DWA. Mixing those two layers was the tradeoff that actually transferred to the vehicle.