3D Perception & Localization Stack
Built components of a 3D autonomy perception stack including sensor calibration, point-cloud processing, object detection, localization, and tracking.
A 3D autonomy stack is only as good as the transform between LiDAR and camera, the localization it feeds, and whether detections persist as tracks the planner can reason about. On KITTI, that means target-free calibration, a graph-based pose estimate, BEV detection, and multi-object tracking — then a planner that can actually use the result.
Raw odometry on the evaluated sequences started at 0.68 m ATE RMSE. The question was whether a Gauss–Newton Graph-SLAM layer could cut that in half without a special calibration target in the scene.
- Implemented target-free LiDAR–camera calibration via stereo unprojection and ICP, then Graph-SLAM with Gauss–Newton optimization, reducing KITTI ATE RMSE from 0.68 m to 0.33 m.
- Trained and evaluated a CenterPoint-style BEV 3D detector on LiDAR point clouds (> 0.6 mAP on KITTI) and added Kalman-filter tracking with Hungarian association for online multi-object tracking.
- Downstream, ran a Frenet-frame sampling planner with Pure Pursuit tracking in Waymax on the resulting tracks.
Perception
CenterPoint-style bird's-eye-view 3D detection on LiDAR, producing boxes that a tracker can hold across frames. Calibration uses stereo unprojection plus ICP so the camera and cloud agree without a checkerboard in every scene.
Localization / State Estimation
Graph-SLAM with Gauss–Newton optimization on the pose graph. ATE RMSE dropped from 0.68 m to 0.33 m on the evaluated KITTI sequences — the number that tells you whether later modules are localized or just guessing.
Planning
Frenet-frame sampling with Pure Pursuit tracking in Waymax, consuming the tracked 3D objects rather than a hand-authored scene.
Calibration error masquerades as detection error. If LiDAR and camera disagree by a few pixels at range, the tracker looks noisy and the planner looks timid.
Graph-SLAM is not a drop-in replacement for odometry — it is a constraint problem. The 0.68 → 0.33 m ATE drop came from the residual model, not from adding more points to the cloud.