Vision-Guided Robotic Manipulation
Vision-guided 6-DoF UR3 arm: camera calibration, object detection, inverse kinematics, and closed-loop pick-and-place.
A 6-DoF UR3 with a suction gripper has to find colored blocks in an overhead camera, map them into the robot world frame, solve inverse kinematics, and execute pick-and-place — then refuse to place if the vacuum bit never came on.
The interesting failures are geometric: a wrong image-to-world transform, an elbow-up IK solution that clips the camera, or a grasp height that looks right in pixels and is 8 mm off in the world.
- Wrote the application stack: HSV blob detection, a calibrated image-to-world transform, product-of-exponentials forward kinematics, closed-form geometric IK, and a pick-and-place state machine with vacuum verification.
- Ran the loop on a real UR3: clear the camera, detect green then yellow blocks, approach, suction, confirm, transfer, place.
Perception
Overhead RGB camera. HSV thresholds isolate green / yellow blocks; OpenCV SimpleBlobDetector returns pixel centers. A 0.1 m scale stick gives pixels-per-meter, camera yaw, and translation so those pixels become world (x, y).
Control
Space-frame screw axes and home pose M for PoE forward kinematics; closed-form elbow-up IK for the gripper pose. Execution: approach from a safe height, enable suction, confirm the digital vacuum bit, transfer, place. Drop-off slots are fixed world coordinates for green and yellow.
Vision-guided manipulation fails at the calibration seam. Once image-to-world is honest, IK and the state machine are straightforward; if it is off, every later number looks like a kinematics bug.
The vacuum bit is the only real grasp detector. Visual confirmation that the cup is over the block is not the same as the block being on the cup.