Wenlong Huang
Wenlong Huang
Description: ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
foundation models (14) robotic manipulation (5) structural representation (1) model-based planning (1)
Representing robotic manipulation tasks as constraints that associate the robot and the environment is a promising way to encode desired robot behaviors. However, it remains unclear how to formulate the constraints such that they are 1) versatile to diverse tasks, 2) free of manual labeling, and 3) optimizable by off-the-shelf solvers to produce robot actions in real-time. In this work, we introduce Relational Keypoint Constraints (ReKep), a visually-grounded representation for constraints in robotic manipu
Given RGB-D observation and free-form language instruction, DINOv2 is used to propose keypoint candidates on fine-grained meaningful regions in the scene. The image overlaid with keypoints and the instruction are fed into GPT-4o to generate a series of ReKep constraints as python programs that specify desired relations between keypoints at different stages of the task and any requirement on the transitioning behaviors. Finally, a constrained optimization solver is used to obtain a dense sequence of end-effe
Since keypoints are tracked in real-time, the system can replan its actions in closed loop, both within stages and across stages . Here, the operator randomly perturbs the objects and the robot, but the robot can quickly react to it. Note that after the robot tilts the teapot, if at this moment the cup is moved away from the robot, it would restore the teapot to be upright and attempt the pouring action again. Real-time solution is visualized on the right.
Wenlong Huang
Wenlong Huang