ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
用关系关键点约束实现机器人操作的时空推理Spatio-temporal reasoning for robotic manipulation via relational keypoint constraints
Wenlong Huang · Chen Wang · Yunzhu Li · Ruohan Zhang · Li Fei-Fei | Stanford University & Columbia University
ReKep 将机器人操作任务表示为作用于三维关键点的关系约束函数（Python 程序），由 GPT-4o 结合 RGB-D 观测自动生成，再经分层优化求解机器人动作。无需任务专属训练数据，即可完成多阶段单臂与双臂操作，并具备实时响应外部干扰的能力。ReKep represents a robotic manipulation task as relational constraint functions (Python programs) defined over 3D keypoints, generated automatically by GPT-4o from RGB-D observations and then solved by hierarchical optimization to produce robot actions. Without task-specific training data it completes multi-stage single-arm and bimanual manipulation, and can react to external disturbances in real time.
机器人操作任务涉及多阶段、双臂协作、以及与动态物体的复杂空间关系。现有方法要么依赖大量任务专属训练数据，要么难以对操作过程中的接触关系做精确时空建模，在面对多样化场景时泛化能力有限。Robotic manipulation tasks involve multiple stages, bimanual coordination, and complex spatial relations with dynamic objects. Existing methods either rely on large amounts of task-specific training data, or fail to model the contact relations arising during manipulation precisely in space and time, so their generalization to diverse scenes remains limited.
"We present ReKep, a visually-grounded constraint representation for robotic manipulation … [which] automatically generates constraints from language instructions and RGB-D observations using a vision-language model, which are then solved by a hierarchical optimization procedure."
图 1：ReKep 在倒茶任务中的示意。蓝色关键点标记茶壶把手，红色标记壶嘴，绿色标记杯口。系统通过关键点之间的关系约束函数，驱动机器人完成精细的多阶段操作。Figure 1: Illustration of ReKep on the tea-pouring task. Blue keypoints mark the teapot handle, red marks the spout, and green marks the cup rim. Relational constraint functions over these keypoints drive the robot through a fine-grained multi-stage manipulation.
ReKep 的核心是将操作任务分解为若干阶段（stages），每个阶段包含两类约束：在阶段末必须满足的 sub-goal constraints 和在整个阶段过程中持续满足的 path constraints。约束函数以 Python 程序形式表达，接收关键点的三维坐标并输出数值代价，由 GPT-4o 根据语言指令和 RGB-D 观测自动生成。At its core ReKep decomposes a manipulation task into a number of stages, each carrying two kinds of constraints: sub-goal constraints that must hold at the end of the stage, and path constraints that must hold throughout it. The constraint functions are written as Python programs that take the 3D coordinates of keypoints and return a numeric cost; they are generated automatically by GPT-4o from the language instruction and RGB-D observations.
图 2：ReKep 系统流程。DINOv2 提取 patch 特征并结合 SAM 生成关键点建议，叠加到 RGB 图像后送入 GPT-4o，生成各操作阶段的约束函数 Python 程序，再由分层优化器求解机器人动作。Figure 2: The ReKep pipeline. DINOv2 extracts patch features that, combined with SAM, yield keypoint proposals; overlaid on the RGB image, these are fed to GPT-4o, which emits Python constraint functions for each manipulation stage, and a hierarchical optimizer then solves for robot actions.
关键点提议（Keypoint Proposal）Keypoint Proposal
使用 DINOv2 进行 patch 级特征提取，双线性插值到原始分辨率后，结合 Segment Anything（SAM）获取物体掩码，再用 k-means（k=5）对特征聚类得到语义关键点。关键点以三维 Cartesian 坐标表示，直接对应操作中有意义的部件（把手、壶嘴、杯口等）。Patch-level features are extracted with DINOv2 and bilinearly interpolated back to the original resolution; object masks are obtained with Segment Anything (SAM), and the features are then clustered with k-means (k=5) to give semantic keypoints. Keypoints are expressed as 3D Cartesian coordinates and correspond directly to the parts that matter for manipulation (handle, spout, cup rim, and so on).
约束自动生成（Constraint Generation via GPT-4o）Constraint Generation via GPT-4o
将叠加了关键点标记的 RGB 图像与自然语言任务指令共同输入 GPT-4o，模型输出包含 NumPy 操作的 Python 约束函数。sub-goal constraint 定义阶段目标（如壶嘴与杯口对齐），path constraint 定义过程限制（如避免碰撞）。The RGB image with overlaid keypoint markers is fed to GPT-4o together with the natural-language task instruction, and the model outputs Python constraint functions built from NumPy operations. A sub-goal constraint defines the goal of a stage (e.g. aligning the spout with the cup rim), while a path constraint defines restrictions along the way (e.g. avoiding collisions).
两层优化：首先以 Dual Annealing + SLSQP 全局搜索满足 sub-goal 约束的末端执行器位姿（约 1 秒），随后以局部优化器在满足 path constraint 的前提下生成运动轨迹（约 0.1 秒每步）。前向模型假设末端执行器与被抓关键点之间刚性连接，每 0.1 秒由视觉重跟踪更新关键点位置，实现实时感知-动作闭环，频率约 10 Hz。Optimization has two levels: a global search with Dual Annealing + SLSQP first finds an end-effector pose satisfying the sub-goal constraints (about 1 second), after which a local optimizer generates a motion trajectory subject to the path constraints (about 0.1 second per step). The forward model assumes a rigid connection between the end-effector and the grasped keypoints, and keypoint positions are refreshed by visual re-tracking every 0.1 second, closing the perception-action loop in real time at roughly 10 Hz.
图 3：七项实验任务，覆盖单臂多阶段操作（倒茶、回收易拉罐、收纳书本、封装纸盒、折叠衣物、整理鞋子）和双臂协同折叠，测试系统在多样化真实场景下的泛化能力。Figure 3: The seven evaluation tasks, covering single-arm multi-stage manipulation (pouring tea, recycling a can, stowing a book, taping a box, folding a garment, packing shoes) and bimanual collaborative folding, probing generalization across diverse real-world scenes.
03实验Experiments
在真实机器人平台上评测 7 个任务（每任务各 10 次），对比基线为 VoxPoser（zero-shot 代码生成规划）和 Auto-Annotated（使用 GPT-4V 自动生成关键点标注的改进版本）。另设扰动条件测试（外力干扰）以评估反应式恢复能力，并在 8 类服装折叠任务上测试双臂泛化。Seven tasks are evaluated on a real robot platform (10 trials each), against the baselines VoxPoser (zero-shot code generation for planning) and Auto-Annotated (a variant whose keypoint annotations are produced automatically by GPT-4V). A disturbance condition (external perturbation) additionally assesses reactive recovery, and bimanual generalization is tested on 8 garment-folding categories.
图 5：双臂协作折叠在 8 种服装类别上的策略泛化。Strategy Success（策略生成成功）：52.5%；Execution Success（执行成功）：73.8%。各类别策略成功率从 30%（礼服）到 70%（裤子、短裤）不等。Figure 5: Strategy generalization of bimanual collaborative folding over 8 garment categories. Strategy Success: 52.5%; Execution Success: 73.8%. Per-category strategy success ranges from 30% (dress) to 70% (trousers, shorts).图 4：系统失败案例的错误来源分解。点跟踪（point tracking）是最主要的错误来源，这主要由频繁的遮挡问题导致。其次为约束生成（constraint generation）和优化求解（optimization）中的错误。Figure 4: Breakdown of the error sources behind the failure cases of the system. Point tracking is the dominant source of error, caused mainly by frequent occlusion. Constraint generation and optimization account for the remaining two categories of error.
消融分析（Ablations）Ablations
错误来源分析（图 4）表明，点跟踪失败是系统瓶颈，主要因操作过程中存在"heavy intermittent occlusions"。约束生成（GPT-4o 的输出质量）与优化求解失败（局部极值、不可行问题）构成另外两类主要错误源。The error-source analysis (Figure 4) shows that point-tracking failure is the bottleneck of the system, mainly because of the "heavy intermittent occlusions" that occur during manipulation. Constraint generation (the output quality of GPT-4o) and optimization failure (local minima, infeasible problems) make up the other two major sources of error.
04局限性Limitations
Note:以下局限性均由作者在论文中明确陈述（stated by the authors）。every limitation below is explicitly stated by the authors in the paper.
刚性假设的适用范围有限The rigidity assumption is valid only over a narrow range
前向模型假设末端执行器与被抓关键点之间保持刚性连接（rigidity assumption between end-effector and grasped keypoints）。论文指出此假设仅在约 0.1 秒的短时间窗口内有效，之后须由视觉重跟踪更新关键点位置。对于柔性物体或接触复杂的场景，该假设会引入较大误差。The forward model assumes a rigidity assumption between end-effector and grasped keypoints. The paper notes that this assumption holds only within a short window of about 0.1 second, after which keypoint positions must be refreshed by visual re-tracking. For deformable objects or contact-rich scenes it introduces considerable error.
点跟踪在遮挡条件下易失败Point tracking fails readily under occlusion
系统依赖持续的关键点视觉跟踪。操作过程中"heavy intermittent occlusions"（频繁的间歇性遮挡）是目前最大的单一失败来源（见图 4 错误分解），限制了在杂乱或自遮挡场景中的可靠性。The system relies on continuous visual tracking of the keypoints. The "heavy intermittent occlusions" that arise during manipulation are currently the single largest source of failure (see the error breakdown in Figure 4), which limits reliability in cluttered or self-occluding scenes.
阶段序列固定，难以动态重规划The stage sequence is fixed, making dynamic replanning difficult
当前框架"assumes a fixed sequence of stages for each task"。若需在任务执行中途以不同阶段顺序重新规划，则须以高频重新运行关键点提议和 VLM 推理，带来显著的计算瓶颈，实时性无法保证。The current framework "assumes a fixed sequence of stages for each task". Replanning mid-execution with a different ordering of stages would require re-running keypoint proposal and VLM inference at high frequency, a significant computational bottleneck that rules out real-time operation.
