GRAPE: Generalizing Robot Policy via Preference Alignment

01 动机 · MotivationMotivation

VLA 模型经过 supervised fine-tuning（SFT）后，在控制环境下表现出色，但在新物体、新语义、新场景等 out-of-distribution（OOD）条件下泛化能力严重不足。根本原因在于：SFT 仅从成功示范中做行为克隆，无法建立对任务目标和潜在失败模式的整体理解；同时，数据集通常收集自不同专家，隐含不同价值取向（完成率、安全性、效率），简单模仿反而造成目标混乱。After supervised fine-tuning (SFT), VLA models perform well in controlled environments, but generalize poorly under out-of-distribution (OOD) conditions such as novel objects, novel semantics and novel scenes. The root cause: SFT only does behavior cloning on successful demonstrations and cannot build a holistic understanding of the task goal or of potential failure modes; meanwhile datasets are usually collected from different experts and implicitly carry different value preferences (success rate, safety, efficiency), so naive imitation instead confuses the objective.

"VLAs simply imitate actions from successful rollouts via behavior cloning while not developing a holistic understanding of the task goal or potential failure patterns."

GRAPE 与 SOTA VLA 模型在多种泛化任务上的雷达图对比 — **Figure 1.** GRAPE 与当前最优 VLA 模型（Octo-SFT、OpenVLA-SFT、OpenVLA-DPO）在真实机器人和仿真环境中多种泛化维度上的成功率对比雷达图。GRAPE 在领域内（In-domain）及主体、物理、语义等泛化类别上全面领先。**Figure 1.** Radar comparison of success rates between GRAPE and the strongest current VLA models (Octo-SFT, OpenVLA-SFT, OpenVLA-DPO) across several generalization dimensions, on real robots and in simulation. GRAPE leads across the board on in-domain tasks and on the subject, physical and semantic generalization categories.

+51.79%领域内任务成功率提升（vs. SFT baseline）In-domain task success rate gain (vs. SFT baseline)

+58.20%未见操作任务成功率提升Success rate gain on unseen manipulation tasks

−37.44%碰撞率下降（GRAPE-Safety）Collision rate reduction (GRAPE-Safety)

−11.15%平均步长减少（GRAPE-Efficiency）Average step length reduction (GRAPE-Efficiency)

现有强化学习（RL）方案（如 PPO）理论上可增强泛化能力，但存在三大瓶颈：Reinforcement learning (RL) approaches such as PPO could in principle strengthen generalization, but face three bottlenecks:

操作任务目标多样复杂，奖励难以解析定义；manipulation task objectives are diverse and complex, so rewards are hard to define analytically;
人工标注机器人操作偏好数据成本极高；manually annotating preference data for robot manipulation is prohibitively expensive;
数值梯度近似不精确，导致 RL 算法（如 PPO）容易崩溃。numerical gradient approximation is imprecise, which makes RL algorithms such as PPO prone to collapse.

GRAPE 通过轨迹级偏好对比绕过显式奖励建模，并引入自动化偏好生成流水线解决数据标注瓶颈。GRAPE bypasses explicit reward modeling through trajectory-level preference contrast, and introduces an automated preference generation pipeline that resolves the annotation bottleneck.

02 方法 · MethodMethod

GRAPE 框架由两个核心模块构成：轨迹级偏好优化（TPO）定义从轨迹对比中学习的训练目标；引导代价偏好生成（GCPG）利用 VLM 自动为任意对齐目标合成偏好排序数据。二者结合，通过在线迭代采样不断提升策略质量。The GRAPE framework comprises two core modules: Trajectory-wise Preference Optimization (TPO) defines a training objective that learns from trajectory comparison, while Guided-Cost Preference Generation (GCPG) uses a VLM to automatically synthesize preference rankings for an arbitrary alignment objective. Combined, they keep improving policy quality through iterative online sampling.

GRAPE 方法概览图 — **Figure 2.** GRAPE 整体框架。上半部分：VLM 将复杂操作任务分解为时序阶段，并为每个阶段提出关键空间点（keypoints）；根据用户指定的对齐目标（安全、效率、任务完成），提示 VLM 生成阶段代价函数。下半部分：离线/在线采样基础 VLA 的轨迹，通过多阶段代价、自我评分和任务成功指示器综合排序，形成偏好对后用 TPO 损失更新策略，迭代直至收敛。**Figure 2.** Overall GRAPE framework. Top: a VLM decomposes a complex manipulation task into temporal stages and proposes key spatial points (keypoints) for each stage; given the user-specified alignment objective (safety, efficiency, task completion), the VLM is prompted to generate stage cost functions. Bottom: trajectories of the base VLA are sampled offline/online and ranked jointly by multi-stage costs, self-scoring and a task success indicator; the resulting preference pairs update the policy through the TPO loss, iterating until convergence.

轨迹级偏好优化（Trajectory-wise Preference Optimization, TPO）Trajectory-wise Preference Optimization (TPO)

传统 DPO 在 step 级别做偏好优化，忽视了轨迹的全局语义。GRAPE 将 KL-正则化 RL 目标（Eq. 2）通过 Bradley-Terry 模型重参数化，推导出轨迹级 TPO 损失：Conventional DPO optimizes preference at the step level and overlooks the global semantics of a trajectory. GRAPE reparameterizes the KL-regularized RL objective (Eq. 2) through the Bradley-Terry model and derives a trajectory-level TPO loss:

L_TPO = −E_{(ζ_w,ζ_l)} log σ( β · [log π_θ(ζ_w)/π_ref(ζ_w) − log π_θ(ζ_l)/π_ref(ζ_l)] )

进一步利用 MDP 性质将轨迹对数似然分解为逐步 state-action 对之和（Eq. 6），使 VLA 直接以 step-wise rollout 执行轨迹级对齐。相比 step-wise DPO（OpenVLA-DPO），TPO 在全局轨迹维度对策略施加约束，避免低层噪声干扰，并同时从成功与失败轨迹中提取信号，显著提升泛化能力。The MDP property is then used to decompose the trajectory log-likelihood into a sum over step-wise state-action pairs (Eq. 6), so that a VLA carries out trajectory-level alignment directly with step-wise rollouts. Compared with step-wise DPO (OpenVLA-DPO), TPO constrains the policy along the global trajectory dimension, avoids interference from low-level noise, and draws signal from successful and failed trajectories alike, markedly improving generalization.

引导代价偏好生成（Guided-Cost Preference Generation, GCPG）Guided-Cost Preference Generation (GCPG)

GCPG 自动构建偏好数据，消除人工标注需求：GCPG builds preference data automatically, removing the need for manual annotation:

多阶段时序关键点约束： VLM（如 GPT-4o）将任务分解为 S 个连续阶段 {ζ¹,…,ζˢ}，并在每个阶段提出关键空间点 {κ^Sᵢ}。针对对齐目标生成代价函数 C^Sᵢ，通过指数衰减聚合得到轨迹外部奖励 R_ext（Eq. 8），捕捉阶段间的因果依赖。Multi-stage temporal keypoint constraints: a VLM (e.g. GPT-4o) decomposes the task into S consecutive stages {ζ¹,…,ζˢ} and proposes key spatial points {κ^Sᵢ} within each stage. Cost functions C^Sᵢ are generated for the alignment objective and aggregated by exponential decay into the trajectory external reward R_ext (Eq. 8), capturing the causal dependency between stages.
综合 GCPG 奖励： R_GCPG = λ₁ R_self + λ₂ R_ext + λ₃ I_success。其中 R_self 为策略对轨迹的自我评分（log-likelihood），I_success 为任务成功二值指示器。三者共同确保所选轨迹在多维目标上均具优势。Composite GCPG reward: R_GCPG = λ₁ R_self + λ₂ R_ext + λ₃ I_success, where R_self is the policy's own score of the trajectory (log-likelihood) and I_success is a binary task-success indicator. Together the three terms ensure the selected trajectories are advantageous along every objective dimension.
迭代在线偏好优化： 每轮从当前策略在线采样轨迹，计算 R_GCPG 排序，取 top-m 与 bottom-m 组成偏好对，执行 TPO 更新后重复采样，迭代 K 轮直至收敛（Algorithm 1）。Iterative online preference optimization: each round samples trajectories online from the current policy, computes the R_GCPG ranking, takes the top-m and bottom-m as preference pairs, performs a TPO update and resamples, iterating for K rounds until convergence (Algorithm 1).

支持的对齐目标Supported alignment objectives

Task Completion：对齐到任务成功，最大化完成率。Task Completion: align to task success and maximize the completion rate.
Safety：最小化与物体的碰撞代价，学习避障路径。Safety: minimize the cost of collision with objects and learn obstacle-avoiding paths.
Efficiency：最小化路径代价，减少平均操作步长。Efficiency: minimize path cost and shorten the average manipulation step length.

通过调整 GCPG 代价函数的阈值参数即可切换对齐目标，无需重新设计训练流程。Switching the alignment objective only takes adjusting the threshold parameters of the GCPG cost function; the training pipeline needs no redesign.

实现细节Implementation details

Backbone: OpenVLA（7B），LoRA fine-tuning，AdamW optimizer。Backbone: OpenVLA (7B), LoRA fine-tuning, AdamW optimizer.
SFT 阶段：lr = 4×10⁻⁵，batch size = 16。SFT stage: lr = 4×10⁻⁵, batch size = 16.
偏好优化阶段：lr = 2×10⁻⁵，batch size = 16。Preference optimization stage: lr = 2×10⁻⁵, batch size = 16.
阶段分解 VLM：GPT-4o；关键点检测：DINOv2。Stage-decomposition VLM: GPT-4o; keypoint detection: DINOv2.

03 实验 · ExperimentsExperiments

在两个仿真平台（Simpler-Env、LIBERO）和真实机器人环境（30 个任务，300 次实验）上与 Octo-SFT、OpenVLA-SFT、OpenVLA-DPO 进行系统比较。评测维度涵盖领域内任务及五类分布外泛化：视觉（visual）、主体（subject）、动作（action）、语义（semantic）和语言接地（language grounding）泛化。Systematic comparison against Octo-SFT, OpenVLA-SFT and OpenVLA-DPO on two simulation platforms (Simpler-Env, LIBERO) and on a real robot (30 tasks, 300 trials). The evaluation covers in-domain tasks plus five out-of-distribution generalization types: visual, subject, action, semantic and language grounding generalization.

仿真环境结果Simulation results

仿真环境实验结果 — **Figure 3 & 4.** 左：Simpler-Env 上 GRAPE 与基线的成功率对比，涵盖领域内及主体、物理、语义三类泛化。右：LIBERO 上四类任务（Spatial/Object/Goal/Long）的平均成功率。GRAPE 在 Simpler-Env 上分别超越 Octo-SFT 和 OpenVLA-SFT **131.72%** 和 **46.10%**，在 LIBERO 上超越 **8.53%** 和 **7.36%**。**Figure 3 & 4.** Left: success rates of GRAPE versus baselines on Simpler-Env, covering in-domain tasks and the subject, physical and semantic generalization types. Right: average success rate over the four LIBERO task suites (Spatial/Object/Goal/Long). On Simpler-Env GRAPE surpasses Octo-SFT and OpenVLA-SFT by **131.72%** and **46.10%** respectively, and on LIBERO by **8.53%** and **7.36%**.

环境 / 对比方法Environment / compared method	Octo-SFT	OpenVLA-SFT	OpenVLA-DPO	GRAPE（本文）GRAPE (ours)
Simpler-Env 平均成功率 ↑Simpler-Env average success rate ↑	15.22%	29.33%	—	38.67%
LIBERO 平均成功率 ↑LIBERO average success rate ↑	73.9%	77.4%	—	83.1%
真实机器人总平均成功率 ↑Real robot, overall average success rate ↑	5.7%	32.3%	39.3%	50.3%
真实机器人领域内成功率 ↑Real robot, in-domain success rate ↑	20.0%	45.0%	50.0%	67.5%

真实机器人泛化结果Real-robot generalization results

真实机器人实验结果 — **Figure 5.** 真实机器人环境下，GRAPE 与基线在领域内及五类分布外泛化（视觉/主体/动作/语义/语言接地）上的成功率对比（共 30 个任务，300 次实验）。GRAPE 总平均成功率 50.3%，较 OpenVLA-DPO（39.3%）、OpenVLA-SFT（32.3%）和 Octo-SFT（5.7%）均有显著提升。**Figure 5.** Success rates of GRAPE versus baselines on a real robot, on in-domain tasks and on five out-of-distribution generalization types (visual/subject/action/semantic/language grounding), over 30 tasks and 300 trials in total. GRAPE reaches an overall average success rate of 50.3%, a clear gain over OpenVLA-DPO (39.3%), OpenVLA-SFT (32.3%) and Octo-SFT (5.7%).

多目标对齐分析（Table 2）Multi-objective alignment analysis (Table 2)

方法Method	Collision Rate ↓（真实）Collision Rate ↓ (real)	Step Length ↓（真实）Step Length ↓ (real)	Success Rate ↑（真实）Success Rate ↑ (real)	Collision Rate ↓（仿真）Collision Rate ↓ (sim)	Step Length ↓（仿真）Step Length ↓ (sim)	Success Rate ↑（仿真）Success Rate ↑ (sim)
OpenVLA-SFT	53.33	142.32	34.61	66.50	72.68	27.50
GRAPE-Safety	29.84	146.11	54.31	46.00	74.49	37.00
GRAPE-Efficiency	58.45	125.79	51.67	57.50	64.92	38.50
GRAPE-TC	38.60	131.66	58.46	59.50	70.24	42.50

注：GRAPE-Safety 将碰撞率降低 37.44%，GRAPE-Efficiency 将步长减少 11.15%，同时维持与 OpenVLA-SFT 相近的任务成功率——证明通过调整代价函数阈值即可灵活切换对齐目标。Note: GRAPE-Safety cuts the collision rate by 37.44% and GRAPE-Efficiency cuts step length by 11.15%, while keeping a task success rate comparable to OpenVLA-SFT — evidence that adjusting the cost-function threshold suffices to switch alignment objectives flexibly.

消融实验与迭代优化分析Ablations and iterative-optimization analysis

消融实验与迭代优化结果 — **Figure 6 & Table 1.** 左：GRAPE 在 Simpler-Env 上迭代偏好优化过程中各类泛化任务成功率的变化曲线（SFT → iter 1 → iter 2 → iter 3），领域内任务提升 17.5%，主体/物理/语义泛化分别提升 9.0%/15.0%/21.0%。右：奖励分量消融结果——去掉任意分量均导致性能下降，三者（R_self、R_ext、I_success）缺一不可；随机选择成功/失败对（Random w/ I_success）相比 GRAPE 全奖励排序性能显著更差。**Figure 6 & Table 1.** Left: success-rate curves of each generalization category over the iterative preference optimization of GRAPE on Simpler-Env (SFT → iter 1 → iter 2 → iter 3); in-domain tasks gain 17.5%, and subject/physical/semantic generalization gain 9.0%/15.0%/21.0% respectively. Right: ablation of the reward components — dropping any component degrades performance, so all three (R_self, R_ext, I_success) are indispensable; randomly picking success/failure pairs (Random w/ I_success) is markedly worse than GRAPE's full reward ranking.

消融结果表明：The ablations show that:

使用完整的 R_GCPG 评分排序，相比随机选取成功/失败对，性能显著更高；ranking with the full R_GCPG score performs markedly better than randomly picking success/failure pairs;
R_self 提升策略鲁棒性，倾向于选择生成概率更高的轨迹；R_self improves policy robustness by favoring trajectories with higher generation probability;
R_ext 引导策略学习特定行为（安全/效率）；R_ext steers the policy toward specific behaviors (safety/efficiency);
I_success 作为关键稀疏信号，驱动策略优先选择成功轨迹。I_success acts as the crucial sparse signal that drives the policy to prefer successful trajectories.

04 局限性 · LimitationsLimitations

Note： 原论文未包含专门的 "Limitations" 章节，以下各条均为从方法设计和实验条件中推断（inferred）的局限，并非作者明确陈述。Note: the original paper contains no dedicated "Limitations" section; every item below is inferred from the method design and the experimental setup, and is not stated explicitly by the authors.

依赖 VLM 进行阶段分解与代价生成Relies on a VLM for stage decomposition and cost generation

GRAPE 的 GCPG 模块需要调用外部 VLM（如 GPT-4o）来自动生成阶段分解和代价函数。这引入了对强大商业 API 的依赖，增加了推理成本，且代价函数质量受限于 VLM 本身的推理能力和 prompt 设计，在极端任务场景下可能失效。GRAPE's GCPG module has to call an external VLM (e.g. GPT-4o) to generate the stage decomposition and cost functions automatically. This introduces a dependence on a powerful commercial API, raises inference cost, and caps cost-function quality at the VLM's own reasoning ability and prompt design, which may fail in extreme task scenarios.

仅支持二进制偏好对（binary preference pair）Only binary preference pairs are supported

论文明确指出仅讨论"binary case where only one chosen/rejected trajectory is present"，以控制训练成本。扩展到多轨迹排序（list-wise preference）可能进一步提升性能，但会显著增加计算开销。The paper explicitly states that it discusses only the "binary case where only one chosen/rejected trajectory is present", so as to control training cost. Extending to multi-trajectory ranking (list-wise preference) could improve performance further, but would substantially increase computational overhead.

迭代在线采样的计算代价较高Iterative online sampling is computationally expensive

GRAPE 需要在每轮迭代中在线采样大量轨迹、分解阶段、计算代价并重新训练，整体训练成本高于直接 SFT。论文在 Implementation Details 中指出训练细节，但未提供具体的训练时间或 GPU 小时对比。Every GRAPE iteration must sample a large number of trajectories online, decompose stages, compute costs and retrain, so overall training cost exceeds plain SFT. The paper reports training details under Implementation Details but gives no concrete training-time or GPU-hour comparison.

安全与效率目标存在权衡Safety and efficiency objectives trade off against each other

Table 2 数据显示，GRAPE-Safety 在降低碰撞率的同时步长略有增加（146.11 vs 142.32），GRAPE-Efficiency 减少步长但碰撞率略升（58.45 vs 53.33）。两种对齐目标之间存在一定的 trade-off，同时优化多个目标的方法尚未被探索。The Table 2 data show that GRAPE-Safety lowers the collision rate while step length rises slightly (146.11 vs 142.32), and GRAPE-Efficiency shortens steps but sees the collision rate creep up (58.45 vs 53.33). A trade-off exists between the two alignment objectives, and methods optimizing several objectives at once remain unexplored.

评测局限于固定任务集合Evaluation limited to a fixed task set

真实机器人实验共 30 个任务、300 次实验，仿真实验在 Simpler-Env 和 LIBERO 两个标准 benchmark 上进行。这些任务集合相对固定，方法在更大规模、更开放世界的场景（如 Open X-Embodiment 全分布）中的表现尚不明确。The real-robot experiments cover 30 tasks and 300 trials in total, and the simulation experiments run on the two standard benchmarks Simpler-Env and LIBERO. These task sets are relatively fixed, so the method's behavior in larger-scale, more open-world settings (such as the full Open X-Embodiment distribution) remains unclear.