TGRPO: Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimization
无需 Critic 网络的 VLA 在线强化学习微调框架An online RL fine-tuning framework for VLA models that needs no Critic network
Zengjue Chen, Runliang Niu, He Kong, Qi Wang, Qianli Xing, Zipei Fan · 吉林大学人工智能学院Zengjue Chen, Runliang Niu, He Kong, Qi Wang, Qianli Xing, Zipei Fan · College of Artificial Intelligence, Jilin University
TGRPO 针对 VLA 模型依赖成功演示数据、无法从失败中自我学习的根本缺陷，提出了一种基于轨迹分组的在线 RL 微调框架：以 LLM 自动生成多阶段密集奖励取代稀疏二值反馈，再通过步骤级与轨迹级双层优势估计融合来降低策略梯度方差，在 LIBERO 四项基准上实现平均 80.7% 成功率，比 SFT 提升 4.2%。TGRPO targets a fundamental defect of VLA models — their reliance on successful demonstration data and their inability to learn from failure — and proposes an online RL fine-tuning framework based on trajectory grouping: an LLM automatically generates multi-stage dense rewards to replace sparse binary feedback, and a fusion of step-level and trajectory-level advantage estimation reduces policy-gradient variance, reaching an average success rate of 80.7% on the four LIBERO benchmarks, 4.2% above SFT.
VLA 模型的 SFT 范式将机器人局限于"动作记忆"，无法自主探索与自我修正。稀疏的二值奖励信号则让在线 RL 训练极为困难——这正是 TGRPO 要解决的两大核心矛盾。The SFT paradigm of VLA models confines the robot to "action memorization", leaving it unable to explore autonomously or to self-correct. Sparse binary reward signals further make online RL training extremely difficult — these are exactly the two core contradictions TGRPO sets out to resolve.
"VLA models trained solely on human-provided successful demonstrations … lacks the ability to learn from failures, restricting autonomous exploration and self-correction capabilities. Additionally, reward signals in real-world robotic tasks are often highly sparse, frequently reduced to binary success/failure feedback."
图 1：左侧对比 SFT 的"动作记忆"与 TGRPO 的策略优化路径；右侧展示 TGRPO 在 LIBERO 四个子集的成功率，明显高于所有基线。注意 LIBERO-Long（长时序任务）相比 SFT 提升幅度最大（+8.1%）。Figure 1: on the left, the "action memorization" of SFT is contrasted with the policy-optimization path of TGRPO; on the right, the success rates of TGRPO on the four LIBERO suites are shown, clearly higher than all baselines. Note that LIBERO-Long (long-horizon tasks) gains the most over SFT (+8.1%).
80.7%LIBERO 平均成功率（TGRPO）LIBERO average success rate (TGRPO)
+4.2%vs. SFT 基线vs. SFT baseline
+8.1%LIBERO-Long vs. SFT
4并行环境数（N=4 最优）Parallel environments (N=4 is optimal)
背景：GRPO 在 VLA 上的挑战Background: challenges of GRPO for VLA
Group Relative Policy Optimization (GRPO) 通过在组内归一化奖励来估计优势，无需额外的 Critic 网络，已在 LLM 数学推理中展现出色效率。然而直接迁移到机器人操作面临两大障碍：①机器人任务奖励极稀疏，组内方差过大导致梯度估计不稳定；②原版 GRPO 以单步 token 为粒度，与轨迹级别的机器人任务不匹配。TGRPO 通过多阶段密集奖励设计与双层分组策略解决这两点。Group Relative Policy Optimization (GRPO) estimates advantages by normalizing rewards within a group, needs no extra Critic network, and has shown remarkable efficiency in LLM mathematical reasoning. Transferring it directly to robotic manipulation, however, faces two obstacles: (i) rewards in robotic tasks are extremely sparse, and the excessive intra-group variance makes gradient estimation unstable; (ii) the original GRPO works at the granularity of a single token step, which does not match the trajectory-level nature of robotic tasks. TGRPO addresses both points with a multi-stage dense reward design and a two-level grouping strategy.
02方法Method
TGRPO 在相同初始状态下采样多条轨迹，以 LLM 分解任务并生成多阶段密集奖励，再同时在步骤级和轨迹级两个粒度上估计优势并加权融合，最终以 PPO 风格的 clipped surrogate loss 更新策略——全程无需 value network。TGRPO samples multiple trajectories from the same initial state, uses an LLM to decompose the task and generate multi-stage dense rewards, then estimates advantages at both the step level and the trajectory level and fuses them with weights, finally updating the policy with a PPO-style clipped surrogate loss — all without any value network.
图 2：TGRPO 整体流程。① 在 N 个并行环境中采样轨迹组；② LLM（Claude 3.7 Sonnet）将任务分解为 K 个子阶段并为每阶段定义奖励函数；③ 对同一时间步的步骤奖励做组内归一化（步骤级优势 Ai,t），同时对轨迹累计奖励做归一化（轨迹级优势 Ai）；④ 融合双层优势后以 KL 约束的 clipped loss 更新 OpenVLA（LoRA）策略。Figure 2: overall pipeline of TGRPO. (i) trajectory groups are sampled in N parallel environments; (ii) an LLM (Claude 3.7 Sonnet) decomposes the task into K sub-stages and defines a reward function for each stage; (iii) the step rewards at the same timestep are normalized within the group (step-level advantage Ai,t), while the cumulative trajectory rewards are normalized as well (trajectory-level advantage Ai); (iv) after fusing the two levels of advantage, the OpenVLA (LoRA) policy is updated with a KL-constrained clipped loss.
针对稀疏奖励问题，作者借助 LLM 将每个任务分解为 K 个子阶段，并为每阶段定义基于物体位姿与末端执行器位姿的奖励函数：To address the sparse-reward problem, the authors use an LLM to decompose each task into K sub-stages and define, for every stage, a reward function based on the object pose and the end-effector pose:
Rt = f₁(Pobject(t), Pkpose) + f₂(Pkpose, st)
其中 f₁ 根据任务相关物体与目标位姿的距离给分，f₂ 根据末端执行器与参考位姿的距离（来自成功演示数据）给出密集引导信号。这一设计将二值成功/失败信号转变为连续、分阶段的稠密反馈，大幅降低了 RL 训练难度。Here f₁ scores the distance between the task-relevant object and the target pose, while f₂ gives a dense guidance signal from the distance between the end-effector and a reference pose (taken from successful demonstration data). This design turns the binary success/failure signal into continuous, stage-wise dense feedback, greatly lowering the difficulty of RL training.
图 3：以长时序任务"put alphabet soup and tomato sauce in basket"为例，LLM 将其分解为多个子阶段（抓取汤罐、移动到篮子、放入、重复另一物体等），并为每步分配对应的奖励计算方式，展示了密集奖励生成的实际效果。Figure 3: taking the long-horizon task "put alphabet soup and tomato sauce in basket" as an example, the LLM decomposes it into several sub-stages (grasp the soup can, move to the basket, place it in, repeat for the other object, and so on) and assigns a corresponding reward computation to each step, illustrating dense reward generation in practice.
双层优势估计与融合Two-level advantage estimation and fusion
TGRPO 同时在两个粒度上计算优势：TGRPO computes advantages at two granularities at once:
步骤级优势（Step-level Advantage）：对组内同一时间步 t 的所有轨迹的步骤奖励 ri,t 做均值/方差归一化，捕捉细粒度的时间步偏好。Step-level Advantage: the step rewards ri,t of all trajectories in the group at the same timestep t are normalized by mean and variance, capturing fine-grained per-timestep preferences.
轨迹级优势（Trajectory-level Advantage）：对轨迹累计奖励做组内归一化，捕捉宏观的整体轨迹质量偏好。Trajectory-level Advantage: the cumulative trajectory rewards are normalized within the group, capturing macroscopic preferences over overall trajectory quality.
两者线性融合为最终优势：Advi,t = α₁Ai,t + α₂Ai，消融实验确定最优权重 α₁=0.3，α₂=0.7。最终使用 PPO 风格的 clipped surrogate loss 并加 KL 正则项约束策略漂移，无需额外 Critic 网络。The two are linearly fused into the final advantage: Advi,t = α₁Ai,t + α₂Ai, and the ablation study identifies the optimal weights α₁=0.3 and α₂=0.7. The policy is finally updated with a PPO-style clipped surrogate loss plus a KL regularizer constraining policy drift, with no extra Critic network.
图 4：α₁-α₂ 权重组合对 LIBERO-Goal 成功率的影响热力图。α₁=0.3、α₂=0.7 达到最优 81.0%；右侧展示分组大小 N 对训练稳定性与效率的权衡，N=4 为最佳平衡点。Figure 4: heatmap of how α₁-α₂ weight combinations affect the success rate on LIBERO-Goal. α₁=0.3 and α₂=0.7 reach the best 81.0%; on the right, the trade-off between group size N and training stability/efficiency is shown, with N=4 the best balance point.
03实验Experiments
在 LIBERO 基准的四个子集（各含 10 项任务）上评估，每任务 50 个测试 episode；基座模型为 OpenVLA（LoRA 微调，AdamW lr=1×10⁻⁵），4 个并行环境；基线包括 Octo、SFT、DPO、GRAPE。Evaluation is carried out on the four suites of the LIBERO benchmark (10 tasks each), with 50 test episodes per task; the base model is OpenVLA (LoRA fine-tuning, AdamW lr=1×10⁻⁵) with 4 parallel environments; the baselines include Octo, SFT, DPO and GRAPE.
主要结果Main results
测试集Test suite
Octo
SFT
DPO
GRAPE
TGRPO（本文）TGRPO (ours)
LIBERO-Spatial
77.6%
84.7%
—
88.5%
90.4%
LIBERO-Object
84.9%
88.4%
—
92.1%
92.2%
LIBERO-Goal
82.9%
79.2%
—
83.1%
81.0%
LIBERO-Long
50.3%
51.1%
—
57.2%
59.2%
平均Average
73.9%
75.9%
—
80.2%
80.7%
TGRPO 在 Spatial、Object、Long 三个子集上超越所有基线；在 Goal 子集上（81.0%）略低于 GRAPE（83.1%），低于 Octo（82.9%）。作者注：LIBERO-Goal 任务的多样目标条件使 LLM 生成奖励时有一定噪声。TGRPO surpasses all baselines on the Spatial, Object and Long suites; on the Goal suite (81.0%) it is slightly below GRAPE (83.1%) and below Octo (82.9%). Authors’ note: the diverse goal conditions of the LIBERO-Goal tasks make the LLM-generated rewards somewhat noisy.
图 5：LIBERO 四个子集上各方法成功率的柱状图对比。TGRPO（蓝色）在 Spatial 和 Long 子集上优势明显，Long 任务相比 SFT 提升 8.1 个百分点，体现了 RL 在长时序任务上的优越性。Figure 5: bar-chart comparison of the success rates of every method on the four LIBERO suites. TGRPO (blue) holds a clear advantage on the Spatial and Long suites, gaining 8.1 percentage points over SFT on Long tasks, which reflects the superiority of RL on long-horizon tasks.
消融实验（LIBERO-Object）Ablation study (LIBERO-Object)
方法Method
Task0
Task1
Task2
Task3
Task4
Task5
Task6
Task7
Task8
Task9
平均Average
SFT
86
76
90
74
92
92
98
92
92
92
88.4%
w/o Trajectory-level
88
56
86
60
92
82
92
92
92
60
80.2%
w/o Step-level
78
78
98
58
94
82
96
96
92
96
86.8%
TGRPO（完整）TGRPO (full)
88
82
98
76
98
94
98
98
94
96
92.2%
去除轨迹级优势（→80.2%）和去除步骤级优势（→86.8%）均显著低于完整方法（92.2%），证明两个层级的优势估计缺一不可。Task1 和 Task3 在去除轨迹级优势后下降尤为明显，说明宏观轨迹质量信号对部分任务至关重要。Removing the trajectory-level advantage (→80.2%) and removing the step-level advantage (→86.8%) both fall well below the full method (92.2%), proving that neither level of advantage estimation can be dropped. Task1 and Task3 degrade especially sharply once the trajectory-level advantage is removed, showing that the macroscopic trajectory-quality signal is critical for some tasks.
04局限性Limitations
说明：作者在论文中仅简短提及未来方向，未设专门的 Limitations 章节。以下第一条为作者明确陈述，第二、三条为从方法设计中归纳（inferred）。Note: the paper only briefly mentions future directions and has no dedicated Limitations section. The first item below is stated explicitly by the authors; the second and third are inferred from the method design (inferred).
仅在仿真环境中验证（作者明确陈述）Validated only in simulation (stated explicitly by the authors)
所有实验在 LIBERO 模拟器中进行，作者明确指出未来工作方向为"extend TGRPO to real-world and multi-task settings"。真实机器人的传感器噪声、接触动力学和状态估计误差对密集奖励计算的鲁棒性尚未评估。All experiments are run in the LIBERO simulator, and the authors explicitly point to "extend TGRPO to real-world and multi-task settings" as future work. The robustness of the dense reward computation to sensor noise, contact dynamics and state-estimation error on real robots has not been evaluated.
奖励函数依赖 LLM 且需要状态特权信息（inferred）The reward function depends on an LLM and requires privileged state information (inferred)
多阶段奖励的生成需要 LLM（Claude 3.7 Sonnet）对任务进行分解，并在运行时访问仿真器提供的精确物体位姿（Pobject(t)）和末端执行器状态。在无法获取完整状态观测的真实场景中，该奖励设计需要额外的感知模块支持，增加了部署复杂度。Generating the multi-stage rewards requires an LLM (Claude 3.7 Sonnet) to decompose the task, plus runtime access to the exact object pose (Pobject(t)) and end-effector state provided by the simulator. In real scenarios where full state observation is unavailable, this reward design needs extra perception modules, which increases deployment complexity.
每个实验仅对单一任务进行 RL 微调（"single task per experiment"），尚未验证 TGRPO 在多任务联合训练设置下的性能与稳定性。LIBERO-Goal 子集上略逊于 GRAPE 和 Octo 也暗示目标条件多样性下的泛化仍有改进空间。Each experiment fine-tunes with RL on a single task only ("single task per experiment"), and the performance and stability of TGRPO under joint multi-task training have not been verified. Being slightly behind GRAPE and Octo on the LIBERO-Goal suite also hints that generalization under diverse goal conditions still has room for improvement.
