SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
自参照策略优化：无需额外监督，让 VLA 在 200 步内从 48.9% 跃升至 99.2%Self-Referential Policy Optimization: lifting VLA from 48.9% to 99.2% within 200 steps, with no additional supervision
Senyu Fei, Siyin Wang, Li Ji, Ao Li, Shiduo Zhang, Liming Liu, Jinlong Hou, Jingjing Gong, Xianzhong Zhao, Xipeng Qiu
SRPO 提出一种全新的 VLA 强化学习后训练框架：以模型在 batch 内自身产生的成功轨迹作为"自参照"基准，
通过预训练视觉世界模型（V-JEPA 2）的潜在表征计算进度奖励，彻底摆脱对人工奖励设计或额外专家演示的依赖。
在 LIBERO 基准上仅需 200 步 RL 训练即实现 103% 相对提升，真实机器人实验也显著优于 SFT 基线。SRPO introduces a new reinforcement-learning post-training framework for VLA models: it takes the successful trajectories that the model itself produces within a batch as a "self-referential" baseline, and computes a progress reward from the latent representations of a pretrained visual world model (V-JEPA 2), fully removing any dependence on hand-crafted reward design or additional expert demonstrations. On the LIBERO benchmark it attains a 103% relative improvement with only 200 steps of RL training, and real-robot experiments also clearly outperform the SFT baseline.
当前 VLA 模型过度依赖专家演示，导致严重的 demonstration bias（演示偏差）——模型只学会复制演示中的轨迹，
无法探索更优策略。强化学习（RL）是克服这一限制的关键，但现有 VLA-RL 方法面临两大困境：
稀疏奖励（仅依赖二值成功信号，失败轨迹的信息完全浪费）
以及繁琐的人工奖励工程（需任务相关的密集奖励，难以泛化）。Current VLA models depend excessively on expert demonstrations, which causes severe demonstration bias: the model only learns to copy the trajectories contained in the demonstrations and cannot explore better policies. Reinforcement learning (RL) is the key to overcoming this limit, but existing VLA-RL methods face two major difficulties: reward sparsity (relying on a binary success signal alone, so the information in failed trajectories is entirely wasted) and cumbersome manual reward engineering (task-related dense rewards that are hard to generalize).
"Reinforcement learning (RL) is a vital post-training strategy to overcome these limits,
yet current VLA-RL methods...are crippled by severe reward sparsity."
图 1：三种 VLA-RL 范式对比。(a) 现有方法依赖稀疏的结果奖励（成功/失败二值），失败轨迹信息完全丢弃；
(b) 手工设计过程奖励，需任务专属工程；(c) SRPO 利用 batch 内成功轨迹作为自参照，
通过潜在世界表征为失败轨迹提供细粒度进度奖励，无需任何额外监督。Figure 1: comparison of three VLA-RL paradigms. (a) Existing methods rely on sparse outcome rewards (binary success/failure) and discard the information in failed trajectories entirely; (b) hand-designed process rewards require task-specific engineering; (c) SRPO uses in-batch successful trajectories as a self-reference and provides fine-grained progress rewards for failed trajectories through latent world representations, without any additional supervision.
99.2%LIBERO Online SRPO 平均成功率LIBERO Online SRPO average success rate
SRPO 框架由两个核心模块组成：World Progress Reward Modeling（世界进度奖励建模）
和 Self-Referential Policy Optimization（自参照策略优化）。
前者借助预训练视觉世界模型 V-JEPA 2 在潜在空间中衡量轨迹的行为进度，
后者以此进度信号替代稀疏结果奖励驱动策略更新。The SRPO framework consists of two core modules: World Progress Reward Modeling and Self-Referential Policy Optimization. The former uses the pretrained visual world model V-JEPA 2 to measure the behavioral progress of a trajectory in latent space; the latter uses that progress signal in place of the sparse outcome reward to drive policy updates.
图 2：SRPO 完整流程。① 用 V-JEPA 2 对 batch 内所有轨迹（成功+失败）编码为潜在表征；
② 用 DBSCAN 对成功轨迹的潜在表征聚类，得到多个"行为参照中心"；
③ 对失败轨迹计算其与最近参照中心的 L2 距离，经 Sigmoid 映射为 (0,1) 进度奖励；
④ 成功轨迹直接赋予 reward = 1.0；
⑤ 以轨迹级进度奖励 g 替代结果信号，输入 GRPO 目标函数更新策略。Figure 2: the complete SRPO pipeline. ① V-JEPA 2 encodes every trajectory in the batch (successful + failed) into latent representations; ② DBSCAN clusters the latent representations of the successful trajectories, yielding several "behavioral reference centers"; ③ for each failed trajectory, its L2 distance to the nearest reference center is computed and mapped through a Sigmoid into a (0,1) progress reward; ④ successful trajectories are directly assigned reward = 1.0; ⑤ the trajectory-level progress reward g replaces the outcome signal and is fed into the GRPO objective to update the policy.
World Progress Reward Modeling
对于 batch 中的每条轨迹，V-JEPA 2 将视频片段编码为紧凑的潜在向量序列。
成功轨迹通过 DBSCAN 聚类后形成若干"参照中心"，代表不同的成功行为模式。
失败轨迹 i 到最近参照中心的 L2 距离 di 经标准化后，
通过 Sigmoid 函数 φ 映射为进度奖励：For every trajectory in the batch, V-JEPA 2 encodes the video clip into a compact sequence of latent vectors. After DBSCAN clustering, the successful trajectories form a number of "reference centers" that represent different successful behavior modes. The L2 distance di from a failed trajectory i to its nearest reference center is standardized and then mapped into a progress reward through the Sigmoid function φ:
采用聚类而非单一参照轨迹的关键原因是：随着 RL 训练推进，成功策略愈发多样化，
单一参照会造成"后期性能收益显著衰减"，而多中心参照能持续覆盖探索空间。The key reason for adopting clustering rather than a single reference trajectory is that, as RL training proceeds, successful policies become increasingly diverse, so a single reference leads to "a significant decay of late-stage performance gains", whereas multi-center references keep covering the exploration space.
与像素级世界模型（如 Cosmos-Predict2）不同，V-JEPA 2 的潜在表征天然具备跨任务迁移能力，
"naturally capture transferable behavioral progress patterns"，无需任务专属微调。Unlike pixel-level world models (e.g. Cosmos-Predict2), the latent representations of V-JEPA 2 are inherently transferable across tasks and "naturally capture transferable behavioral progress patterns", requiring no task-specific fine-tuning.
Self-Referential Policy Optimization
策略更新沿用 GRPO（Group Relative Policy Optimization）框架，
将轨迹级进度奖励 g 替代原始结果信号，以标准化优势估计驱动梯度：Policy updates follow the GRPO (Group Relative Policy Optimization) framework, replacing the raw outcome signal with the trajectory-level progress reward g and driving the gradient with a standardized advantage estimate:
 = (g − μg) / σg
目标函数包含 clipped surrogate loss 与 KL 散度正则项，防止策略偏离 SFT 初始化过远。
"自参照"的精髓在于：参照基准完全来自模型自身的 in-batch 成功轨迹，
无需外部专家演示，随训练动态更新，天然契合当前策略的能力边界。The objective comprises a clipped surrogate loss and a KL-divergence regularizer, which prevents the policy from drifting too far from the SFT initialization. The essence of "self-reference" is that the reference baseline comes entirely from the model's own in-batch successful trajectories: no external expert demonstrations are needed, it is updated dynamically along with training, and it naturally matches the capability frontier of the current policy.
图 3：三种进度估计方式的对比（模拟环境 + 真实环境）。
SRPO（潜在表征）的进度曲线单调递增、平滑，
而像素级和 ImageBind 基线的曲线则出现振荡或停滞，无法有效引导策略学习。Figure 3: comparison of three ways of estimating progress (simulation + real world). The progress curve of SRPO (latent representations) rises monotonically and smoothly, whereas the curves of the pixel-level and ImageBind baselines oscillate or stall and cannot effectively guide policy learning.
03实验Experiments
实验在 LIBERO（4 个任务套件：Spatial / Object / Goal / Long，
每套件 10 个任务，每任务 1 条演示）和真实 X-ARM 7 机器人上进行（5 类操作任务）。
基线模型为 OpenVLA*-One（one-shot SFT，仅单条演示训练），评估指标为任务成功率。Experiments are conducted on LIBERO (4 task suites: Spatial / Object / Goal / Long, with 10 tasks per suite and 1 demonstration per task) and on a real X-ARM 7 robot (5 categories of manipulation task). The baseline model is OpenVLA*-One (one-shot SFT, trained on a single demonstration), and the evaluation metric is task success rate.
LIBERO 基准主结果（Table 1）Main results on the LIBERO benchmark (Table 1)
方法Method
Spatial
Object
Goal
Long
平均Average
OpenVLA*-One（One-shot SFT）
61.7
55.5
42.6
35.8
48.9
+ Offline SRPO
97.5
91.4
90.1
91.0
92.5
+ Online SRPO
100.0
99.4
98.6
98.8
99.2
Online SRPO 在 Spatial 套件上实现满分（100.0%），四套件平均达 99.2%，
较 SFT 基线提升 50.3 个百分点，代表"103% 的相对提升，无需任何额外监督"。Online SRPO reaches a perfect score on the Spatial suite (100.0%) and averages 99.2% over the four suites, an improvement of 50.3 percentage points over the SFT baseline, which represents "a 103% relative improvement without any additional supervision".
LIBERO-Plus 鲁棒性测试（Table 2）LIBERO-Plus robustness test (Table 2)
+ Online SRPO + 数据增强+ Online SRPO + data augmentation
82.1%（+51.4 pts）
LIBERO-Plus 涵盖相机位置、机器人初始化、语言描述、光照、背景、噪声、布局 7 个扰动维度，
代表"167% 相对提升"（相对 SFT 19.4%）。LIBERO-Plus covers 7 perturbation dimensions: camera position, robot initialization, language description, lighting, background, noise and layout; this represents "a 167% relative improvement" (relative to the SFT 19.4%).
进度奖励质量定量评估（Table 3）Quantitative evaluation of progress-reward quality (Table 3)
方法Method
SC (Spearman ↑)
Mono ↑
MMD ↑
JS ↑
SMD ↑
Pixel-level
0.125
0.498
0.274
0.548
2.100
ImageBind
0.957
0.837
0.356
0.408
18.111
SRPO（V-JEPA 2）
0.998
0.992
0.615
0.572
188.799
SRPO 在 Spearman 时序相关性（0.998）和单调性（0.992）上接近完美，
成功/失败轨迹分布分离度（SMD = 188.799）远超竞争方案，验证潜在世界模型表征的优越性。SRPO is close to perfect on Spearman temporal correlation (0.998) and monotonicity (0.992), and its separation between the distributions of successful and failed trajectories (SMD = 188.799) far exceeds the competing schemes, confirming the superiority of latent world-model representations.
图 8：真实 X-ARM 7 机器人上 5 类操作任务（水果放置、毛巾折叠、黑板擦拭、卡片选取等）的成功率对比。
SRPO（offline RL）相较 SFT 基线，π₀（扩散策略）平均提升 +66.8%，π₀-FAST（自回归策略）平均提升 +86.7%。Figure 8: comparison of success rates on 5 categories of manipulation task (fruit placing, towel folding, blackboard wiping, card picking, etc.) with a real X-ARM 7 robot. Relative to the SFT baseline, SRPO (offline RL) improves π₀ (a diffusion policy) by +66.8% on average and π₀-FAST (an autoregressive policy) by +86.7% on average.
消融实验Ablation studies
图 10：Object 套件消融实验。去掉"自参照机制"（改用固定外部专家参照）导致性能在次优结果处停滞，
训练步数需增加约 1.4×；去掉"聚类机制"（改用单条最近成功轨迹）在训练后期性能收益显著下降。
超参数分析显示 α=0.8（进度奖励权重）为最优，α=0 仅靠结果奖励不足以高效学习。Figure 10: ablation studies on the Object suite. Removing the "self-referential mechanism" (using a fixed external expert reference instead) makes performance stall at a suboptimal result and requires about 1.4× more training steps; removing the "clustering mechanism" (using the single nearest successful trajectory instead) markedly reduces performance gains in the later stage of training. The hyper-parameter analysis shows that α=0.8 (the progress-reward weight) is optimal, while α=0, which relies on the outcome reward alone, is insufficient for efficient learning.
训练效率Training efficiency
SRPO 在 LIBERO 四套件上达到收敛所需步数：Spatial 79 步、Object 59 步、Goal 103 步、Long 219 步。
相较 GRPO，SRPO 展现出"steeper efficiency slope...especially for long-horizon tasks"，
长时序任务收敛速度提升尤为明显。Steps required for SRPO to converge on the four LIBERO suites: Spatial 79, Object 59, Goal 103, Long 219. Compared with GRPO, SRPO exhibits a "steeper efficiency slope...especially for long-horizon tasks", the gain in convergence speed being particularly evident on long-horizon tasks.
04局限性Limitations
Note：以下局限性部分为论文作者明确陈述，部分为从设计出发的推断，已分别标注。Note: Some of the limitations below are explicitly stated by the paper's authors and others are inferences drawn from the design; each is labeled accordingly.
真实环境采用 Offline RL 而非 Online RL（作者明确陈述）The real-world setting uses offline RL rather than online RL (explicitly stated by the authors)
出于安全顾虑，真实机器人实验采用 Advantage-Weighted Regression（离线 RL 范式），
而非模拟环境中的完整 online RL。这意味着真实场景下无法像模拟中那样进行在线探索，
可能限制自参照机制在动态多样化真实任务中的充分发挥。Out of safety concerns, the real-robot experiments use Advantage-Weighted Regression (an offline RL paradigm) rather than the full online RL used in simulation. This means that online exploration as performed in simulation is impossible in the real setting, which may prevent the self-referential mechanism from being fully exploited on dynamic and diverse real-world tasks.
潜在世界模型依赖 V-JEPA 2 预训练质量（推断）The latent world model depends on the pretraining quality of V-JEPA 2 (inferred)
SRPO 的进度奖励质量高度依赖 V-JEPA 2 所提供的潜在表征。
若目标任务域与 V-JEPA 2 预训练数据差异较大（如工业场景、水下操作等），
潜在表征的迁移能力可能下降，进而影响进度奖励的可靠性。论文本身未对此进行系统性分析。The quality of the SRPO progress reward depends heavily on the latent representations provided by V-JEPA 2. If the target task domain differs substantially from the V-JEPA 2 pretraining data (industrial scenes, underwater manipulation and the like), the transferability of the latent representations may degrade and thereby affect the reliability of the progress reward. The paper itself gives no systematic analysis of this.
像素级世界模型方案存在场景一致性问题（作者明确陈述）Pixel-level world-model solutions suffer from scene-consistency problems (explicitly stated by the authors)
论文附录 E 通过 Cosmos-Predict2 像素级世界模型的实验展示了"generated videos suffer from
poor scene consistency"，证明像素级方案在多样任务下泛化困难，
这也是作者选择潜在表征而非像素预测的核心依据，但同时表明 SRPO 的方法论选择受限于此类问题的存在。Appendix E of the paper uses experiments with the Cosmos-Predict2 pixel-level world model to show that "generated videos suffer from poor scene consistency", demonstrating that pixel-level solutions generalize poorly across diverse tasks. This is also the core justification for the authors' choice of latent representations over pixel prediction, but it likewise indicates that the methodological choice of SRPO is constrained by the existence of such problems.
仅使用第三视角视觉观测（作者明确陈述）Only third-person visual observations are used (explicitly stated by the authors)
SRPO 在实验中"achieves SOTA using only third-person visual observations and language instructions"，
无需腕部图像、本体感知或 3D 数据。虽然这是一个优点，
但也意味着对于依赖精细末端执行器信息的高精度操作任务，该方案尚未完整验证。In its experiments SRPO "achieves SOTA using only third-person visual observations and language instructions", requiring no wrist images, proprioception or 3D data. Although this is an advantage, it also means that the solution has not been fully validated on high-precision manipulation tasks that depend on fine-grained end-effector information.
