RISE: Self-Improving Robot Policy with Compositional World Model

01 动机 MotivationMotivation

Vision-Language-Action（VLA）模型在接触密集型操作任务中表现脆弱，而在物理世界中直接跑 on-policy RL 受制于硬件成本、安全风险和环境重置的高昂代价。如何在无需大量真实物理交互的情况下，持续提升机器人 policy 的鲁棒性？Vision-Language-Action (VLA) models are brittle on contact-rich manipulation tasks, while running on-policy RL directly in the physical world is constrained by hardware cost, safety risk, and the high price of environment resets. How can the robustness of a robot policy be continually improved without large amounts of real physical interaction?

"While reinforcement learning (RL) offers a principled path to robustness, on-policy RL in the physical world is constrained by safety risk, hardware cost, and environment reset."

RISE teaser figure — **Figure 1：RISE 框架概览。** (a) 传统物理世界 RL 因硬件成本高、交互缓慢、需手动重置而受限； (b) RISE 将学习环境转移到 *Compositional World Model*——动态模型预测多视角未来，价值模型评估想象状态，生成 advantage 信号驱动 policy 更新，无需额外物理交互。**Figure 1: Overview of the RISE framework.** (a) Conventional physical-world RL is limited by high hardware cost, slow interaction, and the need for manual resets; (b) RISE moves the learning environment into a *Compositional World Model* — the dynamics model predicts multi-view futures, the value model evaluates imagined states, and together they generate an advantage signal that drives policy updates without additional physical interaction.

+35%Dynamic Brick Sorting
绝对成功率提升absolute success-rate gain

+45%Backpack Packing
绝对成功率提升absolute success-rate gain

+35%Box Closing
绝对成功率提升absolute success-rate gain

95%Box Closing
RISE 最高成功率RISE peak success rate

RISE task suite — **Figure 2：三项真实世界评测任务。** 左：桌面双臂操作平台全景；右：各任务操作细节。 *Dynamic Brick Sorting*——在移动传送带上精确拾取彩色砖块并放入目标区域， *Backpack Packing*——将多个物体装入背包， *Box Closing*——关闭具有柔性铰链的箱盖。三项任务均要求高动态性、精巧度与精度。**Figure 2: Three real-world evaluation tasks.** Left: overview of the tabletop bi-manual manipulation platform; right: manipulation details of each task. *Dynamic Brick Sorting* — precisely picking colored bricks off a moving conveyor belt and placing them into a target area; *Backpack Packing* — packing several objects into a backpack; *Box Closing* — closing a box lid with a flexible hinge. All three tasks demand high dynamics, dexterity, and precision.

02 方法 MethodMethod

RISE 的核心是 Compositional World Model，由两个专用模块组成： Dynamics Model（基于 Genie Envisioner）预测候选动作的多视角未来观测； Value Model（从 π₀.₅ VLA 初始化）评估想象状态并输出 advantage。两者联合驱动一个两阶段自我提升循环，全程无需物理 RL 环境。At the core of RISE is the Compositional World Model, made up of two dedicated modules: the Dynamics Model (based on Genie Envisioner) predicts multi-view future observations for candidate actions; the Value Model (initialized from the π₀.₅ VLA) evaluates imagined states and outputs an advantage. Together they drive a two-stage self-improvement loop that requires no physical RL environment.

Compositional World Model workflow — **Figure 4：Compositional World Model 的训练与推理流程。** 上：各模块的训练方式（动态模型预训练 + 微调；价值模型双目标训练）；下：推理时，动态模型生成多条想象轨迹，价值模型对每条轨迹打分，计算得到 advantage，供 policy 进行 advantage-conditioned 训练。**Figure 4: Training and inference pipeline of the Compositional World Model.** Top: how each module is trained (dynamics model pre-training + fine-tuning; dual-objective training of the value model); bottom: at inference the dynamics model generates multiple imagined trajectories, the value model scores each trajectory, and the resulting advantage feeds advantage-conditioned training of the policy.

Dynamics Model — 多视角未来预测Dynamics Model — Multi-View Future Prediction

以 Genie Envisioner 为基础，接收多视角 RGB 历史帧（history window N=4）与候选 action chunk（horizon H=50），预测未来多视角观测序列。关键创新：Task-Centric Batching——训练时「在同一场景下优先追求动作多样性，而非场景多样性」，显著提升动作可控性（EPE 从 1.05 降至 0.54）。选用 Genie Envisioner 而非 Cosmos 的原因：合成 25 帧多视角观测仅需 <2 秒，较 Cosmos 提速约 300×。Built on Genie Envisioner, it takes multi-view RGB history frames (history window N=4) and a candidate action chunk (horizon H=50), and predicts the future multi-view observation sequence. The key innovation is Task-Centric Batching — during training it "prioritizes action diversity within the same scene over scene diversity", markedly improving action controllability (EPE drops from 1.05 to 0.54). Genie Envisioner was chosen over Cosmos because synthesizing 25 multi-view frames takes only <2 seconds, about 300× faster than Cosmos.

Value Model — 双目标评估Value Model — Dual-Objective Evaluation

从预训练 π₀.₅ VLA 骨干初始化，采用双重训练目标：
① Progress regression：将时间步映射到粗粒度任务进度（0→1），提供密集监督；
② Temporal-Difference (TD) learning：区分成功与失败状态，使用演示数据与失败 rollout 数据联合训练，使评估对关键失败时刻敏感。 Advantage（公式 2）定义为「每条想象轨迹中各未来观测价值与初始观测价值之差的平均累计改善」，离散化为 10 个均匀 bins。Initialized from a pre-trained π₀.₅ VLA backbone, it adopts two training objectives:
① Progress regression: mapping the timestep to coarse-grained task progress (0→1), providing dense supervision;
② Temporal-Difference (TD) learning: distinguishing successful from failed states, trained jointly on demonstration data and failed rollout data so that the evaluation is sensitive to critical failure moments. The advantage (Equation 2) is defined as "the average cumulative improvement of the value of each future observation over that of the initial observation along an imagined trajectory", discretized into 10 uniform bins.

RISE self-improving loop — **Figure 5：RISE 自我提升循环（两阶段）。** *Stage 1 — Policy Warm-up*：在离线数据上微调 π₀.₅，以 advantage 标签为条件进行训练；专家演示数据赋予最优 advantage（值为 1），rollout 数据由学习到的价值模型标注 advantage。 *Stage 2 — 自我提升循环*：Rollout 阶段利用动态模型生成想象轨迹并由价值模型评分； Training 阶段以（状态, 动作, advantage）三元组更新 policy，混合离线与在线数据（比例 0.6 离线最优）防止灾难性遗忘，约迭代 10k 步。**Figure 5: The RISE self-improvement loop (two stages).** *Stage 1 — Policy Warm-up*: fine-tuning π₀.₅ on offline data, conditioned on advantage labels; expert demonstration data is assigned the optimal advantage (a value of 1), while rollout data is labeled with advantages by the learned value model. *Stage 2 — Self-improvement loop*: in the rollout phase the dynamics model generates imagined trajectories that the value model scores; in the training phase the policy is updated with (state, action, advantage) triples, mixing offline and online data (a ratio of 0.6 offline is optimal) to prevent catastrophic forgetting, for roughly 10k iterations.

计算开销Computational Cost

动态模型预训练：16 × NVIDIA H100，约 7 天；微调：8 × H100，约 3 天Dynamics model pre-training: 16 × NVIDIA H100, about 7 days; fine-tuning: 8 × H100, about 3 days
价值模型训练：8 × GPU，约 1 天Value model training: 8 × GPU, about 1 day
推理时零额外计算开销（"Zero computational overhead at inference"）Zero additional compute overhead at inference ("Zero computational overhead at inference")
观测分辨率：192×256 per view；动作空间：14 维（双臂各 7-DoF + 夹爪）Observation resolution: 192×256 per view; action space: 14 dimensions (7-DoF per arm + gripper)

03 实验 ExperimentsExperiments

在三项真实世界双臂操作任务上评测，报告成功率（%）与得分（满分 10），与多个基线对比：π₀.₅ 直接部署、π₀.₅+PPO、π₀.₅+DSRL、π₀.₅+DAgger 及 RECAP。Evaluated on three real-world bi-manual manipulation tasks, reporting success rate (%) and score (out of 10), compared against several baselines: π₀.₅ deployed directly, π₀.₅+PPO, π₀.₅+DSRL, π₀.₅+DAgger and RECAP.

主要结果（Table I）Main Results (Table I)

方法Method	Dynamic Brick Sorting（成功率）Dynamic Brick Sorting (success rate)	Backpack Packing（成功率）Backpack Packing (success rate)	Box Closing（成功率）Box Closing (success rate)
π₀.₅ (baseline)	35%	35%	50%
π₀.₅ + PPO	10%	10%	35%
π₀.₅ + DSRL	10%	10%	10%
π₀.₅ + DAgger	15%	30%	50%
RECAP (prior best)	50%	40%	60%
RISE（本文）RISE (ours)	85%（+35%）85% (+35%)	85%（+45%）85% (+45%)	95%（+35%）95% (+35%)

得分（满分 10）：Sorting 9.78 / Packing 9.50 / Closing 9.88（均为 RISE 最优）。 π₀.₅+PPO 表现出严重不稳定性，甚至低于直接部署基线。Scores (out of 10): Sorting 9.78 / Packing 9.50 / Closing 9.88 (RISE best on all). π₀.₅+PPO exhibits severe instability, falling even below the directly deployed baseline.

Learning dynamics comparison — **Figure 8：与 RL 替代方案的学习曲线对比。** RISE 与 RECAP 及 DSRL 相比，以更快的学习速度获得更高成功率；竞争方法即使经过充分训练也无法达到 RISE 的水平。**Figure 8: Learning-curve comparison against alternative RL approaches.** Compared with RECAP and DSRL, RISE attains a higher success rate with faster learning; the competing methods cannot reach RISE's level even after ample training.

消融实验Ablation Studies

离线数据比例（Table II）：混合比例 0.6（60% 离线）性能最优；极端比例（0.1 → 5% 成功率；0.9 → 30% 成功率）显著退化，说明「balanced offline data is crucial for complex generalization」。Offline data ratio (Table II): a mixing ratio of 0.6 (60% offline) performs best; extreme ratios (0.1 → 5% success rate; 0.9 → 30% success rate) degrade markedly, showing that "balanced offline data is crucial for complex generalization".
在线信号来源（Table III）：仅用在线动作 → 40%；同时使用在线动作+状态 → 70%；不用任何在线信号 → 35%。Source of the online signal (Table III): online actions only → 40%; online actions plus states → 70%; no online signal at all → 35%.
模块消融（Table IV，Sorting 任务）：无预训练 → 15%；无 Task-Centric Batching → 40%；无 Progress Loss → 50%；无 TD Learning → 35%；完整 RISE → 70%。Module ablation (Table IV, Sorting task): without pre-training → 15%; without Task-Centric Batching → 40%; without Progress Loss → 50%; without TD Learning → 35%; full RISE → 70%.
动态模型质量（Table V，EPE↓）： Cosmos 1.21 → Genie Envisioner 1.05 → RISE（无 task-centric）0.68 → RISE 完整 0.54。Dynamics model quality (Table V, EPE↓): Cosmos 1.21 → Genie Envisioner 1.05 → RISE (without task-centric) 0.68 → full RISE 0.54.

04 局限性 LimitationsLimitations

说明：以下三点由作者在论文中明确陈述（stated by the authors）。Note: The three points below are explicitly stated by the authors in the paper.

想象与现实的差距（Imagination-Reality Gap）Imagination-Reality Gap

动态模型在罕见或欠代表性场景下「仍可能产生物理上不合理的转换（physically implausible transitions）」，导致 policy 在真实部署时面临分布偏移风险。提高动态模型的真实性和覆盖度是未来的重要方向。In rare or under-represented scenarios the dynamics model "may still produce physically implausible transitions", exposing the policy to distribution-shift risk at real deployment. Improving the fidelity and coverage of the dynamics model is an important future direction.

仍需一定量真实数据（Real-World Data Dependency）Real-World Data Dependency

「A non-trivial amount of real-world data remains essential to anchor the learning procedure」。纯合成数据无法完全替代真实演示，收集成本依然存在，大规模泛化部署仍需解决真实数据的可扩展性问题。"A non-trivial amount of real-world data remains essential to anchor the learning procedure." Purely synthetic data cannot fully replace real demonstrations, the collection cost remains, and large-scale generalized deployment still has to solve the scalability of real-world data.

计算成本转移而非消除（Computational Bottleneck）Computational Bottleneck

「RISE shifts the primary bottleneck from physical interaction to computation」。动态模型预训练耗资 16×H100 约 7 天，微调 8×H100 约 3 天，前期计算投入门槛较高，对资源有限的研究团队构成挑战。"RISE shifts the primary bottleneck from physical interaction to computation." Pre-training the dynamics model costs about 7 days on 16×H100 and fine-tuning about 3 days on 8×H100; the upfront compute barrier is high and poses a challenge for research teams with limited resources.