本文提出 DreMa，一个将机器人世界模型视为"可学习数字孪生"的框架。系统结合 Gaussian Splatting 与物理仿真器，使机器人能够"想象"新颖的物体配置并预测动作后果，从而以极少的示范数据高效学习操控策略，最终在真实 Franka Emika Panda 机器人上实现每个任务变体仅需单次示范（one-shot）的策略学习。This paper presents DreMa, a framework that treats the robot world model as a "learnable digital twin". The system combines Gaussian Splatting with a physics simulator, enabling the robot to "imagine" novel object configurations and predict the consequences of its actions, so that manipulation policies can be learned efficiently from very few demonstrations, ultimately achieving one-shot policy learning with a single demonstration per task variation on a real Franka Emika Panda robot.
机器人操控策略学习面临一个根本性挑战：如何从极少量的示范中泛化到未见过的物体配置？现有世界模型往往在训练分布之外产生不真实的"幻觉"，无法提供可靠的数据增广。Robot manipulation policy learning faces a fundamental challenge: how can a policy generalize to unseen object configurations from a very small number of demonstrations? Existing world models often produce unrealistic "hallucinations" outside the training distribution and therefore cannot provide reliable data augmentation.
"We present DreMa, which builds a compositional manipulation world model, treated as a learnable digital twin of the environment, that explicitly models the environment composing object assets, a robot agent model, world dynamics, and manipulation tasks."
图1：DreMa 的"想象"流程概览。系统从真实环境图像出发，通过场景分解（scene decomposition）获取物体级 Gaussian Splatting 表示，再经由物理引擎进行等变变换（equivariant transformations），生成多样化的合成示范数据，最终用于训练操控策略。
Figure 1: Overview of DreMa's "imagination" pipeline. Starting from images of the real environment, the system obtains object-level Gaussian Splatting representations through scene decomposition, then applies equivariant transformations with a physics engine to generate diverse synthetic demonstrations, which are finally used to train the manipulation policy.
+9.1%单任务平均成功率提升（vs PerAct 基线，仿真）Average single-task success-rate gain (vs. the PerAct baseline, simulation)
+13.1%多任务平均成功率提升（vs PerAct 基线，仿真）Average multi-task success-rate gain (vs. the PerAct baseline, simulation)
62.9%真实机器人分布内成功率（DreMa，5个任务均值）In-distribution success rate on the real robot (DreMa, mean over 5 tasks)
1-shot真实机器人每任务变体所需示范数Demonstrations needed per task variation on the real robot
02方法Method
DreMa 将世界表示为四个组件的组合：K 个物体资产（Object Assets）、机器人智能体模型、世界动力学算子（Physics Engine）以及操控任务集合。通过在此组合表示上施加等变变换，系统可以低成本合成大量真实且多样化的示范。DreMa represents the world as a composition of four components: K object assets, a robot agent model, a world dynamics operator (physics engine) and a set of manipulation tasks. By applying equivariant transformations to this compositional representation, the system can synthesize large amounts of realistic and diverse demonstrations at low cost.
图2：DreMa 系统处理流程。（1）输入真实环境视频；（2）使用开放词汇跟踪（DEVA）进行场景分解，为每个物体生成分割掩码；（3）基于掩码为每个物体独立重建 2D Gaussian Splatting 表示；（4）将 Gaussian Splat 转换为网格（mesh）并集成进 PyBullet 物理仿真器；（5）在仿真中施加等变变换并渲染新颖视角，生成合成示范数据。
Figure 2: The DreMa processing pipeline. (1) A video of the real environment is taken as input; (2) scene decomposition is performed with an open-vocabulary tracker (DEVA), producing a segmentation mask for each object; (3) a separate 2D Gaussian Splatting representation is reconstructed for each object from the masks; (4) the Gaussian Splats are converted into meshes and integrated into the PyBullet physics simulator; (5) equivariant transformations are applied in simulation and novel views are rendered, yielding synthetic demonstrations.
物体级 3D 重建Object-level 3D reconstruction
系统使用带深度监督的 2D Gaussian Splatting 为每个物体独立建模。每个 Gaussian 表示为 g = (p, r, s, α, c)，分别编码位置、朝向、尺度、不透明度和颜色，支持高质量新视角合成。场景分解通过开放词汇跟踪器 DEVA 自动完成，仅需"object""table"等简单文本提示。The system models each object separately with depth-supervised 2D Gaussian Splatting. Each Gaussian is written as g = (p, r, s, α, c), encoding position, orientation, scale, opacity and color respectively, which supports high-quality novel-view synthesis. Scene decomposition is carried out automatically by the open-vocabulary tracker DEVA and needs only simple text prompts such as "object" and "table".
物理约束动力学Physics-constrained dynamics
Gaussian Splat 被转换为网格 M_k，集成至 PyBullet 物理仿真器。位置与姿态更新遵循牛顿力学：
μ_{t+1} = μ_t + Δμ_{1…K}，ρ_{t+1} = Δρ_{1…K} · ρ_t。物理引擎保证了变换后场景的物理合理性，避免了纯神经网络生成方法的"幻觉"问题。The Gaussian Splats are converted into meshes M_k and integrated into the PyBullet physics simulator. Position and pose updates follow Newtonian mechanics: μ_{t+1} = μ_t + Δμ_{1…K}, ρ_{t+1} = Δρ_{1…K} · ρ_t. The physics engine guarantees the physical plausibility of the transformed scene and avoids the "hallucination" problem of purely neural generative methods.
等变变换数据增广Equivariant-transformation data augmentation
针对操控任务的数据增广需要保持任务语义的等变性。DreMa 设计了三类变换：Data augmentation for manipulation tasks must preserve the equivariance of the task semantics. DreMa designs three kinds of transformation:
Roto-translation（旋转平移）：对整个场景（环境 + 所有物体）施加刚体变换，保持机器人与物体的相对关系不变。Roto-translation: a rigid-body transformation is applied to the whole scene (environment + all objects), keeping the relative relation between robot and objects unchanged.
Object rotation（物体旋转）：围绕末端执行器最终位置旋转目标物体，模拟物体初始朝向变化。Object rotation: the target object is rotated around the final end-effector position, simulating a change of its initial orientation.
Verification（合法性验证）：确保变换后的示范满足任务目标，使用距离阈值 τ = 0.015 m 过滤不合法增广。Verification: ensures that the transformed demonstration still satisfies the task goal, using a distance threshold τ = 0.015 m to filter out invalid augmentations.
图3：三种等变变换效果可视化。上行：示范起始状态；下行：示范目标状态。从左至右依次为：等变平移（Equivariant Translation）、等变旋转（Equivariant Rotation）、物体旋转（Object Rotation）。变换后场景在视觉上真实，且严格保持任务的成功条件。
Figure 3: Visualization of the three equivariant transformations. Top row: initial states of the demonstrations; bottom row: goal states. From left to right: Equivariant Translation, Equivariant Rotation, Object Rotation. The transformed scenes look visually realistic and strictly preserve the success condition of the task.
03实验Experiments
实验分为两部分：（1）基于 RLBench 仿真基准的系统评估，以 PerAct 为基线策略，测试 9 个任务的单次与多次示范设置；（2）在真实 Franka Emika Panda 机器人上的 5 个操控任务验证，每任务各测试 10 个分布内（in-distribution）和 10 个分布外（out-of-distribution）配置。The evaluation has two parts: (1) a systematic assessment on the RLBench simulation benchmark, taking PerAct as the baseline policy and testing 9 tasks under single- and multi-demonstration settings; (2) validation on 5 manipulation tasks with a real Franka Emika Panda robot, where every task is tested on 10 in-distribution and 10 out-of-distribution configurations.
仿真实验：RLBench 基准（one-shot，每变体1个示范）Simulation: the RLBench benchmark (one-shot, one demonstration per variation)
图4：随原始数据量增加，增广数据的持续增益。纵轴为任务成功率，横轴为原始示范数量。蓝色线（DreMa）在两个任务（slide block 和 sort shape）上始终高于橙色线（PerAct Original）。关键发现："Imagined demonstrations keep improving imitation learning even with increasing number of original data"——即使原始数据增多，想象示范仍持续提升性能。特别地，DreMa 使用 5 个示范达到了 PerAct 使用 20 个示范才能达到的精度（sort shape：81% vs 76%）。
Figure 4: Augmented data keeps paying off as the amount of original data grows. The vertical axis is task success rate, the horizontal axis the number of original demonstrations. The blue curve (DreMa) stays above the orange curve (PerAct Original) on both tasks (slide block and sort shape). Key finding: "Imagined demonstrations keep improving imitation learning even with increasing number of original data" — imagined demonstrations keep improving performance even when the original data grows. In particular, DreMa with 5 demonstrations reaches the accuracy that PerAct only attains with 20 demonstrations (sort shape: 81% vs 76%).
消融实验Ablation study
作者在 close jar 任务上分析了各变换组件的贡献（所有结果均为 PerAct 策略、1 shot 设置）：The authors analyze the contribution of each transformation component on the close jar task (all results use the PerAct policy under the 1 shot setting):
配置Configuration
Close Jar 成功率Close Jar success rate
仅 Roto-translationRoto-translation only
41.2%
仅 Object rotationObject rotation only
25.2%
两者组合（DreMa 完整）Both combined (full DreMa)
51.2%
世界模型定位精度（表3，真实机器人实验）：Pick block 的平均定位误差为 0.010 m，Pick shape 为 0.050 m，Push 为 0.049 m，总体均值 0.038 m，验证了 Gaussian Splatting + 物理仿真管线的定量准确性。Localization accuracy of the world model (Table 3, real-robot experiments): the mean localization error is 0.010 m for Pick block, 0.050 m for Pick shape and 0.049 m for Push, with an overall mean of 0.038 m, which confirms the quantitative accuracy of the Gaussian Splatting + physics simulation pipeline.
图5：示范数量对关键点检测的影响（place shape 任务）。从左至右分别为使用 1、5、20 个原始示范时的关键点预测结果（红点为预测关键点）。即使使用极少量示范，DreMa 也能通过增广提供足够的训练信号。
Figure 5: Effect of the number of demonstrations on keypoint detection (place shape task). From left to right: keypoint predictions obtained with 1, 5 and 20 original demonstrations (red dots are the predicted keypoints). Even with very few demonstrations, DreMa can still supply enough training signal through augmentation.
04局限性Limitations
说明：以下局限性均为论文作者明确陈述（stated），见原文 Appendix J 及正文讨论。Note: all limitations below are explicitly stated by the authors; see Appendix J and the discussion in the paper.
当前系统假设整个环境对传感器完全可见，需要完整的环境观测来支持物理建模和外观建模。遮挡严重或视角受限的场景会导致重建质量下降。The current system assumes that the whole environment is fully visible to the sensors, and needs complete observations to support both physical and appearance modeling. Heavy occlusion or restricted viewpoints degrade reconstruction quality.
仅支持刚性物体（Rigid Objects Only）Rigid objects only
现有框架仅能处理刚性物体；铰接（articulated）和可变形（deformable）物体仍是挑战，尽管 Dynamic Gaussian Splatting 领域已有初步进展。The current framework can only handle rigid objects; articulated and deformable objects remain a challenge, although early progress has been made in Dynamic Gaussian Splatting.
开环设计（Open-Loop Design）Open-loop design
世界模型在构建时不集成感觉反馈（sensory feedback）。论文指出，引入感觉反馈可提升准确性并改进策略学习，但这尚未在当前系统中实现。The world model is built without integrating sensory feedback. The paper notes that adding sensory feedback could raise accuracy and improve policy learning, but this is not yet realized in the current system.
大规模场景可扩展性（Scalability）Scalability to large scenes
对于大型、多样化的环境，仿真复杂度会显著增加，当前系统的可扩展性尚未在此类场景中得到验证。For large and diverse environments the simulation complexity grows significantly, and the scalability of the current system has not been validated in such settings.
系统需要精细的相机标定（camera calibration）和物理参数估计（physical parameter estimation），论文 Appendix J 中有详细说明。这些工程要求提高了部署门槛，限制了"即插即用"的实用性。The system requires careful camera calibration and physical parameter estimation, described in detail in Appendix J of the paper. These engineering requirements raise the deployment barrier and limit "plug-and-play" usability.
