ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation

01 动机 MotivationMotivation

语言条件机器人操控任务需要在非结构化环境中完成复杂的物体交互。现有的感知类方法（perceptive methods）提取语义特征做动作预测，而生成类方法（generative methods）通过自监督重建 3D 场景来辅助学习——但两类方法都忽略了一个关键因素。Language-conditioned robotic manipulation requires complex object interaction in unstructured environments. Existing perceptive methods extract semantic features for action prediction, while generative methods reconstruct the 3D scene in a self-supervised way to aid learning — yet both families overlook one crucial factor.

"Conventional robotic manipulation methods...ignore the scene-level spatiotemporal dynamics that depict the physical interaction among objects during manipulation."

机器人在推、抓、拧、滑等任务中，物体之间存在时序上的因果依赖：夹爪移动如何引发物体位移？下一帧场景会如何变化？若模型无法建模这种时空动态，就难以在 long-horizon 或需要多步推理的任务中保持稳定。ManiGaussian 的核心思路是：以「预测未来场景」作为额外监督信号，迫使模型内化场景动态。In tasks such as pushing, grasping, screwing and sliding, temporal causal dependencies exist among objects: how does gripper motion induce object displacement? How will the scene change in the next frame? A model that cannot capture such spatiotemporal dynamics struggles to stay stable on long-horizon tasks or on tasks that require multi-step reasoning. The core idea of ManiGaussian is to use "predicting the future scene" as an extra supervisory signal that forces the model to internalize scene dynamics.

teaser — 图 1：对比展示。在 "stack two rose blocks" 任务中，GNFactor 无法正确抓取绿色底座，而 ManiGaussian 通过编码场景动态成功完成任务。右侧显示两个模型在 RLBench 全部 10 个任务上的平均成功率对比：ManiGaussian（44.8%）vs GNFactor（31.7%），以及训练效率（2.29× 更快达到相同性能）。Figure 1: Comparison. On the "stack two rose blocks" task, GNFactor fails to grasp the green base correctly, whereas ManiGaussian completes the task by encoding scene dynamics. The right panel compares the average success rate of the two models over all 10 RLBench tasks: ManiGaussian (44.8%) vs GNFactor (31.7%), together with training efficiency (2.29× faster to reach the same performance).

44.8%平均成功率（10 tasks）Average success rate (10 tasks)

+13.1%超越 GNFactor（SOTA）Gain over GNFactor (SOTA)

2.29×训练速度（相同性能）Training speed (same performance)

166任务变体总数Total task variations

02 方法 MethodMethod

ManiGaussian 由两个核心模块组成：（1）动态 Gaussian Splatting 框架——将语义特征传播建模在 Gaussian 嵌入空间中；（2）Gaussian 世界模型——基于当前观测和动作预测未来场景，为策略学习提供监督信号。ManiGaussian consists of two core modules: (1) a dynamic Gaussian Splatting framework that models semantic feature propagation in the Gaussian embedding space; (2) a Gaussian world model that predicts the future scene from the current observation and action, providing a supervisory signal for policy learning.

pipeline — 图 2：ManiGaussian 整体框架。输入 RGB-D 图像经提升（lifting）和体素化（voxelization）后进入动态 Gaussian Splatting 模块。Gaussian 世界模型由表征网络 q_ϕ、Gaussian regressor g_ϕ、变形预测器 p_ϕ 和 Gaussian renderer 组成，输出未来场景预测用于辅助监督；最终动作预测头输出末端执行器的平移（ℝ^100³ 体素空间）、旋转（ℝ^(72×3)）、开合度和碰撞信号。Figure 2: Overall ManiGaussian framework. Input RGB-D images are lifted and voxelized before entering the dynamic Gaussian Splatting module. The Gaussian world model comprises a representation network q_ϕ, a Gaussian regressor g_ϕ, a deformation predictor p_ϕ and a Gaussian renderer, and outputs future scene predictions used as auxiliary supervision; the final action prediction head outputs the end-effector translation (ℝ^100³ voxel space), rotation (ℝ^(72×3)), gripper openness and the collision signal.

动态 Gaussian Splatting（Dynamic Gaussian Splatting）Dynamic Gaussian Splatting

标准 3DGS 用静态 Gaussian 基元表示场景，参数 θᵢ = (μᵢ, cᵢ, rᵢ, sᵢ, σᵢ, fᵢ) 包含位置、颜色、旋转、尺度、不透明度和语义特征。ManiGaussian 将其扩展为时间相关形式：Standard 3DGS represents a scene with static Gaussian primitives, whose parameters θᵢ = (μᵢ, cᵢ, rᵢ, sᵢ, σᵢ, fᵢ) cover position, color, rotation, scale, opacity and semantic feature. ManiGaussian extends them into a time-dependent form:

θᵢ⁽ᵗ⁾ = (μᵢ⁽ᵗ⁾, cᵢ⁽ᵗ⁾, rᵢ⁽ᵗ⁾, sᵢ⁽ᵗ⁾, σᵢ⁽ᵗ⁾, fᵢ⁽ᵗ⁾)

针对刚体操控任务，位置和旋转随时间变化：μᵢ⁽ᵗ⁺¹⁾ = μᵢ⁽ᵗ⁾ + Δμᵢ⁽ᵗ⁾，而颜色、尺度、不透明度和语义特征保持不变。这样，变形预测器只需预测 Δμ 和 Δr，大大降低了建模难度。For rigid-body manipulation, position and rotation vary over time: μᵢ⁽ᵗ⁺¹⁾ = μᵢ⁽ᵗ⁾ + Δμᵢ⁽ᵗ⁾, while color, scale, opacity and semantic feature stay unchanged. The deformation predictor therefore only has to predict Δμ and Δr, which greatly reduces the modeling difficulty.

Gaussian 世界模型（Gaussian World Model）Gaussian World Model

世界模型由四个子模块串联构成：The world model is composed of four cascaded submodules:

表征网络 q_ϕ：从 RGB-D 观测中提取 per-point 特征。Representation network q_ϕ: extracts per-point features from the RGB-D observation.
Gaussian regressor g_ϕ：将 point 特征回归为当前时刻的 Gaussian 参数集合。Gaussian regressor g_ϕ: regresses the point features into the set of Gaussian parameters at the current timestep.
变形预测器 p_ϕ：以当前 Gaussian 参数和动作为条件，预测 t→t+1 的形变 Δθ。Deformation predictor p_ϕ: conditioned on the current Gaussian parameters and the action, predicts the deformation Δθ from t to t+1.
Gaussian renderer：利用更新后的 Gaussian 参数可微渲染出未来场景图像，与真实 t+1 帧计算重建损失。Gaussian renderer: differentiably renders the future scene image from the updated Gaussian parameters and computes the reconstruction loss against the ground-truth t+1 frame.

训练包含 3,000 次迭代的热身阶段（warm-up），期间冻结变形预测器，先建立稳定的几何表征，再加入动态建模。Training includes a warm-up stage of 3,000 iterations during which the deformation predictor is frozen, so that a stable geometric representation is established before dynamics modeling is added.

训练目标（Learning Objectives）Learning Objectives

总损失由四项组成：The total loss consists of four terms:

当前几何重建 ℒ_Geo：L₂ 图像重建损失 ‖C⁽ᵗ⁾ − Ĉ⁽ᵗ⁾‖₂²Current geometry reconstruction ℒ_Geo: L₂ image reconstruction loss ‖C⁽ᵗ⁾ − Ĉ⁽ᵗ⁾‖₂²
语义特征一致性 ℒ_Sem：基于 Stable Diffusion 语义特征的余弦距离 1 − σ_cos(F⁽ᵗ⁾, F̂⁽ᵗ⁾)Semantic feature consistency ℒ_Sem: cosine distance over Stable Diffusion semantic features, 1 − σ_cos(F⁽ᵗ⁾, F̂⁽ᵗ⁾)
动作预测 ℒ_Act：平移、旋转、开合度、碰撞的交叉熵损失Action prediction ℒ_Act: cross-entropy losses on translation, rotation, gripper openness and collision
未来动态重建 ℒ_Dyna：‖Ĉ⁽ᵗ⁺¹⁾(a⁽ᵗ⁾, o⁽ᵗ⁾) − C⁽ᵗ⁺¹⁾‖₂²Future dynamics reconstruction ℒ_Dyna: ‖Ĉ⁽ᵗ⁺¹⁾(a⁽ᵗ⁾, o⁽ᵗ⁾) − C⁽ᵗ⁺¹⁾‖₂²

ℒ = ℒ_Act + λ_Geo · ℒ_Geo + λ_Sem · ℒ_Sem + λ_Dyna · ℒ_Dyna

03 实验 ExperimentsExperiments

在 RLBench 基准的 10 个任务、166 个变体上进行评测，每个任务提供 20 个专家演示。使用单前置摄像头（128×128 分辨率）做推理，20 路多视角摄像头作为 Gaussian Splatting 的训练监督。基线包括 PerAct（感知类）、PerAct(4cam)（4 摄像头版本）和 GNFactor（生成类）。Evaluation is carried out on 10 tasks and 166 variations of the RLBench benchmark, with 20 expert demonstrations provided per task. Inference uses a single front camera (128×128 resolution), while 20 multi-view cameras serve as training supervision for Gaussian Splatting. Baselines include PerAct (perceptive), PerAct(4cam) (the four-camera version) and GNFactor (generative).

Table 1：各任务成功率对比（%）Table 1: Per-task success rate comparison (%)

任务 TaskTask	PerAct	PerAct(4cam)	GNFactor	ManiGaussian
close jar	18.7	21.3	25.3	28.0
open drawer	54.7	44.0	76.0	76.0
sweep to dustpan	0.0	0.0	28.0	64.0
meat off grill	40.0	65.3	57.3	60.0
turn tap	38.7	46.7	50.7	56.0
slide block	18.7	16.0	20.0	24.0
put in drawer	2.7	6.7	0.0	16.0
drag stick	5.3	12.0	37.3	92.0
push buttons	18.7	9.3	18.7	20.0
stack blocks	6.7	5.3	4.0	12.0
Average	20.4	22.7	31.7	44.8

注：meat off grill 任务中 PerAct(4cam) 以 65.3% 优于 ManiGaussian 的 60.0%，open drawer 中两者并列最优（76.0%）。其余 8 个任务 ManiGaussian 均为最优。Note: on meat off grill, PerAct(4cam) reaches 65.3% and outperforms ManiGaussian's 60.0%; on open drawer the two tie for the best result (76.0%). ManiGaussian is the best method on the remaining 8 tasks.

qualitative results — 图 4：定性对比。左列展示 "slide block to yellow target"，右列展示 "turn left tap"。GNFactor（上行）在这两个任务中失败，ManiGaussian（下行）通过理解场景动态成功完成操控。Figure 4: Qualitative comparison. The left column shows "slide block to yellow target" and the right column shows "turn left tap". GNFactor (top row) fails on both tasks, whereas ManiGaussian (bottom row) succeeds by understanding scene dynamics.

novel view synthesis — 图 5：新视角合成与未来场景预测。上行为当前场景的 Gaussian 渲染（不同视角），下行为执行动作后的未来场景预测。模型不仅重建了当前几何，还能预测夹爪交互引发的物体形变。Figure 5: Novel view synthesis and future scene prediction. The top row shows Gaussian renderings of the current scene from different viewpoints; the bottom row shows the predicted future scene after the action is executed. The model not only reconstructs the current geometry but also predicts the object deformation induced by gripper interaction.

Table 2：消融实验（%，按任务类别分组）Table 2: Ablation study (%, grouped by task category)

Geo	Sem	Dyna	Planning	Long-horizon	Tools	Motion	Screw	Occlusion	Average
✗	✗	✗	36.0	2.0	25.3	52.0	4.0	28.0	23.6
✓	✗	✗	46.0	4.0	52.0	52.0	24.0	60.0	39.2
✓	✓	✗	46.0	8.0	53.3	64.0	28.0	56.0	41.6
✓	✗	✓	54.0	10.0	49.3	64.0	24.0	72.0	43.6
✓	✓	✓	40.0	14.0	60.0	56.0	28.0	76.0	44.8

消融分析显示：几何重建（Geo）带来最显著的提升（23.6% → 39.2%，+15.6%），说明 3D 几何理解是操控成功的基础；动态预测（Dyna）在 long-horizon 和 occlusion 类任务上效果最突出；三者协同使用达到最优的 44.8%。The ablation shows that geometry reconstruction (Geo) brings the most significant gain (23.6% → 39.2%, +15.6%), indicating that 3D geometric understanding is the foundation of successful manipulation; dynamics prediction (Dyna) is most effective on long-horizon and occlusion tasks; combining all three reaches the best result of 44.8%.

learning curve — 图 3：学习曲线对比。ManiGaussian 以 2.29× 更快的速度超越 GNFactor 的最终性能，并最终取得 1.18× 的绝对性能提升。横轴为训练步数，纵轴为平均成功率。Figure 3: Learning curve comparison. ManiGaussian surpasses the final performance of GNFactor 2.29× faster and eventually achieves a 1.18× absolute performance gain. The horizontal axis is training steps and the vertical axis is average success rate.

04 局限性 LimitationsLimitations

Note: 以下第一条为论文作者明确陈述（stated）的局限性，其余为从设计中推断的局限性（inferred from the design）。The first item below is a limitation explicitly stated by the authors; the rest are inferred from the design.

多视角监督与相机标定要求（Stated）Multi-view supervision and camera calibration requirements (Stated)

论文明确指出："The limitations stem from the necessity of multiple view supervision with camera calibration for the Gaussian Splatting framework." 训练时需要 20 路多视角摄像头及其内外参数，在现实部署场景中（如单目移动机器人）代价较高，限制了方法的实用性。The paper states explicitly: "The limitations stem from the necessity of multiple view supervision with camera calibration for the Gaussian Splatting framework." Training requires 20 multi-view cameras together with their intrinsic and extrinsic parameters, which is costly in real deployment scenarios (e.g. a monocular mobile robot) and limits the practicality of the method.

刚体假设（Inferred）Rigid-body assumption (Inferred)

当前的动态建模仅对刚体物体进行位置和旋转的变形预测，不支持柔性物体（如绳子、布料）或流体的形变。对于高度非刚性的操控任务，方法可能需要扩展。The current dynamics modeling only predicts position and rotation deformations of rigid objects, and does not support the deformation of soft objects (such as ropes or cloth) or of fluids. Highly non-rigid manipulation tasks would likely require an extension of the method.

单步动作预测（Inferred）Single-step action prediction (Inferred)

ManiGaussian 预测 t→t+1 的未来场景作为监督信号，但对于需要多步规划的 long-horizon 任务（如 "stack blocks" 仅达 12.0%），单步世界模型的预测范围可能不足以充分约束策略。ManiGaussian predicts the future scene from t to t+1 as a supervisory signal, but for long-horizon tasks that require multi-step planning (e.g. "stack blocks" reaches only 12.0%), the prediction horizon of a single-step world model may not constrain the policy sufficiently.

分辨率与推理效率（Inferred）Resolution and inference efficiency (Inferred)

训练使用 128×128 分辨率图像，推理仅依赖单一前置摄像头。当场景中存在大量细粒度操控对象时，低分辨率可能导致感知精度下降。此外，Gaussian Splatting 的渲染开销在实时部署中尚待优化。Training uses 128×128 resolution images and inference relies on a single front camera only. When the scene contains many fine-grained manipulation objects, the low resolution may degrade perception accuracy. In addition, the rendering cost of Gaussian Splatting still needs optimization for real-time deployment.