3D Diffusion Policy

01 动机 MotivationMotivation

模仿学习为机器人提供了一条习得复杂技能的有效路径，但现有方法主要依赖 2D 图像或深度图作为视觉输入，在处理空间理解、视角变化与实例泛化等问题时存在明显局限。如何以最简单的方式引入 3D 信息，同时保持策略学习的高效性与泛化性，是本文要解决的核心问题。Imitation learning offers an effective route for robots to acquire complex skills, yet existing methods rely mainly on 2D images or depth maps as visual input and show clear limitations in spatial understanding, viewpoint change and instance-level generalization. How to introduce 3D information in the simplest possible way while preserving the efficiency and generalization of policy learning is the core problem this paper addresses.

"Enabling robots to dexterous manipulation of objects is a longstanding challenge for roboticists, primarily due to the necessity to manage a broader spectrum of motor skills… Visual imitation learning, which takes high-dimensional visual observations such as images or depth maps, eases the need for task-specific state estimation and thus gains its popularity."

现有方法（如 Diffusion Policy）以 2D RGB 图像为输入，面临两大核心挑战：Existing methods such as Diffusion Policy take 2D RGB images as input and face two core challenges:

• 空间感知不足：2D 图像缺乏深度信息，难以精确理解物体在三维空间中的位置与形状，对灵巧操作（如倒液、卷薄片）尤为不利。Insufficient spatial perception: 2D images lack depth information, making it hard to understand precisely where objects are and what shape they have in 3D space, which is especially harmful for dexterous manipulation (e.g. pouring liquid, rolling a thin sheet).
• 泛化能力有限：基于图像的策略对视角变化、光照变化和目标外观变化高度敏感，往往需要大量多样化的演示数据才能泛化。Limited generalization: image-based policies are highly sensitive to changes in viewpoint, lighting and target appearance, and usually need large amounts of diverse demonstration data to generalize.

本文核心假设：稀疏点云提供的紧凑 3D 表征天然具备几何一致性，无需颜色通道即可实现优越的泛化，且单视角点云（仅一台 RealSense L515 相机）已足够。The core hypothesis of this paper: the compact 3D representation provided by a sparse point cloud is inherently geometrically consistent and achieves superior generalization without any color channel, and a single-view point cloud (from just one RealSense L515 camera) already suffices.

02 方法 MethodMethod

DP3 由两个模块构成：Perception（感知）模块将来自单视角深度相机的稀疏点云编码为紧凑的 3D 特征；Decision（决策）模块以 Diffusion Policy 为骨干，在机器人状态（关节角）条件下预测动作序列。两模块端到端联合训练。DP3 consists of two modules: the Perception module encodes the sparse point cloud from a single-view depth camera into a compact 3D feature; the Decision module takes Diffusion Policy as its backbone and predicts action sequences conditioned on the robot state (joint angles). The two modules are trained jointly end-to-end.

Perception：从点云到紧凑 3D 表征Perception: From Point Clouds to a Compact 3D Representation

深度图经相机内参投影为点云，随后经过以下处理：The depth map is projected into a point cloud with the camera intrinsics and then processed as follows:

• 裁剪（Cropping）：保留以机器人为中心的边界框内的点，去除地面与背景噪声，减少冗余点。Cropping: keep only the points inside a robot-centered bounding box, removing ground and background noise and reducing redundant points.
• 下采样（Downsampling）：使用最远点采样（FPS）将点云稀疏化至 512 或 1024 点，大幅降低计算量。Downsampling: farthest point sampling (FPS) sparsifies the point cloud to 512 or 1024 points, greatly reducing computation.
• 颜色通道去除：不使用 RGB 颜色，仅保留 XYZ 坐标，从而对外观变化天然鲁棒。Removing the color channel: no RGB color is used and only XYZ coordinates are kept, which makes the policy naturally robust to appearance changes.
• DP3 Encoder：采用轻量 MLP 网络对点云编码，最终输出 64 维紧凑特征向量 v。该设计远比 PointNet++、Point Transformer 等复杂编码器计算高效，实验表明其精度也更优（见 Table V）。DP3 Encoder: a lightweight MLP network encodes the point cloud and finally outputs a 64-dimensional compact feature vector v. This design is far more compute-efficient than complex encoders such as PointNet++ or Point Transformer, and experiments show it is also more accurate (see Table V).

Decision：Diffusion Policy 动作生成Decision: Action Generation with Diffusion Policy

决策模块采用基于卷积网络的 Diffusion Policy（DDIM 噪声调度器）。以机器人关节状态 s 与感知特征 v 为条件，通过去噪过程预测未来 T 步动作序列。训练目标为：The decision module adopts a convolutional Diffusion Policy (DDIM noise scheduler). Conditioned on the robot joint state s and the perception feature v, it predicts the action sequence for the next T steps through a denoising process. The training objective is:

L = MSE(ε^θ(α_ka⁰ + β_kε, k, v, s), ε)

其中 α_k、β_k 为噪声调度参数，采用 one-step noise adding 策略。推理时使用 sample prediction（而非 epsilon prediction）以获得更快的收敛速度。实现细节：使用卷积网络 Diffusion Policy，DDIM 噪声调度；在 MetaWorld 任务上训练 1000 epochs，其他任务训练 8000 epochs；batch size 为 128；Projection head 输出维度为 1024 或 256。Here α_k and β_k are noise-schedule parameters, and a one-step noise adding strategy is used. At inference, sample prediction (rather than epsilon prediction) is used for faster convergence. Implementation details: a convolutional Diffusion Policy with DDIM noise scheduling; training for 1000 epochs on MetaWorld tasks and 8000 epochs on the others; batch size 128; projection head output dimension 1024 or 256.

03 实验 ExperimentsExperiments

仿真基准：72 个任务，跨 7 个领域（Adroit、Bi-DexHands、DexArt、DexDeform、DexMV、HORA、MetaWorld），涵盖阴影手 / Allegro 手 / 平行夹爪等多种机器人形态，任务类型包含刚性/变形物体操作。每个任务仅使用 10 条演示（DexArt 和 HORA 除外）。真实机器人：4 个灵巧任务（Roll-Up、Dumpling、Drill、Pour），每任务 40 条演示，使用 Frankia 手臂 + Allegro 手 / 平行夹爪，RealSense L515 单目深度相机。Simulation benchmark: 72 tasks across 7 domains (Adroit, Bi-DexHands, DexArt, DexDeform, DexMV, HORA, MetaWorld), covering robot morphologies such as the Shadow Hand, the Allegro Hand and parallel grippers, with task types spanning rigid and deformable object manipulation. Each task uses only 10 demonstrations (except DexArt and HORA). Real robot: 4 dexterous tasks (Roll-Up, Dumpling, Drill, Pour) with 40 demonstrations each, using a Frankia arm plus an Allegro Hand or a parallel gripper and a RealSense L515 monocular depth camera.

关键发现Key Findings

• 效率优势：DP3 在 ~500 epochs 内收敛，而 Diffusion Policy 需要 3000 epochs 且常陷入次优。Efficiency: DP3 converges within ~500 epochs, whereas Diffusion Policy needs 3000 epochs and often ends up in a sub-optimal solution.
• 少演示扩展性：仿真中大多数任务仅用 10 条演示；真实任务用 40 条演示。DP3 在 Adroit 类任务中以更少演示达到可比精度。Scaling with few demonstrations: most simulated tasks use only 10 demonstrations, and the real tasks use 40. On Adroit-style tasks DP3 reaches comparable accuracy with fewer demonstrations.
• 安全性优势：DP3 安全违规率（Safety Violation Rate）远低于基于图像的基线，在 Roll-Up 任务上为 0，总平均为 0.0，而 Diffusion Policy (Depth) 平均为 21.0。Safety: the Safety Violation Rate of DP3 is far below that of image-based baselines — 0 on the Roll-Up task and 0.0 on overall average, versus 21.0 on average for Diffusion Policy (Depth).
• 推理速度：DP3 推理 58.1ms、预测 4 个动作步，而 PointNeXt 等复杂编码器推理更慢（60–120ms）。Inference speed: DP3 takes 58.1ms per inference and predicts 4 action steps, while complex encoders such as PointNeXt are slower (60–120ms).

消融实验 AblationsAblations

论文对 3D 表征选择、点云编码器设计和 DP3 设计细节进行了系统消融（Tables IV–VII）：The paper systematically ablates the choice of 3D representation, the point-cloud encoder design and the design details of DP3 (Tables IV–VII):

• 3D 表征对比（Table IV）：点云 > RGB-D 图像 > Depth 图像 > Voxel，使用点云的 DP3 在 6 个消融任务上平均成功率 100±0，而 RGB-D 为 62±c、Depth 为 80±c（各任务差异较大）。3D representation comparison (Table IV): point cloud > RGB-D image > depth image > voxel; with point clouds, DP3 attains an average success rate of 100±0 over 6 ablation tasks, while RGB-D reaches 62±c and depth 80±c (with large variance across tasks).
• 点云编码器对比（Table V）：DP3 的轻量 MLP Encoder 平均 72.3，优于 PointNet（15.7）、PointNet++（16.0）、Point Transformer（26.0）等。作者将其归因于预训练编码器使用固定相机、需特征变换，与 DP3 的单视角设置不兼容。Point-cloud encoder comparison (Table V): the lightweight MLP encoder of DP3 averages 72.3, ahead of PointNet (15.7), PointNet++ (16.0) and Point Transformer (26.0). The authors attribute this to pre-trained encoders assuming a fixed camera and requiring feature transforms, which is incompatible with the single-view setting of DP3.
• 设计选项（Table VII）：裁剪点云 (+)、使用 LayerNorm、Sample prediction、Projection head 尺寸 1024 等组合最终达到平均 72.3，消除任意一项均会导致性能下降。Design options (Table VII): combining cropped point clouds (+), LayerNorm, sample prediction and a projection head of size 1024 finally reaches an average of 72.3; removing any single item degrades performance.

04 局限性 LimitationsLimitations