用多模态目标学习多样化机器人操作策略Learning versatile robot manipulation policies from multimodal goals
Moritz Reuss, Ömer Erdinç Yağmurlu, Fabian Wenzel, Rudolf Lioutikov · RSS 2024
MDT（Multimodal Diffusion Transformer）提出了一种基于扩散模型的机器人操作策略框架，能够同时接受图像目标和语言目标作为条件，仅需极少量的语言标注（2%）便可超越全标注 baseline。两个核心辅助任务——Masked Generative Foresight（MGF）和 Contrastive Latent Alignment（CLA）——通过自监督方式对齐多模态表征，在 CALVIN 和 LIBERO 基准上均取得当时最优性能。MDT (Multimodal Diffusion Transformer) proposes a diffusion-based framework for robot manipulation policies that can be conditioned on both image goals and language goals, and surpasses a fully annotated baseline while using only a very small fraction of language annotations (2%). Two core auxiliary tasks — Masked Generative Foresight (MGF) and Contrastive Latent Alignment (CLA) — align multimodal representations in a self-supervised manner, achieving state-of-the-art performance at the time on both the CALVIN and LIBERO benchmarks.
现实中的机器人示教数据往往只有图像目标，语言标注极为稀缺。如何在只有少量语言标注的情况下，让策略同时支持图像目标和语言目标的条件输入，是构建通用机器人操作系统的核心挑战。Real-world robot demonstration data often provides image goals only, while language annotations are extremely scarce. Enabling a policy to accept both image-goal and language-goal conditioning when only few language annotations are available is a core challenge in building general-purpose robot manipulation systems.
"We propose a multimodal diffusion policy that learns robot manipulation behaviors from multimodal goals with few language annotations."
Figure 1：MDT 整体架构。系统由多模态 Transformer Encoder（处理当前观测和多模态目标）和 Diffusion Decoder（生成动作序列）组成。Encoder 通过两个自监督辅助任务（MGF 和 CLA）学习对齐图像与语言目标的隐表征，Decoder 通过 10 步 DDIM 采样生成机器人动作序列。Figure 1: Overall MDT architecture. The system consists of a multimodal Transformer Encoder (processing the current observation and the multimodal goal) and a Diffusion Decoder (generating the action sequence). The Encoder learns to align the latent representations of image and language goals through two self-supervised auxiliary tasks (MGF and CLA), while the Decoder generates the robot action sequence with 10-step DDIM sampling.
4.52CALVIN ABCD→D 平均完成链长（MDT-V）CALVIN ABCD→D average completed chain length (MDT-V)
+15%超越 RoboFlamingo 在 ABCD→D 子集上的绝对提升over RoboFlamingo absolute gain on the ABCD→D split
2%语言标注比例 仍超越全标注 Transformer-BClanguage annotation ratio still surpasses the fully annotated Transformer-BC
<10%相比 RoboFlamingo 所用 可训练参数量比例trainable parameter ratio relative to RoboFlamingo
在大多数现实数据集中，语言指令只覆盖小部分演示，而图像目标（goal image）则可以免标注地从轨迹末帧获取。MDT 通过同时支持两种目标模态，充分利用了这类不完整标注数据集。与依赖大规模预训练（如 RoboFlamingo）的方法相比，MDT 无需预训练，参数量更少，却取得更好的效果。In most real datasets, language instructions cover only a small portion of the demonstrations, whereas image goals (goal image) can be obtained annotation-free from the last frame of a trajectory. By supporting both goal modalities, MDT makes full use of such partially annotated datasets. Compared with methods that rely on large-scale pretraining (e.g. RoboFlamingo), MDT needs no pretraining and has fewer parameters, yet achieves better results.
02 Method
MDT 由多模态 Transformer Encoder 和扩散式 Decoder 组成，通过两个自监督辅助目标——Masked Generative Foresight（MGF）和 Contrastive Latent Alignment（CLA）——在极少语言标注下学习统一的多模态目标条件策略。MDT consists of a multimodal Transformer Encoder and a diffusion Decoder, and learns a unified multimodal goal-conditioned policy under very few language annotations through two self-supervised auxiliary objectives — Masked Generative Foresight (MGF) and Contrastive Latent Alignment (CLA).
整体架构Overall Architecture
Encoder 使用冻结的 CLIP 文本编码器处理语言目标，使用 ResNet 编码当前观测和图像目标，并融合为统一的 latent goal-conditioned state representation。Decoder 为 GPT 风格的因果 Transformer，通过 Adaptive Layer Normalization（AdaLN）注入扩散时间步噪声，迭代去噪生成长度为 10 的动作序列。训练采用连续时间 SDE，噪声范围为 [0.001, 80]；推理使用 10 步 DDIM 采样。The Encoder processes language goals with a frozen CLIP text encoder and encodes the current observation and the image goal with a ResNet, fusing them into a unified latent goal-conditioned state representation. The Decoder is a GPT-style causal Transformer that injects the diffusion timestep noise via Adaptive Layer Normalization (AdaLN) and iteratively denoises to produce an action sequence of length 10. Training uses a continuous-time SDE with a noise range of [0.001, 80]; inference uses 10-step DDIM sampling.
MGF 是一个自监督辅助任务：用 Vision Transformer 在当前观测的 latent 表征条件下，预测并重建未来第 v 步（论文设 v=3）帧的被遮挡 patch（masking ratio=0.75）。这迫使 encoder 学习包含足够未来状态信息的表征，且与目标模态（图像 or 语言）无关，从而对齐两类目标的表征空间。MGF 单独使用即可为 CALVIN 带来约 25% 的性能提升；迁移到 MT-ACT baseline 时，提升达 44%。MGF is a self-supervised auxiliary task: a Vision Transformer, conditioned on the latent representation of the current observation, predicts and reconstructs the masked patches (masking ratio=0.75) of the frame v steps into the future (the paper sets v=3). This forces the encoder to learn representations carrying sufficient information about future states, independent of the goal modality (image or language), thereby aligning the representation spaces of the two goal types. MGF alone brings about a 25% performance gain on CALVIN; transferred to the MT-ACT baseline, the gain reaches 44%.
Figure 2：Masked Generative Foresight 辅助任务。Encoder 编码当前观测和目标（图像或语言），ViT 解码器在 latent 表征条件下重建未来帧的被遮挡 patch，迫使表征编码充足的未来预测信息，实现图像/语言目标表征的自然对齐。Figure 2: The Masked Generative Foresight auxiliary task. The Encoder encodes the current observation and the goal (image or language), and a ViT decoder reconstructs the masked patches of a future frame conditioned on the latent representation, forcing the representation to encode sufficient future-prediction information and naturally aligning the image- and language-goal representations.
CLA 应用 InfoNCE 对比损失，直接拉近同一轨迹中图像目标条件和语言目标条件下的 latent state embedding。这弥补了 CLIP 对空间任务动态理解不足的缺陷。在 LIBERO-90（任务多样性高）子集上，CLA 的增益尤为显著。两个辅助目标的损失权重均设为 α=β=0.1。CLA applies an InfoNCE contrastive loss to directly pull together the latent state embeddings under image-goal and language-goal conditioning within the same trajectory. This compensates for CLIP's insufficient understanding of spatial task dynamics. On the LIBERO-90 subset (high task diversity), the gain from CLA is particularly pronounced. The loss weights of both auxiliary objectives are set to α=β=0.1.
预训练潜力Pretraining Potential
CLA 和 MGF 均不依赖动作标签，因此可在无动作标注的视频数据上做预训练。论文在 LIBERO-90 上验证：仅用 5 条带动作演示 fine-tune，预训练版本比随机初始化提升约 100%；用 20 条演示，提升约 25%。Neither CLA nor MGF depends on action labels, so both can be pretrained on video data without action annotations. The paper verifies this on LIBERO-90: fine-tuned with only 5 action-labeled demonstrations, the pretrained version improves over random initialization by about 100%; with 20 demonstrations the gain is about 25%.
03 Experiments
在 CALVIN（长时序多任务操作）和 LIBERO（多任务模仿学习）两个基准上评测，并在真实机器人（toy kitchen，4.5 小时非分段 play data，20% 语言标注）上验证。Evaluated on two benchmarks, CALVIN (long-horizon multi-task manipulation) and LIBERO (multi-task imitation learning), and validated on a real robot (toy kitchen, 4.5 hours of unsegmented play data, 20% language annotations).
CALVIN 基准（Table I）CALVIN Benchmark (Table I)
方法Method
D→D Avg
ABCD→D Avg
HULC
2.68
3.06
LAD
2.88
—
Distill-D
2.97
3.16
MT-ACT
2.98
2.80
RoboFlamingo
—
4.09
MDT（ours）MDT (ours)
3.59
4.41
MDT-V（ours）MDT-V (ours)
3.72
4.52
MDT-V（视频预训练版本）在 ABCD→D 子集达到 4.52 平均链长，较 RoboFlamingo（4.09）提升约 15%；在 D→D 子集超越第二名约 20%。MDT 可训练参数量不足 RoboFlamingo 的 10%，且无需大规模预训练。MDT-V (the video-pretrained version) reaches an average chain length of 4.52 on the ABCD→D split, about 15% higher than RoboFlamingo (4.09); on the D→D split it exceeds the runner-up by about 20%. MDT uses fewer than 10% of RoboFlamingo's trainable parameters and needs no large-scale pretraining.
LIBERO 基准（Table II，仅 2% 语言标注）LIBERO Benchmark (Table II, only 2% language annotations)
仅用 2% 语言标注，MDT+CLA+MGF 在多个子集上超越使用 100% 标注的 Transformer-BC，平均成功率达 74.3%，比 Distill-D（56.0%）高约 18 个百分点。辅助损失平均带来约 8.5% 的提升。With only 2% language annotations, MDT+CLA+MGF surpasses Transformer-BC trained with 100% annotations on several subsets, reaching an average success rate of 74.3%, about 18 percentage points above Distill-D (56.0%). The auxiliary losses yield an average gain of about 8.5%.
真实机器人实验Real-Robot Experiments
在真实 toy kitchen 场景下，单任务成功率：MDT-V 58%，MT-ACT 仅 25%。多任务链式完成：带辅助目标的 MDT 平均完成 1.56 步，展现了从非分段、稀疏标注 play 数据中学习多任务策略的能力。In the real toy kitchen setting, the single-task success rate is 58% for MDT-V and only 25% for MT-ACT. For multi-task chaining, MDT with the auxiliary objectives completes 1.56 steps on average, demonstrating the ability to learn multi-task policies from unsegmented, sparsely annotated play data.
Figure 3：实验环境概览。左：CALVIN 基准，四个子环境各有不同位置和纹理；中：LIBERO 任务集合，涵盖空间、物体、目标和 LIBERO-90 四个子集；右：真实 toy kitchen 机器人实验场景。Figure 3: Overview of the experimental environments. Left: the CALVIN benchmark, whose four sub-environments each have different positions and textures; middle: the LIBERO task suite, covering the spatial, object, goal and LIBERO-90 subsets; right: the real toy kitchen robot setting.
Ablations
Figure 5：MGF 和 CLA 用于无动作视频预训练的效果研究。在 LIBERO-90 上预训练后，仅用 5 条带动作演示 fine-tune 即可获得约 100% 的性能提升；用 20 条演示仍有约 25% 提升，验证了辅助目标的预训练迁移能力。Figure 5: A study of using MGF and CLA for action-free video pretraining. After pretraining on LIBERO-90, fine-tuning with only 5 action-labeled demonstrations yields about a 100% performance gain; with 20 demonstrations the gain is still about 25%, confirming the pretraining transferability of the auxiliary objectives.
MGF masking ratio=0.75 为最优；foresight distance v=3 在性能与方差之间平衡最佳。MGF masking ratio=0.75 is optimal; a foresight distance of v=3 gives the best balance between performance and variance.
冻结 CLIP 编码器并非必须——使用可训练 ResNet 可达到相近效果。Freezing the CLIP encoder is not mandatory — a trainable ResNet reaches comparable results.
去除 Transformer Encoder 时，CALVIN 平均链长降至 1.41，证明编码器结构至关重要。Removing the Transformer Encoder drops the average CALVIN chain length to 1.41, showing that the encoder architecture is essential.
MGF 在 LIBERO-Long（无子目标标注）子集上效果有限，CLA 在高任务多样性（LIBERO-90）场景增益更明显。MGF is of limited benefit on the LIBERO-Long subset (without sub-goal annotations), while CLA gives a clearer gain in high task-diversity settings (LIBERO-90).
04 Limitations
Note:以下局限性部分由作者在论文中明确指出（标注为 stated），部分从系统设计推断（标注为 inferred）。Some of the limitations below are explicitly stated by the authors in the paper (marked stated), while others are inferred from the system design (marked inferred).
MGF 在无子目标标注的长时任务上效果受限（stated）MGF is limited on long-horizon tasks without sub-goal annotations (stated)
MGF 依赖轨迹中子目标的图像表征来学习未来预测，当任务数据缺乏子目标标注（如 LIBERO-Long）时，辅助任务无法有效发挥作用，该子集上 MGF 带来的提升有限。MGF relies on the image representations of sub-goals within a trajectory to learn future prediction; when the task data lacks sub-goal annotations (e.g. LIBERO-Long), the auxiliary task cannot take effect, and the gain from MGF on that subset is limited.
多步扩散推理速度慢于非扩散方法（stated）Multi-step diffusion inference is slower than non-diffusion methods (stated)
MDT 在推理时需要 10 步 DDIM 采样，这使得它比单步策略（如 ACT）慢，在对延迟敏感的实时控制场景中存在瓶颈。MDT requires 10-step DDIM sampling at inference, which makes it slower than single-step policies (e.g. ACT) and creates a bottleneck in latency-sensitive real-time control settings.
真实机器人多任务性能仍有提升空间（stated）Real-robot multi-task performance still leaves room for improvement (stated)
在非分段的稀疏标注 play 数据上，多任务链式完成平均仅 1.56 步，任务之间的转换仍具挑战，作者指出相机位置导致的自遮挡（self-occlusion）是部分失败原因。On unsegmented, sparsely annotated play data, multi-task chaining completes only 1.56 steps on average; transitions between tasks remain challenging, and the authors point to self-occlusion caused by the camera position as one cause of failure.
辅助损失权重需要手动调参（inferred）Auxiliary loss weights require manual tuning (inferred)
MGF 和 CLA 的损失权重均固定为 α=β=0.1，不同任务、数据集下最优权重可能存在差异，自动化调参策略尚未探索。The loss weights of MGF and CLA are both fixed to α=β=0.1; the optimal weights may differ across tasks and datasets, and automated tuning strategies remain unexplored.
