RDT-1B: 面向双臂操作的扩散基础模型

01 动机Motivation

双臂操作是机器人走向真实世界的关键能力，但现有方法面临两大核心挑战：一是双臂协调产生的多模态动作分布（同一任务可能存在多种合理的执行方式），二是双臂演示数据的严重匮乏，限制了模型的泛化能力。作者指出，现有扩散模型在机器人操作上最多只达到 93M 参数规模，难以充分利用大规模多机器人数据。Bimanual manipulation is a key capability for robots moving into the real world, yet existing methods face two core challenges: first, the multi-modal action distribution produced by two-arm coordination (one and the same task may admit several reasonable ways of being carried out); second, the severe scarcity of bimanual demonstration data, which limits the generalization ability of models. The authors point out that existing diffusion models for robot manipulation reach at most 93M parameters, making it hard to fully exploit large-scale multi-robot data.

"Bimanual manipulation is essential in robotics, yet developing foundation models is extremely challenging due to the inherent complexity of coordinating two robot arms (leading to multi-modal action distributions) and the scarcity of training data."

**Figure 1.** RDT-1B 总览：作为语言条件化的视觉运动策略，RDT-1B 在未见物体、未见场景、新语言指令、few-shot 新技能和精细灵巧操作等 5 类任务上均展现出最优泛化性能。Overview of RDT-1B: as a language-conditioned visuomotor policy, RDT-1B shows the best generalization performance across 5 categories of tasks — unseen objects, unseen scenes, novel language instructions, few-shot new skills, and fine-grained dexterous manipulation.

1.2B模型参数量（机器人扩散模型最大）Model parameters (largest among robot diffusion models)

56%在多项挑战任务上提升幅度（相对基线）Improvement on several challenging tasks (relative to baselines)

1M+预训练轨迹数 / 21TB 数据Pre-training trajectories / 21TB of data

6K+微调双臂演示条数 / 300+ 任务Bimanual demonstrations for fine-tuning / 300+ tasks

双臂机器人与多模态问题 — **Figure 2.** (a) ALOHA 双臂机器人示意图，配备两臂、夹爪和三个摄像头。 (b) 双臂抓取的多模态示例：相比单臂操作，双臂协调存在更多可行的动作模式，演示数据中的随机选择导致动作分布具有强多模态性。(a) Illustration of the ALOHA bimanual robot, equipped with two arms, grippers and three cameras. (b) An example of multi-modality in bimanual grasping: compared with single-arm manipulation, two-arm coordination admits many more feasible action patterns, and the arbitrary choices made in the demonstrations give the action distribution strong multi-modality.

02 方法Method

RDT 以扩散 Transformer 为骨干，通过三项关键架构改进适配机器人操作的特殊性；同时设计"物理可解释统一动作空间"，将来自 46 个数据集、多种机器人的异构动作统一表示，从而实现跨机器人知识迁移并缓解数据稀缺问题。RDT takes a diffusion Transformer as its backbone and adapts it to the particularities of robot manipulation through three key architectural improvements; it further designs a “physically interpretable unified action space” that gives a unified representation to the heterogeneous actions of many kinds of robots coming from 46 datasets, thereby enabling cross-robot knowledge transfer and alleviating data scarcity.

RDT 架构 — **Figure 3.** RDT 整体框架：各机器人的异构动作空间经统一动作空间映射后输入模型。模型接受本体感知状态、带噪动作、控制频率、扩散时间步、多视角图像（SigLIP 编码）和语言指令（T5-XXL 编码）作为条件，输出去噪后的动作序列（action chunk）。Overall framework of RDT: the heterogeneous action space of each robot is mapped through the unified action space before being fed to the model. The model takes proprioceptive state, noisy actions, control frequency, diffusion timestep, multi-view images (encoded by SigLIP) and language instructions (encoded by T5-XXL) as conditions, and outputs the denoised action sequence (action chunk).

三项关键架构改进Three Key Architectural Improvements

QKNorm & RMSNorm

机器人物理量（如关节角度、速度）数值范围差异极大，标准 Transformer 在大规模预训练时会出现数值不稳定甚至梯度爆炸。引入 Query/Key Normalization 和 RMSNorm 后，训练损失曲线稳定收敛。论文指出："Large-scale pre-training tends to be very unstable or even explode without this modification."The numerical ranges of robot physical quantities (such as joint angles and velocities) differ enormously, and a standard Transformer becomes numerically unstable or even suffers gradient explosion during large-scale pre-training. After introducing Query/Key Normalization and RMSNorm, the training loss curve converges stably. The paper notes: “Large-scale pre-training tends to be very unstable or even explode without this modification.”

MLP Decoder

将标准 Diffusion Transformer 的线性解码器替换为非线性 MLP，以更好地捕捉机器人动力学的非线性特征。消融实验表明："without this design, RDT cannot effectively capture nonlinear dynamics."The linear decoder of the standard Diffusion Transformer is replaced by a nonlinear MLP, in order to better capture the nonlinear characteristics of robot dynamics. The ablation study shows: “without this design, RDT cannot effectively capture nonlinear dynamics.”

Alternating Condition Injection (ACI)

在 Transformer 各层中交替注入图像 token 和文本 token，防止视觉信息淹没语言条件信号。这一设计使 RDT 能够精确理解细粒度语言指令（如"倒三分之一的水"），而不仅仅是粗粒度的任务描述。Image tokens and text tokens are injected alternately across the layers of the Transformer, preventing visual information from drowning out the language conditioning signal. This design enables RDT to understand fine-grained language instructions precisely (for example “pour one third of the water”), rather than only coarse-grained task descriptions.

物理可解释统一动作空间Physically Interpretable Unified Action Space

不同机器人的动作空间（关节角度、末端执行器位姿、夹爪开合等）在维度和物理单位上各不相同。 RDT 设计了一套统一表示，在保留物理含义的同时消除了跨平台异构性，从而能够在 46 个不同机器人数据集（超 1M 轨迹、21TB）上联合预训练，而不引入负迁移。The action spaces of different robots (joint angles, end-effector poses, gripper opening and closing, and so on) differ in both dimensionality and physical units. RDT designs a unified representation that eliminates cross-platform heterogeneity while preserving physical meaning, which makes it possible to pre-train jointly on 46 different robot datasets (over 1M trajectories, 21TB) without introducing negative transfer.

多模态输入编码Multi-modal Input Encoding

图像：使用预训练 SigLIP 视觉编码器，支持多视角输入Images: a pre-trained SigLIP visual encoder, supporting multi-view input
语言：使用预训练 T5-XXL 语言模型，支持细粒度指令Language: a pre-trained T5-XXL language model, supporting fine-grained instructions
低维输入：本体感知状态、控制频率等使用带 Fourier 特征的 MLP 编码Low-dimensional inputs: proprioceptive state, control frequency and the like are encoded by an MLP with Fourier features
随机独立掩码：训练时随机屏蔽部分模态，防止模型过度依赖某一输入Random independent masking: some modalities are randomly masked during training, preventing the model from over-relying on any single input

推理加速：使用 DPM-Solver++ 将扩散步数从 100 步缩减至 5 步，实现 6 Hz chunk 频率 / 381 Hz 实际动作频率，满足实时控制需求。Inference acceleration: DPM-Solver++ reduces the number of diffusion steps from 100 to 5, achieving a 6 Hz chunk frequency / 381 Hz effective action frequency, which meets the requirements of real-time control.

03 实验Experiments

在 ALOHA 双臂机器人上评测 7 类挑战任务，涵盖 5 个研究问题：未见物体泛化、未见场景泛化、语言指令跟随、few-shot 新技能学习、精细灵巧操作。基线方法包括 ACT、OpenVLA 和 Octo。预训练使用 48 块 H100 80GB GPU 训练一个月（1M 步）；微调同样硬件运行 3 天（130K 步）。7 categories of challenging tasks are evaluated on the ALOHA bimanual robot, covering 5 research questions: generalization to unseen objects, generalization to unseen scenes, language instruction following, few-shot new-skill learning, and fine-grained dexterous manipulation. The baselines include ACT, OpenVLA and Octo. Pre-training used 48 H100 80GB GPUs for one month (1M steps); fine-tuning ran on the same hardware for 3 days (130K steps).

七类任务定义与可视化 — **Figure 5.** 七类任务的定义与可视化，包括对每个子任务的语言指令、随机化条件和成功判定标准。任务从左至右难度递增：洗杯（未见物体）、倒水（未见场景）、精确倒水（语言指令跟随）、传递物品（5-shot）、折叠短裤（1-shot）、机器狗手柄操作（灵巧操作）。Definition and visualization of the seven task categories, including the language instruction, the randomization conditions and the success criterion for every sub-task. Difficulty increases from left to right: Wash Cup (unseen object), Pour Water (unseen scene), precise pouring (language instruction following), Handover (5-shot), Fold Shorts (1-shot), and robot-dog handle manipulation (dexterous manipulation).

主实验结果（Table 3，成功率 %）Main Results (Table 3, success rate %)

任务Task	ACT	OpenVLA	Octo	RDT (ours)
Wash Cup · 总体（未见物体）Wash Cup · overall (unseen object)	12.5	0	0	62.5
Pour Water · 总体（未见场景）Pour Water · overall (unseen scene)	12.5	0	0	62.5
Pour Water-L-1/3（语言指令）Pour Water-L-1/3 (language instruction)	—	0	0	87.5
Pour Water-R-2/3（语言指令）Pour Water-R-2/3 (language instruction)	—	0	0	87.5
Fold Shorts（1-shot）Fold Shorts (1-shot)	0	0	4	62.5
Handover · 总体（5-shot）Handover · overall (5-shot)	88 (pickup)	84 (pickup)	100 (pickup)	100 (pickup)
Robot Dog（灵巧操作）Robot Dog (dexterous manipulation)	32	0	0	48

消融实验（Table 2，成功率 %）Ablation Study (Table 2, success rate %)

变体Variant	未见物体Unseen object	未见场景Unseen scene	语言指令跟随Language instruction following
RDT (regress) — 去掉扩散，改为回归RDT (regress) — diffusion removed, replaced by regression	12.5	50	12.5
RDT (small) — 小参数量模型RDT (small) — model with fewer parameters	37.5	62.5	25
RDT (scratch) — 无预训练，从头训练RDT (scratch) — no pre-training, trained from scratch	0	25	62.5
RDT (ours)	50	62.5	100

消融结果表明："diffusion modeling, large model size, and large data size all contribute to superior performance"—— 三者缺一不可。去掉扩散建模后，语言指令跟随从 100% 骤降至 12.5%；去掉大规模预训练后，未见物体任务成功率从 50% 降至 0%。The ablation results show: “diffusion modeling, large model size, and large data size all contribute to superior performance”—none of the three can be dispensed with. Once diffusion modeling is removed, language instruction following plunges from 100% to 12.5%; once large-scale pre-training is removed, the success rate on the unseen-object task drops from 50% to 0%.

消融实验结果 — **Figure 4.** (a) 去掉 QKNorm 和 RMSNorm 后，大规模预训练的损失曲线出现严重不稳定甚至爆炸。 (b) 去掉 MLP 解码器或 ACI 后的任务成功率对比，验证了两项设计的必要性。(a) After removing QKNorm and RMSNorm, the loss curve of large-scale pre-training becomes severely unstable and even explodes. (b) Comparison of task success rates after removing the MLP decoder or ACI, which verifies the necessity of both designs.

04 局限性Limitations

Note：论文未包含显式的 Limitations 章节。以下各项：标注"（作者陈述）"的源于论文正文，标注"（设计推断）"的为审稿人/读者从方法设计中合理推导。Note: the paper contains no explicit Limitations section. Among the items below, those marked “(author statement)” come from the main text of the paper, while those marked “(design inference)” are reasonable inferences drawn from the method design by a reviewer/reader.

仍需针对目标机器人进行微调（设计推断）Fine-tuning on the target robot is still required (design inference)

尽管 RDT 在海量多机器人数据上预训练，但在 ALOHA 上的评测仍依赖 6K+ 条目标平台演示数据进行微调。零样本跨平台部署（不经任何微调）的可行性尚未验证，存在一定的 embodiment gap。Although RDT is pre-trained on massive multi-robot data, its evaluation on ALOHA still relies on fine-tuning with 6K+ demonstrations collected on the target platform. The feasibility of zero-shot cross-platform deployment (without any fine-tuning) has not been verified, and a certain embodiment gap remains.

统一动作空间需人工映射（设计推断）The unified action space requires manual mapping (design inference)

物理可解释统一动作空间虽然有效，但对于新机器人平台，仍需手动设计从其原生动作空间到统一空间的映射规则。这一过程需要机器人专业知识，限制了"即插即用"的可扩展性。The physically interpretable unified action space is effective, but for a new robot platform one still has to hand-design the rules that map its native action space into the unified space. This process requires robotics expertise, which limits “plug-and-play” scalability.

数据采集成本高（作者陈述）High data collection cost (author statement)

论文指出双臂操作数据 "scarcity" 是核心挑战。尽管 RDT 通过大规模预训练缓解了数据稀缺问题，但收集 6K+ 高质量双臂演示仍需大量人力成本。论文使用 GPT-4-Turbo 增强语言多样性，但并未解决物理演示数据的采集瓶颈。The paper points out that the “scarcity” of bimanual manipulation data is a core challenge. Although RDT alleviates data scarcity through large-scale pre-training, collecting 6K+ high-quality bimanual demonstrations still demands a great deal of human labor. The paper uses GPT-4-Turbo to enrich language diversity, but it does not solve the bottleneck of collecting physical demonstration data.

推理延迟（设计推断）Inference latency (design inference)

即使使用 DPM-Solver++ 将扩散步数减至 5 步并达到 6 Hz chunk 频率，对于需要极高实时反应速度的任务（如动态抓取运动物体）仍可能存在延迟问题。经典回归策略（如 ACT）在推理速度上仍具优势。Even when DPM-Solver++ reduces the diffusion steps to 5 and reaches a 6 Hz chunk frequency, latency may still be a problem for tasks that require extremely fast real-time reactions (such as dynamically grasping a moving object). Classical regression policies (such as ACT) still hold an advantage in inference speed.