Robot Learning with Sensorimotor Pre-training

01 动机 MotivationMotivation

自监督预训练在视觉和语言领域带来了巨大突破，但在机器人学习领域尚未得到充分探索。机器人操作需要同时处理多模态感知（图像、本体感知）和动作序列，直接从零训练样本效率低、难以在任务间迁移。Self-supervised pre-training has produced major breakthroughs in vision and language, yet it remains under-explored in robot learning. Robotic manipulation has to handle multimodal perception (images, proprioception) and action sequences at the same time; training directly from scratch is sample-inefficient and transfers poorly between tasks.

"We ask: can we learn good sensorimotor representations from robotic trajectories? Our key hypothesis is that if the robot can predict the missing content it has acquired a good model of the physical world."

在机器人操作这一数据稀缺场景中，预训练能否帮助模型学到可迁移的物理世界模型？这是 RPT 试图回答的核心问题。与视觉预训练（ViT、MAE）不同，RPT 预训练的对象是跨模态、跨时间步的感知运动序列，旨在捕捉动作与感知之间的因果依赖关系。In robotic manipulation, a data-scarce setting, can pre-training help the model learn a transferable model of the physical world? This is the core question RPT sets out to answer. Unlike visual pre-training (ViT, MAE), RPT pre-trains on sensorimotor sequences that span modalities and time steps, aiming to capture the causal dependencies between action and perception.

02 方法 MethodMethod

RPT 分为两个阶段：（1）在大规模真实轨迹上进行感知运动掩码预训练；（2）以 behavior cloning fine-tune 到特定下游任务。整个流水线无需任务标签或语言监督，仅依赖机器人自身采集的轨迹数据。RPT has two stages: (1) sensorimotor masked pre-training on large-scale real trajectories; (2) fine-tuning to a specific downstream task with behavior cloning. The whole pipeline requires no task labels or language supervision, relying only on trajectory data collected by the robot itself.

Sensorimotor Tokenization（感知运动 Token 化）Sensorimotor Tokenization

每个时间步包含三个模态：多视角图像、本体感知状态（关节角度等）、动作。图像经预训练视觉编码器（ViT）提取 latent 表示，与状态、动作分别线性映射为固定维度 token，按时间步交错拼接为序列输入 Transformer。这一设计将视觉编码器从感知运动上下文长度中解耦，使模型可使用 10× 更大的 context 而不增加视觉计算量。Each time step contains three modalities: multi-view images, proprioceptive state (joint angles, etc.) and action. Images pass through a pre-trained visual encoder (ViT) to yield latent representations, which — like states and actions — are linearly mapped to fixed-dimensional tokens and interleaved by time step into a sequence fed to the Transformer. This design decouples the visual encoder from the sensorimotor context length, letting the model use a 10× larger context without increasing visual computation.

"We predict in the latent representation space rather than the pixel space, which enables 10 times larger models."

Masked Sensorimotor Prediction（掩码预训练目标）Masked Sensorimotor Prediction (Pre-training Objective)

对跨模态、跨时间步的 tokens 进行随机掩码（默认比例 0.9），训练模型从可见 tokens 预测被掩盖内容。对掩码 tokens，模型基于每个 token 的 hidden state 预测对应的原始值；对可见 tokens，直接重构原始输入以鼓励跨模态信息整合。掩码策略同时覆盖所有模态和时间步，迫使模型理解感知与动作之间的时序因果关系。Tokens across modalities and time steps are randomly masked (default ratio 0.9), and the model is trained to predict the masked content from the visible tokens. For masked tokens, the model predicts the corresponding original value from each token's hidden state; for visible tokens, it directly reconstructs the original input so as to encourage cross-modal information integration. The masking strategy covers all modalities and all time steps at once, forcing the model to understand the temporal causal relationship between perception and action.

Fine-tuning via Behavior Cloning（行为克隆微调）Fine-tuning via Behavior Cloning

预训练完成后，将 Transformer 作为特征提取骨干，顶部接一线性层预测 10 步动作（自回归推理）。以 behavior cloning 在下游演示数据上微调，可选冻结或解冻视觉编码器。推理时模型以 10 Hz 运行在真实机器人，context 保持 300 tokens。After pre-training, the Transformer serves as a feature-extraction backbone, with a linear layer on top predicting 10 action steps (autoregressive inference). It is fine-tuned with behavior cloning on downstream demonstration data, optionally freezing or unfreezing the visual encoder. At inference the model runs at 10 Hz on the real robot, keeping a context of 300 tokens.

03 实验 ExperimentsExperiments

实验在 7-DoF 机械臂上进行，任务包括 Pick（单物体抓取）、Destack（拆叠）、Stack（堆积），涵盖物体位姿、形状、外观变化。评估指标为真实机器人执行成功率（%），与从零训练基线对比。Experiments are run on a 7-DoF robotic arm; the tasks are Pick (grasping a single object), Destack (unstacking) and Stack (stacking), covering variation in object pose, shape and appearance. The evaluation metric is the execution success rate (%) on the real robot, compared against a train-from-scratch baseline.

Scaling Studies（缩放性质）Scaling Studies (Scaling Properties)

• 视觉编码器缩放：使用在 Ego4D 上预训练的 ViT（从 ViT-S 到 ViT-H），更大编码器带来持续收益。Visual encoder scaling: using ViTs pre-trained on Ego4D (from ViT-S to ViT-H), larger encoders bring steady gains.
• Context 长度缩放：context 从 1 到 300 tokens，更长上下文持续改善性能（300 tokens ≈ 30 时间步）。Context length scaling: going from 1 to 300 tokens, a longer context steadily improves performance (300 tokens ≈ 30 time steps).
• 预训练数据量缩放：从 480 到 20,000 条轨迹，数据越多性能越高，体现良好的 data scaling。Pre-training data scaling: going from 480 to 20,000 trajectories, more data yields higher performance, showing good data scaling.

跨任务与跨机器人迁移 TransferCross-task and Cross-robot Transfer

论文还验证了两种迁移场景：The paper further validates two transfer settings:

• Cross-task transfer（跨任务）：在 xArm 机器人上，以其他任务数据预训练再 fine-tune 到 Stack，优于从零训练基线（成功率提升 ~24%）。Cross-task transfer: on the xArm robot, pre-training on data from other tasks and then fine-tuning to Stack beats the train-from-scratch baseline (success rate up by ~24%).
• Cross-robot transfer（跨机器人）：在 Franka 机器人（训练）→ xArm 机器人（部署），预训练模型明显优于从零训练，展示跨机器人平台的泛化能力。Cross-robot transfer: from the Franka robot (training) to the xArm robot (deployment), the pre-trained model clearly outperforms training from scratch, demonstrating generalization across robot platforms.

Ablations 消融实验Ablations

• 掩码比例（Masking Ratio）：0.9 的高掩码比例最优（从 0.1 到 0.9 逐渐提升）；感知运动预训练需要更高掩码比例才能学到有效表示。Masking Ratio: a high masking ratio of 0.9 is optimal (performance rises gradually from 0.1 to 0.9); sensorimotor pre-training needs a higher masking ratio to learn effective representations.
• 掩码策略（Masking Type）：统一随机掩码优于仅掩码某一模态或某一时间步；跨模态跨时序联合掩码对性能至关重要。Masking Type: uniform random masking beats masking only one modality or only one time step; joint masking across modalities and across time is critical for performance.
• Partial vs. Full fine-tuning：更多演示数据下，全量 fine-tune 略优；少量演示下 partial fine-tuning 更鲁棒。Partial vs. Full fine-tuning: with more demonstration data, full fine-tuning is slightly better; with few demonstrations, partial fine-tuning is more robust.
• Linear probing：线性探针无法取得非平凡性能，说明下游任务需要特征适应（fine-tuning 必要）。Linear probing: linear probes cannot reach non-trivial performance, showing that downstream tasks require feature adaptation (fine-tuning is necessary).

04 局限性 LimitationsLimitations