HPT: Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers

01 动机Motivation

机器人学习领域面临一个核心挑战：不同机器人形态（embodiment）拥有各异的硬件结构、传感器配置和动作空间，导致现有方法通常为特定形态和任务单独收集数据、单独训练模型，难以实现跨形态的知识复用和泛化。Robot learning faces a core challenge: different robot embodiments have distinct hardware structures, sensor configurations and action spaces, so existing methods typically collect data and train models separately for each specific embodiment and task, which makes cross-embodiment knowledge reuse and generalization hard to achieve.

"We want to pre-train task-agnostic and embodiment-agnostic foundational models that can map raw sensor signals from individual embodiments into a shared latent space."

现有的大规模机器人数据集（如 Open-X-Embodiment）汇集了多种形态，但由于形态异质性（heterogeneity），简单地将所有数据混合训练往往效果不佳。本文的核心问题是：能否设计一种通用预训练框架，使得跨形态、跨任务的数据能够真正互补，而非相互干扰？Existing large-scale robot datasets (such as Open-X-Embodiment) aggregate many embodiments, but because of embodiment heterogeneity, naively mixing all the data for training often works poorly. The central question of this paper is: can a general pre-training framework be designed so that cross-embodiment, cross-task data genuinely complement rather than interfere with one another?

02 方法Method

HPT 将策略神经网络拆分为三个模块：形态专属的 stem（输入对齐层）、可共享的大型 trunk（Transformer 主干）、以及任务专属的 head（动作输出层）。预训练阶段共享 trunk，迁移时仅微调 head 或全部参数。HPT splits the policy network into three modules: an embodiment-specific stem (input alignment layer), a shareable large trunk (Transformer backbone), and a task-specific head (action output layer). The trunk is shared during pre-training; at transfer time only the head, or all parameters, are fine-tuned.

Stem：本体感知与视觉 Token 化Stem: proprioceptive and visual tokenization

对于形态 k，proprioceptive tokenizer 将任意维度的本体感知序列（关节角度、末端位姿等）映射为 N_p=16 个固定维度 token：先用 MLP 映射到特征空间，再施加 sinusoidal 位置编码，通过 cross-attention 将特征压缩到 16 个可学习 query token 上。Vision tokenizer 则先用冻结的 ResNet-18 提取图像特征，再同样通过 attention 映射到 16 个 token。两组 token 拼接后形成 32 个输入 token 送入 trunk。For embodiment k, the proprioceptive tokenizer maps a proprioceptive sequence of arbitrary dimension (joint angles, end-effector poses, etc.) into N_p=16 tokens of fixed dimension: an MLP first projects it into a feature space, sinusoidal positional encoding is then applied, and cross-attention compresses the features onto 16 learnable query tokens. The vision tokenizer first extracts image features with a frozen ResNet-18, then likewise maps them to 16 tokens through attention. The two groups of tokens are concatenated into the 32 input tokens fed to the trunk.

Trunk：共享 Transformer 主干Trunk: the shared Transformer backbone

Trunk 是标准的 Transformer encoder，提供五种规格的参数量：The trunk is a standard Transformer encoder, provided in five parameter-count configurations:

预训练阶段，trunk 对所有数据集共享权重，通过 switch 机制在同一个 batch 中激活不同形态的 stem/head 对，实现真正的异构联合训练。迁移时，trunk 初始化来自预训练权重，head 重新初始化后再端到端微调。During pre-training the trunk shares weights across all datasets, and a switch mechanism activates the stem/head pairs of different embodiments within the same batch, realizing genuinely heterogeneous joint training. At transfer time the trunk is initialized from the pre-trained weights, while the head is re-initialized before end-to-end fine-tuning.

预训练数据规模Pre-training data scale

默认设置使用 27 个 RT-X 数据集（16k 轨迹，5M 样本，batch size 256）。大规模设置扩展到 52 个数据集（270k 轨迹，155M 样本，batch size 2048），涵盖 42 个真实机器人数据集、7 个仿真数据集、3 个人类视频数据集和 1 个已部署机器人数据集。The default setting uses 27 RT-X datasets (16k trajectories, 5M samples, batch size 256). The large-scale setting extends to 52 datasets (270k trajectories, 155M samples, batch size 2048), covering 42 real-robot datasets, 7 simulation datasets, 3 human video datasets and 1 deployed-robot dataset.

03 实验Experiments

实验在仿真和真实机器人两条线上展开：仿真使用 Meta-world、RoboMimic、Fleet-Tools 和 Simpler（Google EDR 机器人）基准；真实机器人实验在接触丰富型操作任务上与多个基线对比。The experiments unfold along two lines, simulation and real robots: simulation uses the Meta-world, RoboMimic, Fleet-Tools and Simpler (Google EDR robot) benchmarks, while the real-robot experiments compare against several baselines on contact-rich manipulation tasks.

仿真迁移学习Transfer learning in simulation

在 Meta-world、RoboMimic、Fleet-Tools 三个仿真基准上，HPT 预训练权重经微调后相比从头训练的基线，在未见任务上成功率平均提升超过 20%。在 Simpler 基准上，HPT-Base 与 Octo、RT1-X、RT2-X 表现相当，验证了跨形态预训练的迁移能力。On the three simulation benchmarks Meta-world, RoboMimic and Fleet-Tools, HPT pre-trained weights after fine-tuning raise the average success rate by more than 20% on unseen tasks compared with train-from-scratch baselines. On the Simpler benchmark, HPT-Base performs on par with Octo, RT1-X and RT2-X, confirming the transferability of cross-embodiment pre-training.

真实机器人实验Real-robot experiments

在真实机器人操作任务上（Sweep Leftover 等接触丰富型任务），HPT 显著优于从头训练的基线：On real-robot manipulation tasks (contact-rich tasks such as Sweep Leftover), HPT clearly outperforms train-from-scratch baselines:

规模化消融（Scaling Laws）Scaling ablations (Scaling Laws)

实验表明 HPT 呈现出良好的 scaling 特性：The experiments show that HPT exhibits favorable scaling behavior:

• 数据规模：从 16k 轨迹扩展到 270k 轨迹，验证集 loss 持续下降，模型持续受益于更多数据。Data scale: going from 16k trajectories to 270k trajectories, the validation loss keeps decreasing and the model keeps benefiting from more data.
• 模型规模：从 HPT-Small（3.1M）扩展到 HPT-Huge（1.1B），性能随参数量增长单调提升。Model scale: going from HPT-Small (3.1M) to HPT-Huge (1.1B), performance improves monotonically with parameter count.
• 数据多样性：加入仿真数据集和人类视频数据进行联合预训练，进一步提升下游任务迁移效果。Data diversity: adding simulation datasets and human video data to the joint pre-training further improves downstream task transfer.
• 本体感知对齐：消融实验显示，仅使用视觉而不加入本体感知信息会导致成功率明显下降（No Prop. Finetuned vs. HPT-Base：63.3% vs. 70.0%），表明本体感知 token 化是 HPT 的关键设计。Proprioception alignment: ablations show that using vision alone without proprioceptive input markedly lowers the success rate (No Prop. Finetuned vs. HPT-Base: 63.3% vs. 70.0%), indicating that proprioceptive tokenization is a key design of HPT.

04 局限性Limitations