LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestion

01 动机Motivation

现有机器人基础模型主要依赖行为克隆（behavior cloning），无法充分利用异质具身数据中蕴含的可迁移动力学知识；统一世界模型（Unified World Model）虽然有潜力，却受限于"coarse data usage and fragmented datasets"。如何将数十万小时的人类视频、低质量轨迹与高质量机器人轨迹统一纳入训练，是提升可扩展性的核心挑战。Existing robot foundation models rely primarily on behavior cloning, which fails to exploit the transferable dynamics knowledge embedded in heterogeneous embodied data; the Unified World Model is promising yet constrained by "coarse data usage and fragmented datasets". Unifying hundreds of thousands of hours of human video, low-quality trajectories and high-quality robot trajectories into a single training recipe is the core challenge for scalability.

"Existing methods primarily rely on behavior cloning, which discards transferable dynamics knowledge embedded in heterogeneous embodied data."

LDA-1B overview — 图 1：LDA-1B 总览。模型对策略学习（policy learning）、前向动力学（forward dynamics）、逆向动力学（inverse dynamics）和视觉预测（visual forecasting）进行联合训练，通过任务嵌入（task embedding）区分不同质量与类型的数据。Figure 1: Overview of LDA-1B. The model is jointly trained on policy learning, forward dynamics, inverse dynamics and visual forecasting, and uses a task embedding to distinguish data of different quality and type.

+21%contact-rich 操作相对 π0.5 提升gain on contact-rich manipulation over π0.5

+48%灵巧操作相对 π0.5 提升gain on dexterous manipulation over π0.5

+23%长视野任务相对 π0.5 提升gain on long-horizon tasks over π0.5

30k+EI-30k 数据集小时数hours in the EI-30k dataset

task overview — 图：真实世界评测任务概览，涵盖 Pick & Place、Contact-rich、精细操作（Fine）、长视野（Long-horizon）以及灵巧手（Dexterous）五类任务，横跨 Galbot G1 双臂夹爪机器人与 Unitree G1 灵巧手平台。Figure: Overview of the real-world evaluation tasks, covering five categories — Pick & Place, Contact-rich, Fine manipulation, Long-horizon and Dexterous — across the Galbot G1 dual-arm gripper robot and the Unitree G1 dexterous-hand platform.

02 方法Method

LDA-1B 的核心是一个多模态扩散 Transformer（MM-DiT），对动作序列与未来视觉 latent 联合去噪，配合通用具身数据摄取（Universal Embodied Data Ingestion）框架，将不同质量与模态的数据统一分配训练目标。At the core of LDA-1B is a multimodal diffusion Transformer (MM-DiT) that jointly denoises the action sequence and the future visual latent, paired with a Universal Embodied Data Ingestion framework that assigns training objectives to data of differing quality and modality.

MM-DiT architecture — 图 2：LDA 架构。以冻结的 Qwen3-VL-4B-Instruct 提取 VLM token、冻结的 DINOv3-ViT-s 提供视觉 latent，MM-DiT 通过共享 self-attention 层使动作专家（action expert）与视觉专家（visual expert）交互，同时完成策略学习、前向 / 逆向动力学和视觉预测四项目标的联合训练。Figure 2: The LDA architecture. A frozen Qwen3-VL-4B-Instruct extracts VLM tokens and a frozen DINOv3-ViT-s supplies visual latents; through shared self-attention layers the MM-DiT lets the action expert and the visual expert interact, jointly training the four objectives of policy learning, forward and inverse dynamics, and visual forecasting.

通用具身数据摄取（Universal Embodied Data Ingestion）Universal Embodied Data Ingestion

框架将异质数据划分为三个角色：无动作人类视频监督视觉预测（visual forecasting）；低质量轨迹提供动力学监督；高质量轨迹同时支持策略学习与动力学学习。Register token 充当缺失模态的占位符，使模型能在统一架构下处理不完整输入。此外，论文构建了 EI-30k 数据集——8,030 小时真实机器人数据 + 8,600 小时仿真数据 + 7,200 小时有动作人类示范 + 10,000 小时无动作人类视频，总计超 30,000 小时，全部以 LeRobot 格式标准化并手动对齐末端执行器坐标系。The framework assigns heterogeneous data to three roles: action-free human video supervises visual forecasting; low-quality trajectories provide dynamics supervision; high-quality trajectories support both policy learning and dynamics learning. Register tokens serve as placeholders for missing modalities, letting the model handle incomplete inputs within a unified architecture. The paper further builds the EI-30k dataset — 8,030 hours of real-robot data + 8,600 hours of simulation data + 7,200 hours of action-labeled human demonstrations + 10,000 hours of action-free human video, more than 30,000 hours in total, all standardized in the LeRobot format with manually aligned end-effector coordinate frames.

图 4：EI-30k 数据集统计。四类数据来源分布与片段时长直方图，覆盖真实机器人、仿真、有动作和无动作人类视频。Figure 4: EI-30k dataset statistics. Distribution over the four data sources and a histogram of clip durations, covering real robots, simulation, and action-labeled and action-free human video.

DINO Latent Space 用于视觉预测DINO Latent Space for Visual Forecasting

与像素空间 VAE 不同，LDA-1B 采用 DINO 特征作为视觉预测目标，"reduce redundant appearance modeling"并避免"entangling appearance, geometry, and dynamics at low-level feature granularity"。仿真实验显示，从 VAE 切换到 DINO 表示使成功率从 20.0% 大幅提升至 55.4%。Unlike pixel-space VAEs, LDA-1B takes DINO features as the visual forecasting target, to "reduce redundant appearance modeling" and to avoid "entangling appearance, geometry, and dynamics at low-level feature granularity". In simulation, switching from the VAE to the DINO representation raises the success rate from 20.0% to 55.4%.

DINO latent dynamics visualization — 图 9：DINO latent 前向动力学可视化。预测的未来视觉表示准确捕捉了语义物体结构（如杯子位置与形状），验证了动力学模型的有效性。Figure 9: Visualization of DINO latent forward dynamics. The predicted future visual representation accurately captures semantic object structure (such as the position and shape of the cup), confirming the effectiveness of the dynamics model.

训练配置Training Setup

48 块 H800 GPU，400k 迭代，共 4,608 GPU 小时48 H800 GPUs, 400k iterations, 4,608 GPU hours in total
Batch size 32 × 48（预训练），学习率 1e-4，余弦调度Batch size 32 × 48 (pre-training), learning rate 1e-4, cosine schedule
Hidden size 1536，16 层，32 注意力头Hidden size 1536, 16 layers, 32 attention heads
视觉观测采样率 3 Hz，动作采样率 10 Hz；动作 chunk size = 16，历史帧数 = 2Visual observations sampled at 3 Hz and actions at 10 Hz; action chunk size = 16, history length = 2

图 10：数据规模分析。随着训练数据增加至 30,000 小时，动作预测误差降至 6.6，验证了 LDA-1B 的良好可扩展性。Figure 10: Data-scaling analysis. As the training data grow to 30,000 hours, the action prediction error drops to 6.6, confirming the favorable scalability of LDA-1B.

03 实验Experiments

在仿真（RoboCasa-GR1）和真实机器人（Galbot G1 夹爪 + Unitree G1 灵巧手）上与 GR00T-N1.6、GR00T-EI10k、UWM-1B、π0.5 进行对比；采用少样本微调（few-shot fine-tuning）评估迁移效率。LDA-1B is compared with GR00T-N1.6, GR00T-EI10k, UWM-1B and π0.5 in simulation (RoboCasa-GR1) and on real robots (Galbot G1 gripper + Unitree G1 dexterous hand); few-shot fine-tuning is used to assess transfer efficiency.

仿真基准：RoboCasa-GR1Simulation Benchmark: RoboCasa-GR1

模型Model	成功率（Success Rate）Success Rate
UWM-1B	19.3%
GR00T-N1.6	47.6%
GR00T-EI10k	51.3%
LDA-1B（本文）LDA-1B (ours)	55.4%

消融实验显示，将视觉 latent 从 VAE 替换为 DINO 是最关键的设计选择，成功率从 20.0% 跃升至 55.4%。Ablations show that replacing the VAE visual latent with DINO is the single most critical design choice, lifting the success rate from 20.0% to 55.4%.

真实世界抓取操作Real-World Gripper Manipulation

gripper manipulation results — 图 4：真实世界夹爪操作成功率对比（8 类任务）。LDA-1B 在 Contact-rich（+21%）、Long-horizon（+23%）类任务上超越 π0.5；"Clean the Rubbish"长视野双臂任务中 LDA-1B 达 35%，基线全部为 0%。Figure 4: Success-rate comparison on real-world gripper manipulation (8 task categories). LDA-1B surpasses π0.5 on Contact-rich (+21%) and Long-horizon (+23%) tasks; on the "Clean the Rubbish" long-horizon bimanual task LDA-1B reaches 35%, while every baseline scores 0%.

灵巧手操作Dexterous Hand Manipulation

dexterous manipulation results — 图 5：灵巧操作对比（3 低自由度 BrainCo 手 + 2 高自由度 Sharpa 手）。LDA-1B 在"Pull Nail"达 80%（vs π0.5 大幅失败），在"Flip Bread"高自由度任务达 90%（vs π0.5 仅 10%），整体领先 48%。Figure 5: Dexterous manipulation comparison (3 low-DoF BrainCo hands + 2 high-DoF Sharpa hands). LDA-1B reaches 80% on "Pull Nail" (where π0.5 largely fails) and 90% on the high-DoF "Flip Bread" task (vs only 10% for π0.5), leading by 48% overall.

泛化能力（Table III）Generalization (Table III)

测试条件Test Condition	LDA-1B 成功率LDA-1B Success Rate
未见物体 / 背景Unseen objects / backgrounds	60%
分布外（OOD）位置Out-of-distribution (OOD) positions	40%

在两项泛化测试中 LDA-1B 均显著超越基线模型。LDA-1B significantly outperforms the baselines on both generalization tests.

混合质量数据微调（Table IV）Fine-tuning on Mixed-Quality Data (Table IV)

在低质量轨迹加入后，LDA-1B 性能提升 10 个百分点；而 π0.5 在同等条件下下降 10–20 个百分点，验证了通用数据摄取框架对低质量数据的有效利用能力。Once low-quality trajectories are added, LDA-1B improves by 10 percentage points, whereas π0.5 under the same conditions drops by 10–20 percentage points, confirming that the universal data ingestion framework makes effective use of low-quality data.

目标梯度协调性Gradient Agreement Across Objectives

余弦相似度分析（gradient cosine similarity）显示，四项训练目标在 400k 步后梯度方向高度一致，表明联合训练没有导致目标冲突。A gradient cosine similarity analysis shows that the four training objectives keep highly consistent gradient directions after 400k steps, indicating that joint training causes no objective conflict.

04 局限性Limitations

注：以下局限性由论文作者明确说明，并非推断。Note: The limitations below are stated explicitly by the authors, not inferred.

固定 DINO 视觉特征的局限Limits of the fixed DINO visual features

模型依赖冻结的 DINO 视觉编码器（DINOv3-ViT-s），预训练阶段视觉表示不参与更新，"reliance on fixed DINO visual features … may constrain generalization to new visual perspectives"。未来工作计划联合学习视觉表示与潜在动力学。The model relies on a frozen DINO visual encoder (DINOv3-ViT-s), and the visual representation is not updated during pre-training: "reliance on fixed DINO visual features … may constrain generalization to new visual perspectives". Future work plans to learn the visual representation jointly with the latent dynamics.

以自我为中心（egocentric）的摄像头视角为主Predominantly egocentric camera viewpoints

EI-30k 数据集及训练任务主要使用第一人称（head-mounted 或 wrist）视角，"predominantly egocentric camera viewpoints … may constrain generalization to new visual perspectives"，对第三人称视角场景的泛化尚未充分验证。The EI-30k dataset and the training tasks mainly use first-person (head-mounted or wrist) viewpoints — "predominantly egocentric camera viewpoints … may constrain generalization to new visual perspectives" — so generalization to third-person viewpoints has not been fully validated.

计算成本较高High computational cost

预训练需要 48 块 H800 GPU 共 4,608 GPU 小时。虽然推理时只需单次前向传播，但大规模预训练的资源门槛仍较高，限制了社区复现的便利性。（此条为设计推断，论文未单独列出。）Pre-training requires 48 H800 GPUs for a total of 4,608 GPU hours. Although inference needs only a single forward pass, the resource barrier of large-scale pre-training remains high and limits convenient reproduction by the community. (This item is an inferred remark, not listed separately in the paper.)