GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
面向通用仿人机器人的开放基础模型An Open Foundation Model for Generalist Humanoid Robots
NVIDIA GR00T Team · Linxi "Jim" Fan, Yuke Zhu (项目主导) · 共 41 位研究者 · arXiv:2503.14734NVIDIA GR00T Team · Linxi "Jim" Fan, Yuke Zhu (project leads) · 41 researchers in total · arXiv:2503.14734
GR00T N1 是 NVIDIA 发布的首个面向通用仿人机器人的开放 Vision-Language-Action (VLA) 基础模型。它采用受认知科学启发的双系统架构：System 2（VLM）负责语言理解与环境推理，System 1（Diffusion Transformer）负责生成低延迟连续动作。模型在仿真和真实 Fourier GR-1 人形机器人上均超越了当前最先进的模仿学习基线。GR00T N1 is the first open Vision-Language-Action (VLA) foundation model for generalist humanoid robots released by NVIDIA. It adopts a dual-system architecture inspired by cognitive science: System 2 (the VLM) handles language understanding and reasoning about the environment, while System 1 (a Diffusion Transformer) generates low-latency continuous actions. On both simulation and the real Fourier GR-1 humanoid robot, the model surpasses state-of-the-art imitation learning baselines.
2025-03-18 · arXiv:2503.147342.2B 参数 · Eagle-2 VLM + Diffusion Transformer2.2B parameters · Eagle-2 VLM + Diffusion TransformerCC BY 4.0 开放发布CC BY 4.0 open release📄 arXiv:2503.14734PDF
仿人机器人凭借与人类相近的身体结构，被认为是最适合在人类环境中完成通用任务的载体。然而，现有机器人学习系统要么缺乏语言理解能力，要么依赖单一数据源，无法跨机型和任务迁移——这是制约通用仿人机器人落地的根本瓶颈。GR00T N1 的目标是构建第一个覆盖多机型、多任务、多数据源的开放式机器人基础模型。Because their bodies resemble the human form, humanoid robots are regarded as the platform best suited to performing general-purpose tasks in human environments. Existing robot learning systems, however, either lack language understanding or rely on a single data source, and therefore cannot transfer across embodiments and tasks — the fundamental bottleneck that holds back the deployment of generalist humanoid robots. GR00T N1 aims to build the first open robot foundation model that spans multiple embodiments, tasks and data sources.
"We introduce GR00T N1, a generalist model for humanoid robots. GR00T N1 adopts a dual-system architecture inspired by cognitive science—a vision-language model as the high-level reasoning (System 2) and a diffusion transformer as the low-level action (System 1)."
数据金字塔（Data Pyramid）。GR00T N1 的训练语料从底至顶分三层：底层为大规模网络数据与人类视频，中层为物理仿真生成的合成数据（神经轨迹），顶层为真实机器人遥操作数据。顶层数据最稀缺也最关键，神经轨迹生成技术可将遥操作数据量放大约 10 倍。Data Pyramid. The training corpus of GR00T N1 is organized into three layers from bottom to top: large-scale web data and human videos at the base, synthetic data generated by physics simulation (neural trajectories) in the middle, and real-robot teleoperation data at the top. The top layer is the scarcest and the most critical; neural trajectory generation can enlarge the volume of teleoperation data by roughly 10×.
45.0%仿真平均成功率（vs 基线 33.4%）Average success rate in simulation (vs. 33.4% for the baseline)
76.8%真实机器人全数据平均成功率Average success rate on the real robot with the full dataset
2.2B参数（1.34B 在 VLM 中）parameters (1.34B of them in the VLM)
63.9msL40 GPU 上单步推理延迟Single-step inference latency on an L40 GPU
02方法 MethodMethod
GR00T N1 的核心是双系统 VLA 架构：高层 System 2 模块为预训练 Vision-Language Model（基于 Eagle-2，10Hz 频率运行），负责语言理解与视觉推理；低层 System 1 模块为 Diffusion Transformer（基于 flow matching 训练，K=4 推理步），负责生成连续的电机动作序列。两个模块在训练阶段联合优化，共享中间表示。训练数据来自异构混合来源：遥操作真实数据、人类视频和合成生成数据（神经轨迹）。At the core of GR00T N1 is a dual-system VLA architecture: the high-level System 2 module is a pre-trained Vision-Language Model (based on Eagle-2, running at 10Hz) in charge of language understanding and visual reasoning; the low-level System 1 module is a Diffusion Transformer (trained with flow matching, K=4 inference steps) in charge of generating continuous motor action sequences. The two modules are optimized jointly during training and share intermediate representations. The training data comes from heterogeneous mixed sources: real teleoperation data, human videos and synthetically generated data (neural trajectories).
GR00T N1 模型概览（Fig. 2）。左侧 System 2（VLM）接收语言指令和图像输入，输出高层语义表征；右侧 System 1（Diffusion Transformer）将该表征与当前本体感觉（proprioception）融合，通过 flow matching 去噪生成动作序列。两个模块通过共享 token 紧耦合，联合端到端训练。Overview of the GR00T N1 model (Fig. 2). On the left, System 2 (the VLM) receives the language instruction and image inputs and outputs a high-level semantic representation; on the right, System 1 (the Diffusion Transformer) fuses that representation with the current proprioception and denoises it through flow matching into an action sequence. The two modules are tightly coupled via shared tokens and trained jointly end-to-end.模型架构细节（Fig. 3）。模型针对多种机器人形态（embodiment）设计了各自的 MLP 动作编码器，将异构动作空间映射到统一嵌入维度。Diffusion Transformer 的输出经过反向去噪后得到最终动作。训练时同时覆盖双臂协调操作、手部精细操作等多种任务类型。Architecture details (Fig. 3). A dedicated MLP action encoder is designed for each robot embodiment, mapping heterogeneous action spaces into a unified embedding dimension. The output of the Diffusion Transformer yields the final actions after reverse denoising. Training simultaneously covers task types such as bimanual coordinated manipulation and dexterous in-hand manipulation.
System 2 为预训练 VLM（Eagle-2），在推理阶段以 10Hz 频率运行，将图像 + 语言指令编码为高层语义 token。System 1 为 Diffusion Transformer，采用 flow matching 训练范式，推理时仅需 K=4 步去噪即可生成动作序列，在 L40 GPU（bf16 精度）上推理延迟仅 63.9ms。两个系统通过共享中间 token 紧耦合，联合训练以实现端到端优化。System 2 is a pre-trained VLM (Eagle-2) that runs at 10Hz during inference, encoding images plus language instructions into high-level semantic tokens. System 1 is a Diffusion Transformer trained under the flow matching paradigm; at inference it needs only K=4 denoising steps to generate an action sequence, with an inference latency of merely 63.9ms on an L40 GPU (bf16 precision). The two systems are tightly coupled through shared intermediate tokens and trained jointly for end-to-end optimization.
为弥补真实遥操作数据的稀缺，论文提出利用离线视频生成模型（video generation models）将 88 小时的遥操作视频扩展为约 827 小时的神经轨迹视频——约 10× 放大。神经轨迹生成共消耗约 105k L40 GPU 小时（跨 3600 块 GPU 并行）。仿真侧则在 11 小时内生成了等效于 6,500 小时的仿真轨迹数据，大幅降低了人工遥操作成本。To make up for the scarcity of real teleoperation data, the paper proposes using offline video generation models to expand 88 hours of teleoperation video into roughly 827 hours of neural trajectory video — about a 10× enlargement. Generating the neural trajectories consumed about 105k L40 GPU hours in total (parallelized across 3600 GPUs). On the simulation side, trajectory data equivalent to 6,500 hours was generated within 11 hours, sharply reducing the cost of manual teleoperation.
潜在动作表征（Latent Actions）Latent Actions
针对无动作标注的人类视频，模型学习跨机型共享的潜在动作嵌入（latent action embeddings），通过检索相似嵌入实现跨机型知识迁移，使人类视频也能有效参与训练，拓宽了训练数据来源。For human videos without action labels, the model learns latent action embeddings shared across embodiments and transfers knowledge across embodiments by retrieving similar embeddings, so that human videos can also participate effectively in training, broadening the sources of training data.
03实验 ExperimentsExperiments
实验在仿真和真实机器人两个层面展开。仿真使用 RoboCasa、DexMG、GR-1 三个开源基准，各基准提供 100 条示范轨迹；真实机器人部署在 Fourier GR-1 仿人机器人上，覆盖拾取放置、关节操作、工业场景、双臂协调四类任务。基线方法包括 BC-Transformer 和 Diffusion Policy。Experiments are carried out at two levels: simulation and the real robot. Simulation uses three open-source benchmarks, RoboCasa, DexMG and GR-1, each providing 100 demonstration trajectories; the real-robot setting is deployed on the Fourier GR-1 humanoid robot and covers four task families: pick-and-place, articulated-object manipulation, industrial scenarios and bimanual coordination. The baselines include BC-Transformer and Diffusion Policy.
在仅使用 10% 真实数据时，GR00T-N1-2B 的平均成功率（42.6%）已接近 Diffusion Policy 使用全量数据的效果（46.4%），体现了强大的数据效率优势（data efficiency）。Using only 10% of the real data, GR00T-N1-2B already reaches an average success rate (42.6%) close to that of Diffusion Policy trained on the full dataset (46.4%), which reflects a strong data efficiency advantage.
真实世界任务（Fig. 8）。所有图像均来自 GR00T-N1-2B 策略的真实 rollout 录像，覆盖拾取放置（pick-and-place）、关节操作（articulated object）、工业操作（industrial）、双臂协调（bimanual coordination）四类场景，均部署在 Fourier GR-1 仿人机器人上。Real-world tasks (Fig. 8). All images come from recordings of real rollouts of the GR00T-N1-2B policy, covering four kinds of scenarios — pick-and-place, articulated object, industrial and bimanual coordination — all deployed on the Fourier GR-1 humanoid robot.仿真平均成功率（Fig. 10）。在三个仿真基准（RoboCasa、DexMG、GR-1）上，GR00T-N1-2B 平均成功率为 45.0%，优于 BC-Transformer（26.4%）和 Diffusion Policy（33.4%）。Average success rate in simulation (Fig. 10). On the three simulation benchmarks (RoboCasa, DexMG, GR-1), GR00T-N1-2B attains an average success rate of 45.0%, outperforming BC-Transformer (26.4%) and Diffusion Policy (33.4%).
神经轨迹消融（Ablations）Neural Trajectory Ablations
在 RoboCasa 仿真基准上的消融实验表明，加入神经轨迹增强后，在不同数据量级下成功率分别提升 +4.2%、+8.8%、+6.8%，验证了神经轨迹对策略学习的持续正向贡献，且在低数据量（low-data）场景下效果尤为显著。预训练评估方面，在协调双臂任务上成功率达 76.6%（11.5/15），新颖物体操作达 73.3%（11/15）。Ablations on the RoboCasa simulation benchmark show that adding neural trajectory augmentation raises the success rate by +4.2%, +8.8% and +6.8% respectively at different data scales, confirming the consistently positive contribution of neural trajectories to policy learning, with the effect especially marked in the low-data regime. For the pre-training evaluation, the success rate reaches 76.6% (11.5/15) on coordinated bimanual tasks and 73.3% (11/15) on novel object manipulation.
04局限性 LimitationsLimitations
Note:以下局限性均为论文作者明确陈述（stated），原文见论文结论与讨论部分。All the limitations below are explicitly stated by the authors; the original wording is found in the conclusion and discussion sections of the paper.
专注于短时域桌面操作，尚未扩展至长时域移动操作Focused on short-horizon tabletop manipulation, not yet extended to long-horizon loco-manipulation
"GR00T N1 model focuses primarily on short-horizon tabletop manipulation tasks. In future work, we aim to extend its capabilities to tackle long-horizon loco-manipulation, which will require advancements in humanoid hardware, model architecture, and training corpora."
合成数据生成在物理真实性与多样性上仍有瓶颈Synthetic data generation still faces bottlenecks in physical realism and diversity
"existing methods still face challenges in generating diverse and counterfactual data, while adhering to the laws of physics, limiting the quality and variability of synthetic datasets." 当前视频生成模型无法保证精确的物理一致性，合成数据可能引入分布偏移。Current video generation models cannot guarantee exact physical consistency, so synthetic data may introduce distribution shift.
神经轨迹生成计算代价高昂Neural trajectory generation is computationally expensive
生成约 827 小时的神经轨迹视频需要消耗约 105k L40 GPU 小时（跨 3600 块 GPU），对算力资源要求极高，限制了该方法在资源受限场景下的推广。Generating roughly 827 hours of neural trajectory video requires about 105k L40 GPU hours (across 3600 GPUs); the demand on compute resources is extremely high, which limits the wider use of the method in resource-constrained settings.
真实部署仅在单一机型（Fourier GR-1）上验证Real-world deployment validated on only a single embodiment (Fourier GR-1)
尽管模型设计支持多机型，但真实机器人实验均在 Fourier GR-1 上完成，跨机型迁移的实际效果有待进一步验证（inferred from the design）。Although the model is designed to support multiple embodiments, all real-robot experiments were completed on the Fourier GR-1, so the practical effectiveness of cross-embodiment transfer remains to be further verified (inferred from the design).
