GR-1: 大规模视频生成预训练用于视觉机器人操作GR-1: Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, Tao Kong · ByteDance Research
GR-1 是一个 GPT 风格的 Transformer 模型，通过在 Ego4D 大规模视频数据上预训练视频预测任务，再迁移至机器人操作微调，实现了远超先前方法的操作成功率和零样本场景泛化能力。GR-1 is a GPT-style Transformer model that pre-trains a video prediction task on large-scale Ego4D video data and then transfers to robot manipulation fine-tuning, achieving manipulation success rates and zero-shot scene generalization far beyond prior methods.
大规模生成预训练模型（GPT、DALL-E 等）在语言和视觉领域表现出卓越的效果，但视觉机器人操作领域尚未从中受益。核心障碍在于：机器人数据量稀少、且包含图像、状态、动作、语言等多模态信息，难以直接套用通用预训练范式。Large-scale generative pre-training models (GPT, DALL-E, etc.) have shown outstanding results in language and vision, yet visual robot manipulation has not benefited from them. The core obstacle is that robot data is scarce and contains multi-modal information such as images, states, actions and language, which makes it hard to directly apply the general pre-training paradigm.
"Inspired by video prediction models that generate future images conditioned on a sequence of video frames and languages, we observe that a robot trajectory itself contains a video sequence. Therefore, video prediction models could potentially learn from internet videos and leverage the learned knowledge to predict future images and generate robot actions."
机器人轨迹本身就是一段视频序列——这一洞察使得互联网视频预训练与机器人控制形成天然对齐：预测未来帧的能力可以直接迁移为"预测未来动作"的能力。GR-1 将视频预测作为预训练代理任务，填补了大规模预训练与机器人操作之间的鸿沟。A robot trajectory is itself a video sequence — this insight makes internet-video pre-training naturally aligned with robot control: the ability to predict future frames transfers directly into the ability to "predict future actions". GR-1 uses video prediction as the pre-training proxy task, bridging the gap between large-scale pre-training and robot manipulation.
图 1：GR-1 系统概览。左：在 Ego4D 等大规模视频数据上预训练视频预测（给定语言和历史帧，预测未来帧）。右：在机器人演示数据上微调，同时预测机器人动作和未来图像。预训练阶段积累的视觉-时序知识被高效迁移至操作任务。Figure 1: Overview of GR-1. Left: video prediction is pre-trained on large-scale video data such as Ego4D (given language and historical frames, predict future frames). Right: fine-tuning on robot demonstration data, predicting robot actions and future images at the same time. The visual-temporal knowledge accumulated during pre-training is transferred efficiently to manipulation tasks.
4.21CALVIN 平均连续完成任务数（前最优 3.06）CALVIN average number of tasks completed in a row (previous best 3.06)
85.4%零样本未见场景（ABC→D）成功率（前最优 53.3%）Zero-shot unseen-scene (ABC→D) success rate (previous best 53.3%)
79%真实机器人已见物体搬运成功率（RT-1 仅 27%）Real-robot transportation success rate on seen objects (RT-1 only 27%)
02方法Method
GR-1 以 GPT-2 为基础，设计了统一的 token 序列格式，支持"纯视频预训练"和"机器人操作微调"两阶段。模型接受语言指令、历史观测图像和机器人状态作为输入，输出机器人动作和未来图像预测。GR-1 builds on GPT-2 and designs a unified token sequence format that supports the two stages of "pure video pre-training" and "robot manipulation fine-tuning". The model takes language instructions, historical observation images and robot states as input, and outputs robot actions and future image predictions.
图 2：GR-1 编码器与解码器结构。语言编码器（CLIP，冻结）、图像编码器（MAE ViT，冻结）、机器人状态编码器（线性层）分别对各模态进行编码，送入 GPT Transformer。[OBS] token 解码为下一帧图像，[ACT] token 解码为机器人动作（手臂连续动作 + 夹爪二值动作）。Figure 2: Encoder and decoder structure of GR-1. A language encoder (CLIP, frozen), an image encoder (MAE ViT, frozen) and a robot state encoder (linear layer) encode each modality separately and feed the GPT Transformer. The [OBS] token is decoded into the next frame image and the [ACT] token into robot actions (continuous arm action + binary gripper action).
预训练阶段，输入序列格式为：(l, ot-h, [OBS], l, ot-h+1, [OBS], ..., l, ot, [OBS])，其中 l 为语言 token，o 为图像 token，[OBS] 为预测未来帧的特殊 token。Pre-training stage: the input sequence format is (l, ot-h, [OBS], l, ot-h+1, [OBS], ..., l, ot, [OBS]), where l is the language token, o the image token and [OBS] a special token used to predict the future frame.
微调阶段，在每个时间步额外插入机器人状态和 [ACT] token：(l, st-h, ot-h, [OBS], [ACT], ...)。所有 [ACT] 和 [OBS] token 均被 masked，使得其他 token 无法在注意力中看到它们，保持自回归预测的因果性。Fine-tuning stage: at every timestep the robot state and an [ACT] token are additionally inserted: (l, st-h, ot-h, [OBS], [ACT], ...). All [ACT] and [OBS] tokens are masked so that the other tokens cannot see them in attention, preserving the causality of autoregressive prediction.
两阶段训练流程Two-Stage Training Pipeline
Stage 1：视频生成预训练Stage 1: Video Generative Pre-training
数据：Ego4D 数据集 800,000 段视频片段（约 800 万帧）Data: the Ego4D dataset, 800,000 video clips (about 8000000 frames)
目标：给定语言描述和历史帧序列，预测 t+Δt 时刻的图像Objective: given a language description and a sequence of historical frames, predict the image at time t+Δt
损失：像素空间 MSE（参考 MAE 方法）Loss: MSE in pixel space (following the MAE approach)
编码器（CLIP 文字、MAE 图像）在全程保持冻结The encoders (CLIP for text, MAE for images) stay frozen throughout
视频预测：MSE（保持预训练能力，防止灾难性遗忘）Video prediction: MSE (retains the pre-training ability and prevents catastrophic forgetting)
模型规模Model Scale
GPT Transformer 共 12 层、12 个注意力头、384 隐藏维度，总参数量 195M，其中仅 46M 可训练（编码器冻结）。相比语言预训练模型，GR-1 的预训练计算成本大幅降低，同时获得了强大的视觉-时序表征。The GPT Transformer has 12 layers, 12 attention heads and a hidden size of 384, 195M parameters in total, of which only 46M are trainable (the encoders are frozen). Compared with language pre-training models, the pre-training compute cost of GR-1 is far lower, while it still acquires strong visual-temporal representations.
03实验Experiments
在模拟环境 CALVIN 基准（多任务长程操作）和真实机器人（物体搬运 + 铰接体操作）上进行评测，与 RT-1、HULC、MT-R3M、MCIL 等基线对比。Evaluation is carried out on the simulated CALVIN benchmark (multi-task long-horizon manipulation) and on a real robot (object transportation + articulated-object manipulation), and compared with baselines such as RT-1, HULC, MT-R3M and MCIL.
CALVIN 基准测试（模拟环境）CALVIN Benchmark (Simulation)
方法Method
单任务成功率 (%)Single-task success rate (%)
平均连续任务数Avg. consecutive tasks
设定Setting
MCIL
13.3
0.40
ABCD→D
MT-R3M
62.9
2.08
ABCD→D
RT-1
73.8
2.45
ABCD→D
HULC
88.9
3.06
ABCD→D
GR-1（本文）GR-1 (ours)
94.9
4.21
ABCD→D
零样本未见场景泛化（ABC→D）Zero-Shot Generalization to Unseen Scenes (ABC→D)
方法Method
单任务成功率 (%)Single-task success rate (%)
平均连续任务数Avg. consecutive tasks
HULC
53.3
0.67
GR-1（本文）GR-1 (ours)
85.4
3.06
在未见过的场景（桌面颜色、物体位置均不同）中，GR-1 的成功率是 HULC 的 1.6 倍，平均连续任务数提升 4.6 倍，充分验证了视频预训练带来的泛化能力。In unseen scenes (different table colors and object positions), the success rate of GR-1 is 1.6 times that of HULC and the average number of consecutive tasks improves by 4.6 times, fully confirming the generalization brought by video pre-training.
数据效率与语言泛化Data Efficiency and Language Generalization
仅使用 10% 训练数据Using Only 10% of the Training Data
GR-1：77.8% 成功率，平均 2.00 任务GR-1: 77.8% success rate, 2.00 tasks on average
HULC：66.8% 成功率，平均 1.11 任务HULC: 66.8% success rate, 1.11 tasks on average
预训练积累的先验知识在数据稀缺场景下尤为关键。The prior knowledge accumulated during pre-training is especially critical when data is scarce.
未见语言指令（GPT-4 生成同义表达）Unseen Language Instructions (Synonymous Phrasings Generated by GPT-4)
GR-1：76.4% 成功率GR-1: 76.4% success rate
HULC：71.5% 成功率HULC: 71.5% success rate
每个任务生成 50 条同义语言指令，CLIP 的语言泛化能力发挥关键作用。50 synonymous language instructions are generated for each task; the language generalization of CLIP plays a key role.
真实机器人实验Real-Robot Experiments
图 3：真实机器人实验。左：物体搬运任务（已见物体、未见实例、未见类别）。右：铰接体操作任务（抽屉开合）。GR-1 在所有设定下均大幅超越 RT-1 基线。Figure 3: Real-robot experiments. Left: object transportation tasks (seen objects, unseen instances, unseen categories). Right: articulated-object manipulation (opening and closing a drawer). GR-1 substantially surpasses the RT-1 baseline under every setting.
图 4：GR-1 视频预测效果。绿框：真实帧（ground truth）；蓝框：GR-1 预测帧。模型能够准确预测机械臂运动轨迹和场景变化，验证了其时序理解能力。注意：对于部分被遮挡的物体，预测细节存在缺失。Figure 4: Video prediction results of GR-1. Green boxes: ground-truth frames; blue boxes: frames predicted by GR-1. The model predicts arm motion trajectories and scene changes accurately, verifying its temporal understanding. Note: for partially occluded objects, predicted details are missing.
消融实验Ablation Studies
消融实验验证了预训练数据量和微调策略的重要性：移除视频预训练（从头训练）在 ABCD→D 上性能显著下降；仅使用部分 Ego4D 数据预训练同样使成功率降低。联合损失函数（Larm + Lgripper + Lvideo）的设计防止了微调时的灾难性遗忘，确保视频预测能力在操作任务中持续发挥作用。The ablations confirm the importance of the amount of pre-training data and of the fine-tuning strategy: removing video pre-training (training from scratch) degrades performance markedly on ABCD→D; pre-training on only part of the Ego4D data likewise lowers the success rate. The design of the joint loss (Larm + Lgripper + Lvideo) prevents catastrophic forgetting during fine-tuning and ensures that the video prediction ability keeps working in manipulation tasks.
04局限性Limitations
Note：以下局限性部分由作者在论文中直接说明，部分为从设计中合理推断（标注为 inferred）。Note: Some of the limitations below are stated directly by the authors in the paper, while others are reasonably inferred from the design (marked as inferred).
未见类别泛化能力有限（stated）Limited generalization to unseen categories (stated)
在真实机器人实验中，面对完全未见过类别的物体搬运任务，GR-1 成功率仅为 30%（RT-1 为 0%）。尽管相对提升显著，但绝对成功率仍较低，表明跨类别泛化仍是核心挑战。In the real-robot experiments, on object transportation tasks with entirely unseen categories, GR-1 reaches only a 30% success rate (RT-1 reaches 0%). Although the relative improvement is large, the absolute success rate is still low, showing that cross-category generalization remains a core challenge.
视频预测对遮挡物体的细节缺失（stated）Missing detail in video prediction for occluded objects (stated)
论文明确指出："video prediction details may be missing (e.g., occluded objects)"。当物体被机械臂或其他物体遮挡时，未来帧预测出现细节不准确的问题，可能影响依赖精细视觉反馈的操作任务。The paper states explicitly: "video prediction details may be missing (e.g., occluded objects)". When an object is occluded by the arm or by other objects, future-frame prediction becomes inaccurate in its details, which may affect manipulation tasks that rely on fine-grained visual feedback.
机器人数据量与多模态复杂性（stated）Robot data volume and multi-modal complexity (stated)
作者在引言中指出，机器人领域面临"robot data sparsity compared to vision-language data"以及"multi-modal nature of robot data (images, states, actions, language)"两大固有挑战。GR-1 的 Ego4D 预训练方案缓解了第一个问题，但两者仍是领域级别的长期挑战。In the introduction the authors point out two inherent challenges of the robotics field: "robot data sparsity compared to vision-language data" and "multi-modal nature of robot data (images, states, actions, language)". The Ego4D pre-training scheme of GR-1 alleviates the first one, but both remain long-term, field-level challenges.
编码器全程冻结的限制（inferred）Encoders kept frozen throughout (inferred)
CLIP 文字编码器和 MAE 图像编码器在预训练和微调过程中始终保持冻结，以节省计算成本。这意味着模型无法通过端到端优化进一步适配特定机器人任务的视觉表征，对细粒度操作场景可能存在表达瓶颈。The CLIP text encoder and the MAE image encoder stay frozen throughout pre-training and fine-tuning in order to save computation. This means the model cannot further adapt its visual representation to a specific robot task through end-to-end optimization, which may leave a representational bottleneck for fine-grained manipulation scenarios.
仅在 CALVIN 及有限真实场景验证（inferred）Validated only on CALVIN and limited real-world settings (inferred)
实验主要在 CALVIN 模拟器和单一真实机器人平台（物体搬运 + 抽屉）上开展。能否推广到更复杂的双臂操作、多步接触任务或高动态场景，尚未经过系统验证。The experiments are conducted mainly on the CALVIN simulator and a single real-robot platform (object transportation + drawer). Whether the approach extends to more complex bimanual manipulation, multi-step contact-rich tasks or highly dynamic scenes has not been systematically validated.
