GR-2: 面向机器人操作的生成式视频-语言-动作模型

01 动机Motivation

机器人操作数据的采集代价高昂，限制了策略学习的规模化。互联网上大量的视频数据蕴含丰富的物体交互知识——如何将这些"免费"的视觉先验有效迁移到机器人策略学习？Collecting robot manipulation data is expensive, which limits the scaling of policy learning. The vast amount of video data on the internet carries rich knowledge about object interaction — how can such "free" visual priors be transferred effectively to robot policy learning?

"Pre-training on video generation can effectively transfer knowledge to robot policy learning, enabling a generalist robot agent to perform diverse manipulation tasks and generalize to novel environments."

多任务评测设置 — 多任务学习的五种评测设置：Simple（标准场景）、Distractor（视觉干扰）、Unseen Backgrounds（未见背景）、Unseen Environments（未见环境）、Unseen Manipulation（未见操作物体）。GR-2 需要在所有五种设定下保持高成功率。The five evaluation settings of multi-task learning: Simple (standard scene), Distractor (visual distractors), Unseen Backgrounds, Unseen Environments, and Unseen Manipulation (unseen objects to manipulate). GR-2 has to keep a high success rate under all five settings.

97.7%100+ 任务平均成功率（Simple setting）Average success rate over 100+ tasks (Simple setting)

79.0%Bin Picking 平均成功率（GR-1 仅 33.3%）Average Bin Picking success rate (GR-1 only 33.3%)

4.64CALVIN 平均连续完成任务数（avg length）Average number of consecutive tasks completed on CALVIN (avg length)

38M预训练视频剪辑数量Number of video clips used for pre-training

02 方法Method

GR-2 采用两阶段训练策略：大规模视频生成预训练 + 机器人数据微调。骨干网络为 GPT-style Transformer，图像由冻结的 VQGAN 编码为离散 token，文本由冻结的 CLIP 编码；仅 95M 参数参与训练（默认 GR-2-B），其余为冻结参数。GR-2 adopts a two-stage training strategy: large-scale video generation pre-training plus fine-tuning on robot data. The backbone is a GPT-style Transformer; images are encoded into discrete tokens by a frozen VQGAN and text by a frozen CLIP; only 95M parameters are trained (GR-2-B by default) while the rest stay frozen.

GR-2 训练流程 — **GR-2 两阶段训练流程。**左：视频生成预训练阶段——给定文本描述和当前帧，模型通过自回归 next-token prediction 预测后续帧，在 Howto100M、Ego4D、Something-Something V2、EPIC-KITCHENS、Kinetics-700 及 RT-1/Bridge 机器人数据集上进行训练。右：机器人微调阶段——在机器人轨迹数据上同时学习视频预测和动作生成，动作通过 conditional VAE 解码。**The two-stage training pipeline of GR-2.** Left: the video generation pre-training stage — given a text description and the current frame, the model predicts subsequent frames by autoregressive next-token prediction, trained on Howto100M, Ego4D, Something-Something V2, EPIC-KITCHENS, Kinetics-700 and the RT-1/Bridge robot datasets. Right: the robot fine-tuning stage — video prediction and action generation are learned jointly on robot trajectory data, with actions decoded by a conditional VAE.

阶段一：视频生成预训练Stage 1: Video Generation Pre-training

在 3800万 文本-视频对（超过 500亿 token）上训练。数据来源覆盖烹饪、运动、日常操作等多样场景，以及真实机器人操作数据集（RT-1、Bridge）。模型学习根据文本指令和起始帧，自回归预测后续帧序列，从而获取物体外观、运动规律和物理交互的通用表征。Trained on 38000000 text-video pairs (more than 50000000000 tokens). The data covers diverse scenarios such as cooking, sports and everyday manipulation, as well as real robot manipulation datasets (RT-1, Bridge). The model learns to autoregressively predict subsequent frame sequences from a text instruction and an initial frame, thereby acquiring general representations of object appearance, motion patterns and physical interaction.

阶段二：机器人轨迹微调Stage 2: Robot Trajectory Fine-tuning

在机器人演示数据上进行 双重预测（dual prediction）：既生成未来视频帧，又同步预测动作序列。动作由 conditional VAE 编码并解码，以机器人关节角度和夹爪状态为输入。此阶段的视频预测目标充当辅助监督，促进视觉表征与动作策略的协同学习。On robot demonstration data the model performs dual prediction: it generates future video frames while simultaneously predicting action sequences. Actions are encoded and decoded by a conditional VAE that takes robot joint angles and gripper states as input. The video prediction objective at this stage acts as auxiliary supervision and promotes joint learning of visual representations and the action policy.

Whole-Body Control（全身控制）算法Whole-Body Control Algorithm

在真实机器人部署中，GR-2 输出的末端执行器轨迹通过 trajectory optimization 和 real-time motion tracking 转换为关节指令，实现灵活稳定的操作控制，无需针对每台机器人单独设计低层控制器。In real-robot deployment, the end-effector trajectories produced by GR-2 are converted into joint commands through trajectory optimization and real-time motion tracking, achieving flexible and stable manipulation control without designing a dedicated low-level controller for every robot.

模型规模（Scaling）Model Scale (Scaling)

论文测试了四种规格：GR-2-S（30M 可训练参数）、GR-2-B（95M）、GR-2-L（312M）、GR-2-XL（719M），验证损失和任务成功率均随规模单调提升，表现出良好的 scaling 特性。The paper evaluates four sizes: GR-2-S (30M trainable parameters), GR-2-B (95M), GR-2-L (312M) and GR-2-XL (719M). Both validation loss and task success rate improve monotonically with scale, showing favourable scaling behaviour.

03 实验Experiments

三大评测基准：(1) 105项桌面多任务学习（40,000条轨迹）；(2) 端到端 Bin Picking（122个物体，94,000条轨迹）；(3) CALVIN 仿真基准（34项任务，1~5步连续序列）。对比基准包括 RT-1、MT-ACT、HULC、RoboFlamingo 和 GR-1。Three evaluation benchmarks: (1) multi-task learning over 105 tabletop tasks (40,000 trajectories); (2) end-to-end Bin Picking (122 objects, 94,000 trajectories); (3) the CALVIN simulation benchmark (34 tasks, consecutive sequences of 1–5 steps). Baselines include RT-1, MT-ACT, HULC, RoboFlamingo and GR-1.

多任务学习（105 tasks）Multi-Task Learning (105 tasks)

评测设置Setting	GR-1	GR-2	备注Notes
Simple	—	97.7%	标准场景，每任务400条轨迹Standard scene, 400 trajectories per task
Unseen Backgrounds	低于 GR-2below GR-2	71.4%	未见背景泛化Generalization to unseen backgrounds
Unseen Environments	低于 GR-2below GR-2	71.7%	未见环境泛化Generalization to unseen environments
Unseen Environments (w/ DA)	—	87.0%	加入数据增强后With data augmentation
Unseen Manipulation	低于 GR-2below GR-2	55.8%	未见操作物体（最难）Unseen objects to manipulate (hardest)
Simple（每任务~50条）Simple (~50 per task)	—	73.9%	数据高效场景Data-efficient setting

多任务成功率对比 — 多任务学习成功率对比。GR-2 在所有五种评测设置下均优于 GR-1，尤其在泛化场景（Unseen Backgrounds/Environments/Manipulation）中优势显著。图中同时展示了使用400条和50条轨迹/任务两种数据量下的对比结果。Comparison of multi-task success rates. GR-2 outperforms GR-1 under all five evaluation settings, with a particularly clear margin in the generalization settings (Unseen Backgrounds/Environments/Manipulation). The figure also compares the two data budgets of 400 and 50 trajectories per task.

端到端 Bin PickingEnd-to-End Bin Picking

122个物体（含透明、可变形、反光物体），分为 Seen / Unseen / Cluttered Seen / Cluttered Unseen 四种场景。GR-2 平均成功率从 GR-1 的 33.3% 大幅提升至 79.0%，在杂乱场景（两倍物体密度）和未见物体上均保持稳健性能。122 objects (including transparent, deformable and reflective ones) are divided into four settings: Seen / Unseen / Cluttered Seen / Cluttered Unseen. The average success rate of GR-2 rises sharply from 33.3% for GR-1 to 79.0%, staying robust in cluttered scenes (twice the object density) and on unseen objects.

Bin Picking 成功率对比 — Bin Picking 四种场景下 GR-2 vs GR-1 成功率对比。GR-2 在所有场景中均显著领先，尤其在 Unseen 和 Cluttered 设置中提升幅度最大。Success rates of GR-2 vs GR-1 under the four Bin Picking settings. GR-2 leads clearly in every setting, with the largest gains in the Unseen and Cluttered settings.

CALVIN 仿真基准CALVIN Simulation Benchmark

连续完成任务数Consecutive tasks completed	GR-1	GR-2	最强 baseline（RoboFlamingo）Strongest baseline (RoboFlamingo)
1 task	94.9%	98.6%	96.4%
2 tasks	89.6%	96.3%	89.6%
3 tasks	84.0%	93.2%	82.4%
4 tasks	79.7%	90.4%	74.0%
5 tasks	73.1%	85.9%	66.0%
Avg length	4.21	4.64	4.09

CALVIN 成功率 — CALVIN 基准上1~5步连续任务成功率及平均完成长度（avg length）对比。GR-2 在所有连续步数上均超越 GR-1 和其他基准方法，5步成功率达85.9%。Success rates for 1–5 consecutive tasks on the CALVIN benchmark, together with the average completed length (avg length). GR-2 surpasses GR-1 and the other baselines at every sequence length, reaching 85.9% for 5 tasks.

Scaling 分析Scaling Analysis

04 局限性Limitations

说明：以下局限性中，标注"【论文明确指出】"的为作者原文陈述；标注"【设计推断】"的为从系统设计中归纳的潜在限制。Note: Among the limitations below, those marked "[stated explicitly in the paper]" are statements from the authors; those marked "[design inference]" are potential limitations inferred from the system design.

未见操作物体的成功率偏低【论文明确指出】Low success rate on unseen objects to manipulate [stated explicitly in the paper]

在 Unseen Manipulation 设置下，GR-2 的成功率仅为 55.8%，远低于其他场景。论文指出典型失败案例包括"picking unseen objects of novel shapes"和"mistakenly selecting wrong object"。作者明确表示将"explore techniques to further improve generalization for unseen manipulation tasks, including handling novel objects and executing new skills"。Under the Unseen Manipulation setting the success rate of GR-2 is only 55.8%, far below the other settings. The paper notes that typical failure cases include "picking unseen objects of novel shapes" and "mistakenly selecting wrong object". The authors state explicitly that they will "explore techniques to further improve generalization for unseen manipulation tasks, including handling novel objects and executing new skills".

仍需大量示范数据【设计推断】Large amounts of demonstration data are still needed [design inference]

尽管视频预训练提升了数据效率，多任务学习仍需约40,000条机器人轨迹（105项任务），Bin Picking 需要94,000条轨迹。相比于零样本或少样本泛化的目标，数据需求仍然显著。Although video pre-training improves data efficiency, multi-task learning still needs about 40,000 robot trajectories (105 tasks) and Bin Picking requires 94,000 trajectories. Compared with the goal of zero-shot or few-shot generalization, the data demand remains substantial.

预训练计算资源要求高【设计推断】Pre-training demands heavy compute resources [design inference]

在3800万视频、500亿 token 上进行预训练对计算资源要求极高，限制了研究复现性和小机构的可及性。论文未提供完整的预训练成本分析。Pre-training on 38000000 videos and 50000000000 tokens places extremely high demands on compute resources, which limits reproducibility and accessibility for smaller institutions. The paper gives no complete analysis of the pre-training cost.

动作表示与机器人平台耦合【设计推断】Action representation is coupled to the robot platform [design inference]

GR-2 的动作空间基于关节角度和夹爪状态，切换到不同形态（如双臂、足式机器人）需要重新设计状态编码器和 Whole-Body Control 模块，跨平台迁移代价较高。The action space of GR-2 is based on joint angles and gripper states; switching to a different morphology (such as a dual-arm or legged robot) requires redesigning the state encoder and the Whole-Body Control module, so cross-platform transfer is costly.