Gemini Robotics: Bringing AI into the Physical World

01 动机Motivation

现有机器人基础模型普遍缺乏对物理世界的几何与时序理解，难以在多样化环境中完成灵巧操作；同时，现有 VLA 模型在零样本泛化与快速适配新机体方面存在明显瓶颈。Gemini Robotics 旨在以统一的多模态大模型为核心，将强大的语言推理能力延伸至物理世界。Existing robot foundation models generally lack geometric and temporal understanding of the physical world, making dexterous manipulation in diverse environments difficult; meanwhile, existing VLA models show clear bottlenecks in zero-shot generalization and in fast adaptation to new embodiments. Gemini Robotics aims to place a unified multimodal foundation model at the core and extend strong language reasoning into the physical world.

"We present Gemini Robotics, an advanced Vision-Language-Action (VLA) generalist model capable of directly controlling robots to execute complex manipulation tasks while remaining robust to object variations, environmental changes, and natural language variations."

02 方法Method

Gemini Robotics 体系由两个互补的模型构成：负责具身推理的 Gemini Robotics-ER，以及直接输出机器人动作的 VLA 模型 Gemini Robotics。两者共享 Gemini 2.0 基础，分别针对感知理解与实时控制进行训练。The Gemini Robotics family consists of two complementary models: Gemini Robotics-ER, responsible for embodied reasoning, and the VLA model Gemini Robotics, which directly outputs robot actions. Both share the Gemini 2.0 foundation and are trained respectively for perceptual understanding and for real-time control.

Gemini Robotics-ER：具身推理增强Gemini Robotics-ER: Enhanced Embodied Reasoning

Gemini Robotics-ER（Embodied Reasoning）在 Gemini 2.0 基础上，通过专项训练使模型获得对物理世界丰富几何与时序细节的理解。具体能力包括：Gemini Robotics-ER (Embodied Reasoning) builds on Gemini 2.0 and, through specialized training, acquires an understanding of rich geometric and temporal detail in the physical world. Its concrete capabilities include:

• 2D 目标检测与语言引导的点位指定（affordance pointing）2D object detection and language-guided affordance pointing
• 轨迹预测：预见物体运动路径，用于抓取规划Trajectory prediction: anticipating object motion paths for grasp planning
• 3D 多视角对应（multi-view correspondence）与空间感知3D multi-view correspondence and spatial perception
• 链式推理（Chain-of-Thought）：在 ERQA benchmark 上，Gemini 2.0 Pro Experimental 不使用 CoT 为 48.3%，使用后提升至 54.8%Chain-of-Thought reasoning: on the ERQA benchmark, Gemini 2.0 Pro Experimental scores 48.3% without CoT and improves to 54.8% with it

为评估具身推理能力，作者构建了 ERQA（Embodied Reasoning QA） benchmark：400 道多选题，涵盖 spatial reasoning、trajectory reasoning、action reasoning、state estimation、multi-view reasoning 等类别，其中 28% 包含多张图像。To evaluate embodied reasoning, the authors construct the ERQA (Embodied Reasoning QA) benchmark: 400 multiple-choice questions spanning spatial reasoning, trajectory reasoning, action reasoning, state estimation and multi-view reasoning, of which 28% contain multiple images.

Gemini Robotics：VLA 主干 + 本地 Action DecoderGemini Robotics: VLA Backbone + Local Action Decoder

Gemini Robotics 采用云端-边端协同的两段式架构。云端运行 Gemini Robotics-ER 的蒸馏版本作为 backbone，机载计算机运行轻量 action decoder，实现实时控制：Gemini Robotics adopts a two-stage cloud-edge architecture. The cloud runs a distilled version of Gemini Robotics-ER as the backbone, while the onboard computer runs a lightweight action decoder for real-time control:

• 训练数据：在 ALOHA 2 机器人舰队上采集，历时 12 个月，包含数千小时真实场景遥操作演示，覆盖数千种多样化任务Training data: collected on a fleet of ALOHA 2 robots over 12 months, comprising thousands of hours of real-world teleoperated demonstrations across thousands of diverse tasks
• action chunks 输出，有效控制频率 50Hzaction chunks as output, with an effective control frequency of 50Hz
• 查询-响应延迟从秒级压缩至 160ms 以内，端到端约 250msQuery-to-response latency compressed from seconds to under 160ms, about 250ms end to end

专项化与新机体适配Specialization and Adaptation to New Embodiments

通过任务专项数据集（每个长时域任务约 2000–5000 条演示），Gemini Robotics 实现了午餐盒整理、折纸等高灵巧任务，成功率达 100%。对于 bi-arm Franka 机器人与 Apollo 人形机器人（Apptronik）等新机体，只需少量迁移数据即可适配，bi-arm Franka 平均成功率约 63%。With task-specific datasets (about 2000–5000 demonstrations per long-horizon task), Gemini Robotics accomplishes highly dexterous tasks such as lunch-box packing and origami folding at a 100% success rate. For new embodiments such as the bi-arm Franka robot and the Apollo humanoid (Apptronik), only a small amount of transfer data is needed for adaptation, with the bi-arm Franka reaching about 63% average success.

03 实验Experiments

实验在 ALOHA 2 仿真与真实机器人上展开，对比基线包括 GPT-4o、Claude 3.5、π₀ 等；评测维度涵盖零样本控制、少样本快速适配、语言泛化与视觉泛化。Experiments are run in ALOHA 2 simulation and on real robots, with baselines including GPT-4o, Claude 3.5 and π₀; the evaluation covers zero-shot control, few-shot fast adaptation, language generalization and visual generalization.

ERQA Benchmark — 具身推理能力ERQA Benchmark — Embodied Reasoning Ability

模型Model	ERQA 准确率ERQA Accuracy	ERQA + CoT
Claude 3.5	35.5%	—
GPT-4o	47.0%	—
Gemini 2.0 Flash	46.3%	—
Gemini 2.0 Pro Experimental	48.3%	54.8%

零样本机器人控制（ALOHA 2 仿真）Zero-Shot Robot Control (ALOHA 2 Simulation)

Gemini Robotics-ER 通过代码生成实现零样本机器人控制，Gemini 2.0 Flash 平均成功率 27%，Gemini Robotics-ER 平均 53%：Gemini Robotics-ER achieves zero-shot robot control through code generation: Gemini 2.0 Flash reaches 27% average success and Gemini Robotics-ER 53%:

任务Task	Gemini 2.0 Flash	Gemini Robotics-ER
Banana Lift	—	86%
Banana in Bowl	—	84%
Mug on Plate	—	72%
Bowl on Rack	—	60%
Banana Handover	—	54%
Fruit Bowl	—	16%
Pack Toy	—	0%
平均Average	27%	53%

泛化性能Generalization Performance

长时域灵巧操作与快速适配Long-Horizon Dexterous Manipulation and Fast Adaptation

快速适配实验（Section 4.3）表明，在 8 个任务中有 7 个使用不超过 100 条演示即达到 70% 以上成功率（等价于 15 分钟至 1 小时的数据采集），其中两个任务达到 100% 成功率。The fast-adaptation experiments (Section 4.3) show that 7 of 8 tasks exceed 70% success with no more than 100 demonstrations (equivalent to 15 minutes to 1 hour of data collection), and two of them reach a 100% success rate.

3D 感知（SUN-RGBD 基准）3D Perception (SUN-RGBD Benchmark)

Gemini Robotics-ER 在 3D Detection 上达到 48.3 AP@15，超越专项化模型。Gemini Robotics-ER reaches 48.3 AP@15 on 3D Detection, surpassing specialized models.

04 局限性Limitations

零样本灵巧操作成功率有限Limited zero-shot success on dexterous manipulation

论文明确指出："For tasks that require dexterous motions, the zero-shot success rate is not high"（Table 6 caption）。Gemini Robotics-ER 在真实机器人上进行折叠衣物（dress folding）等高灵巧任务时，零样本成功率显著低于简单抓取任务；Pack Toy 任务在仿真中零样本成功率为 0%。The paper states explicitly: "For tasks that require dexterous motions, the zero-shot success rate is not high" (Table 6 caption). On real robots, the zero-shot success rate of Gemini Robotics-ER on highly dexterous tasks such as dress folding is markedly lower than on simple grasping tasks; the Pack Toy task reaches 0% zero-shot success in simulation.

VLM 与实时控制之间存在固有鸿沟An inherent gap between the VLM and real-time control

论文指出："As a VLM, there are inherent limitations for robot control, especially for more dexterous tasks, due to the intermediate steps needed to connect the model's innate embodied reasoning capabilities to robotic actions."即便 Gemini Robotics-ER 具备较强的具身推理能力，将其与精确的关节-级别运动控制衔接仍需复杂中间步骤。The paper notes: "As a VLM, there are inherent limitations for robot control, especially for more dexterous tasks, due to the intermediate steps needed to connect the model's innate embodied reasoning capabilities to robotic actions." Even though Gemini Robotics-ER possesses strong embodied reasoning, bridging it to precise joint-level motion control still requires complex intermediate steps.

长时域任务依赖大规模专项数据集Long-horizon tasks depend on large specialized datasets

午餐盒整理等高难度长时域任务需要 2000–5000 条演示进行专项化微调，数据采集成本较高，难以快速推广到任意新任务。Difficult long-horizon tasks such as lunch-box packing require 2000–5000 demonstrations for specialization fine-tuning; the data collection cost is high and hard to extend quickly to arbitrary new tasks.

仿真与真实环境存在 sim-to-real 差距A sim-to-real gap between simulation and the real world

香蕉递接（banana handover）任务在仿真中成功率为 54%，在真实机器人上因"calibration imperfections and other sources of noise"性能进一步下降。The banana handover task reaches a 54% success rate in simulation, while on real robots performance degrades further because of "calibration imperfections and other sources of noise".