RoboCasa 是一个面向通用家用机器人的大规模仿真框架，专注于厨房场景。它提供 120 个高度多样化的场景配置、2,509 个跨 153 类别的高质量 3D 资产，以及 100 项覆盖原子到复合层级的操作任务。通过 MimicGen 自动生成合成轨迹，框架实现了可扩展的模仿学习，并在真实机器人上验证了 sim-to-real 迁移效果。RoboCasa is a large-scale simulation framework for generalist household robots, focused on kitchen settings. It provides 120 highly diverse scene configurations, 2,509 high-quality 3D assets spanning 153 categories, and 100 manipulation tasks covering atomic to composite levels. By automatically generating synthetic trajectories with MimicGen, the framework enables scalable imitation learning, and validates sim-to-real transfer on a real robot.
训练通用机器人的最大瓶颈在于大规模、多样化数据集的匮乏。现实采集成本极高，而现有仿真环境场景单一、资产稀少，难以覆盖日常家务的复杂性。RoboCasa 旨在通过构建写实感强的大规模厨房仿真，结合 AI 生成工具（text-to-3D、text-to-image、LLM 任务生成），为机器人学习提供可扩展的合成数据源。The biggest bottleneck in training generalist robots is the scarcity of large-scale, diverse datasets. Real-world collection is prohibitively expensive, while existing simulation environments offer monotonous scenes and few assets, making it hard to cover the complexity of everyday household chores. RoboCasa aims to provide a scalable source of synthetic data for robot learning by building a photorealistic, large-scale kitchen simulation combined with AI generative tools (text-to-3D, text-to-image, LLM task generation).
"We present RoboCasa, a large-scale simulation framework for training generalist robots in everyday environments. We focus on the kitchen as a setting that offers diverse challenges, including object diversity, spatial reasoning, and long-horizon task execution."
图 1：RoboCasa 中的厨房平面图类型。参考家居规划杂志设计了 10 种不同形状与尺寸的平面图，从基础 L 形到高端岛台式厨房，覆盖 12 种建筑风格（工业、斯堪的纳维亚、沿海、现代等），共 120 种场景配置。Figure 1: Kitchen floor plan types in RoboCasa. Drawing on home planning magazines, 10 floor plans of different shapes and sizes were designed, from a basic L-shape to high-end island kitchens, spanning 12 architectural styles (industrial, Scandinavian, coastal, modern, etc.), for a total of 120 scene configurations.
图 2：RoboCasa 提供的多样化高质量 3D 物体。2,509 个资产分布在蔬菜、家禽、饮料、容器等 153 个类别中，大部分（1,592 个）通过 Luma.ai text-to-3D 生成，其余来自 Objaverse 数据集。Figure 2: The diverse, high-quality 3D objects provided by RoboCasa. The 2,509 assets are distributed across 153 categories such as vegetables, poultry, beverages and containers; most of them (1,592) were generated with Luma.ai text-to-3D, and the rest come from the Objaverse dataset.
02方法Method
RoboCasa 由三个核心组件构成：（1）写实的厨房仿真环境，支持多种机器人形态；（2）结合 LLM 自动生成的 100 项任务；（3）基于 MimicGen 的大规模合成数据生成流程。框架构建于 RoboSuite / MuJoCo 之上，通过 NVIDIA Omniverse 实现照片级渲染，仿真速度约 25.2 fps。RoboCasa comprises three core components: (1) a photorealistic kitchen simulation environment supporting multiple robot embodiments; (2) 100 tasks generated with the help of an LLM; (3) a large-scale synthetic data generation pipeline based on MimicGen. The framework is built on top of RoboSuite / MuJoCo, achieves photorealistic rendering through NVIDIA Omniverse, and runs at about 25.2 fps.
环境与资产设计Environment and Asset Design
场景设计兼顾视觉多样性与物理真实性：每个厨房随机化平面布局、建筑风格、墙壁/地板/台面纹理（各 100 种 AI 生成纹理），以及电器和物体摆放。支持带轮底座、人形机器人和四足机器人+手臂等多种形态（cross-embodiment）。电器（微波炉、烤箱、咖啡机、水槽等）均有关节结构和状态变化（如旋钮控制炉灶点火）。The scene design balances visual diversity with physical realism: every kitchen randomizes the floor layout, architectural style, wall/floor/countertop textures (100 AI-generated textures each), and the placement of appliances and objects. Multiple embodiments are supported, including wheeled bases, humanoids and quadrupeds with arms (cross-embodiment). Appliances (microwaves, ovens, coffee machines, sinks, etc.) all have articulated structures and state changes (e.g. knobs controlling stove ignition).
图 3：RoboCasa 中的可交互电器示例。框架提供数十种电器，多数为关节式结构，支持开关柜门/抽屉、旋转旋钮、按键等精细操作，并有对应的状态变化（如炉灶点火、微波炉开关门）。Figure 3: Examples of interactable appliances in RoboCasa. The framework provides dozens of appliances, most of them articulated, supporting fine-grained manipulation such as opening/closing cabinet doors and drawers, turning knobs and pressing buttons, with corresponding state changes (e.g. stove ignition, opening and closing the microwave door).
LLM 辅助任务生成LLM-Assisted Task Generation
借助 GPT-4 的双阶段提示（first prompt → 20 种高层厨房活动；second prompt → 每类活动的具体任务描述），生成 75 项复合任务，人工过滤逻辑不一致后得到最终任务集。任务涵盖冲咖啡/茶、洗碗、备菜、热饭、摆桌等日常场景，每项任务按原子技能（开/关柜、拾取/放置等）分解为有序子步骤。Using a two-stage prompt with GPT-4 (first prompt → 20 high-level kitchen activities; second prompt → concrete task descriptions for each activity), 75 composite tasks were generated, and the final task set was obtained after manually filtering out logically inconsistent ones. The tasks cover everyday scenarios such as brewing coffee/tea, washing dishes, preparing vegetables, reheating food and setting the table; each task is decomposed into ordered sub-steps of atomic skills (opening/closing cabinets, picking/placing, etc.).
图 4：使用 LLM 创建多样化复合任务的流程。首先提示 GPT-4 列举厨房高层活动，再针对每类活动生成具体任务描述，最后人工筛选并编写成可执行的任务蓝图，共 75 项复合任务。Figure 4: The pipeline for creating diverse composite tasks with an LLM. GPT-4 is first prompted to enumerate high-level kitchen activities, then concrete task descriptions are generated for each activity, and finally these are manually screened and written into executable task blueprints, yielding 75 composite tasks in total.
MimicGen 合成数据生成MimicGen Synthetic Data Generation
每项原子任务仅收集 50 条人工演示（3D SpaceMouse 遥操作），随后通过扩展的 MimicGen 系统自动生成大规模合成轨迹。MimicGen 将演示分解为以物体为中心的操作片段，将各片段适配到新场景配置后拼接执行，使用 rejection sampling 保证成功率，并支持多进程并行生成。最终发布数据集包含 100K+ 轨迹（Human-50 基线 1,250 条 + Generated-3000 等级 72,000 条）。Only 50 human demonstrations are collected per atomic task (teleoperated with a 3D SpaceMouse), after which an extended MimicGen system automatically generates large-scale synthetic trajectories. MimicGen decomposes a demonstration into object-centric manipulation segments, adapts each segment to the new scene configuration and stitches them together for execution, uses rejection sampling to guarantee success, and supports multi-process parallel generation. The released dataset contains 100K+ trajectories (1,250 for the Human-50 baseline + 72,000 at the Generated-3000 level).
用于评估的基线策略（BC-Transformer）接受 10 步观测历史与语言目标条件，三路相机输入（手眼、左/右工作区），ResNet-18 视觉编码器 + FiLM 融合，6 层 Transformer（约 2000 万参数），训练 500K 步，学习率 1e-4 with warmup。The baseline policy used for evaluation (BC-Transformer) takes a 10-step observation history and a language goal condition, three camera inputs (eye-in-hand, left/right workspace), a ResNet-18 visual encoder with FiLM fusion, and a 6-layer Transformer (about 20000000 parameters), trained for 500K steps with a learning rate of 1e-4 with warmup.
03实验Experiments
实验围绕三条主线展开：（A）原子任务数据规模的扩展效应；（B）复合任务的 fine-tuning 效果；（C）真实机器人的 sim-to-real 迁移验证。评估指标均为任务成功率。The experiments follow three main threads: (A) the scaling effect of data volume on atomic tasks; (B) the effectiveness of fine-tuning on composite tasks; (C) validation of sim-to-real transfer on a real robot. The evaluation metric throughout is task success rate.
A · 原子任务：数据规模扩展A · Atomic Tasks: Data Scaling
在 24 项原子任务上，将人工演示（Human-50）与不同规模合成数据（Generated-100/300/3000）的多任务策略进行对比：On 24 atomic tasks, multi-task policies trained on human demonstrations (Human-50) are compared with those trained on synthetic data of different scales (Generated-100/300/3000):
数据集Dataset
轨迹总数Total trajectories
平均成功率Mean success rate
vs Human-50
Human-50
1,200
28.8%
— (基线)— (baseline)
Generated-100
2,400
26.3%
−2.5%
Generated-300
7,200
35.0%
+6.2%
Generated-3000
72,000
47.6%
+65%
图 5：人工演示与机器生成数据集的对比学习曲线（24 项原子任务，成功率）。随合成数据规模增大，呈现清晰的 scaling trend：Generated-3000 成功率比 Human-50 提升 65%。任务难度两极分化：关抽屉（96%）/关单门（94%）等重复性任务趋于收敛，而 pick-and-place（6–44%）和插入（12%）等高多样性任务仍有较大提升空间。Figure 5: Comparative learning curves for human demonstrations and machine-generated datasets (24 atomic tasks, success rate). As the scale of synthetic data grows, a clear scaling trend emerges: Generated-3000 improves the success rate over Human-50 by 65%. Task difficulty is polarized: repetitive tasks such as closing a drawer (96%) and closing a single door (94%) converge, while highly diverse tasks such as pick-and-place (6–44%) and insertion (12%) still leave substantial room for improvement.
B · 复合任务：从原子任务 Fine-tuningB · Composite Tasks: Fine-tuning from Atomic Tasks
5 项代表性复合任务（ArrangeVegetables、MicrowaveThawing、RestockPantry、PreSoakPan、PrepareCoffee）分别采用"从零学习（50 条人工演示）"与"从原子任务预训练 fine-tuning"两种方式：Five representative composite tasks (ArrangeVegetables, MicrowaveThawing, RestockPantry, PreSoakPan, PrepareCoffee) are each trained in two ways: "learning from scratch (50 human demonstrations)" and "fine-tuning from atomic-task pretraining":
任务Task
从零学习From scratch
Fine-tuning
ArrangeVegetables
2.0%
12.0%
MicrowaveThawing
0%
2.0%
RestockPantry
0%
6.0%
PreSoakPan
0%
4.0%
PrepareCoffee
0%
0%
预训练带来一定提升，但复合任务整体成功率仍极低，长程操作的精细控制和阶段过渡是主要瓶颈。Pretraining brings some improvement, but the overall success rate on composite tasks remains extremely low; fine-grained control and stage transitions in long-horizon manipulation are the main bottlenecks.
C · 真实机器人 Sim-to-Real 迁移C · Real-Robot Sim-to-Real Transfer
在真实厨房中用 Franka Panda 带轮移动平台评估三类 pick-and-place 任务（台面→水槽、水槽→台面、台面→柜子），对比"仅真实数据（50 条/任务）"与"真实 + 仿真数据共训练"：In a real kitchen, a Franka Panda on a wheeled mobile base is evaluated on three kinds of pick-and-place task (counter→sink, sink→counter, counter→cabinet), comparing "real data only (50 demonstrations per task)" with "co-training on real + simulation data":
训练数据Training data
已见物体Seen objects
未见物体Unseen objects
Real only
13.6%
2.6%
Real + Sim（本文）Real + Sim (ours)
24.4%
9.3%
提升Improvement
+79%
+258%
图 6：真实机器人实验配置。在真实厨房环境中使用带轮底座的 Franka Emika Panda，评估 sim-to-real 迁移效果。加入仿真数据共训练后，已见物体成功率提升 79%，未见物体提升 258%。Figure 6: The real-robot experimental setup. A Franka Emika Panda on a wheeled base is used in a real kitchen environment to evaluate sim-to-real transfer. With simulation data added for co-training, the success rate improves by 79% on seen objects and by 258% on unseen objects.
04局限性Limitations
说明：以下局限性部分为论文 Future Work 小节明确陈述（标注"论文陈述"），部分为系统设计推断（标注"推断"）。Note: Some of the limitations below are explicitly stated in the paper's Future Work section (marked "stated in the paper"), while others are inferred from the system design (marked "inferred").
复合任务成功率极低（论文陈述）Composite-task success rates are extremely low (stated in the paper)
5 项复合任务中，从零学习成功率几乎为 0，fine-tuning 后最高也仅 12%（ArrangeVegetables）。论文明确指出"significant room for improvement with better architectures/algorithms"，长程任务的阶段过渡与精细操控是主要失败原因。Across the 5 composite tasks, learning from scratch achieves a success rate of nearly 0, and even after fine-tuning the best is only 12% (ArrangeVegetables). The paper explicitly notes "significant room for improvement with better architectures/algorithms"; stage transitions and fine-grained control in long-horizon tasks are the main causes of failure.
合成轨迹质量问题（论文陈述）Synthetic trajectory quality issues (stated in the paper)
MimicGen 生成的轨迹可能包含"jerky motions, collisions"等不理想行为，论文提出未来需自动过滤低质量轨迹，但当前版本尚未实现。Trajectories generated by MimicGen may contain undesirable behaviors such as "jerky motions, collisions"; the paper proposes automatically filtering out low-quality trajectories in the future, but the current version does not yet implement this.
场景范围局限于厨房（论文陈述）Scene coverage is limited to kitchens (stated in the paper)
当前框架仅覆盖厨房环境，缺乏灵巧操作（dexterous）、可变形物体（deformable）和双手协作（bimanual manipulation）支持，论文将扩展至其他家居场景列为未来方向。The current framework covers only kitchen environments and lacks support for dexterous manipulation, deformable objects and bimanual manipulation; the paper lists extending to other household settings as a future direction.
任务实现仍需人工编程（推断）Task implementation still requires manual programming (inferred)
LLM 仅生成任务描述，实际可执行任务蓝图（task blueprints）仍需人工编写代码，限制了任务规模的进一步扩展。论文提出探索 LLM-based code generation 作为未来方向。The LLM only generates task descriptions; the actual executable task blueprints still have to be hand-coded, which limits further scaling of the task set. The paper proposes exploring LLM-based code generation as a future direction.
Sim-to-Real 泛化仍有差距（推断）Sim-to-real generalization gap remains (inferred)
真实机器人实验中，已见物体成功率为 24.4%，未见物体仅 9.3%，表明模型对新物体的视觉泛化能力有限，仿真与真实环境间的 domain gap 仍然显著。In the real-robot experiments, the success rate is 24.4% on seen objects and only 9.3% on unseen objects, indicating limited visual generalization to novel objects; the domain gap between simulation and the real world remains significant.
