Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation
Minho Heo*, Youngwoon Lee* (UC Berkeley), Doohyun Lee, Joseph J. Lim — KAIST & UC Berkeley · RSS 2023
FurnitureBench 将家具组装作为长时域复杂机器人操作的下一个里程碑基准。它提供了 3D 可打印家具模型、超过 219.6 小时的遥操作演示数据（5000+ 条演示）、一键复现的机器人控制软件以及基于 Isaac Gym 的 FurnitureSim 模拟器，使任何实验室都能以标准化方式评估强化学习与模仿学习算法在真实世界中的表现。FurnitureBench establishes furniture assembly as the next milestone benchmark for long-horizon complex robotic manipulation. It provides 3D-printable furniture models, over 219.6 hours of teleoperated demonstration data (5000+ demonstrations), one-click reproducible robot control software and the Isaac Gym-based FurnitureSim simulator, so that any laboratory can evaluate reinforcement learning and imitation learning algorithms in the real world in a standardized way.
现有真实世界机器人操作基准仅能评估推物、拾放等简单短时域任务。要让机器人真正胜任日常任务（整理房间、烹饪、组装家具），需要更具挑战性、可复现的长时域复杂操作基准。Existing real-world robotic manipulation benchmarks can only evaluate simple short-horizon tasks such as pushing and pick-and-place. For robots to truly handle everyday tasks (tidying a room, cooking, assembling furniture), a more challenging and reproducible benchmark for long-horizon complex manipulation is needed.
"these approaches have been limited to learning simple behaviors in current real-world manipulation benchmarks, such as pushing or pick-and-place."
FurnitureBench 概览。（左）机器人在真实环境中组装好的家具陈列。（右）基准包含 8 种不同家具模型，每种模型均受 IKEA 家具启发、经由 3D 打印制作，从而保证任意实验室均可复现相同实验条件。Overview of FurnitureBench. (Left) Furniture assembled by the robot in the real world. (Right) The benchmark contains 8 different furniture models, each inspired by IKEA furniture and fabricated by 3D printing, so that any laboratory can reproduce identical experimental conditions.
家具组装之所以是理想的基准任务，在于它同时要求：Furniture assembly is an ideal benchmark task because it simultaneously demands:
长时域规划（Long-horizon planning）：单次完整组装平均需 60–230 秒、600–2300 个低层步骤，远超 Roboturk（15–120 秒）等现有基准。Long-horizon planning: one complete assembly takes 60–230 seconds and 600–2300 low-level steps on average, far beyond existing benchmarks such as Roboturk (15–120 seconds).
多样灵巧技能（Diverse dexterous skills）：抓取、重定向、插接（inserting）、螺旋固定（screwing）等技能缺一不可。Diverse dexterous skills: grasping, reorienting, inserting and screwing are all indispensable.
可靠 3D 感知（3D perception）：需通过正面摄像头与腕部摄像头协作完成精准对齐。Reliable 3D perception: precise alignment has to be achieved by combining the front camera and the wrist camera.
8种家具模型furniture models
5000+条遥操作演示teleoperated demonstrations
219.6h演示总时长total demonstration time
10Hz末端执行器控制频率end-effector control rate
02基准设计Benchmark Design
FurnitureBench 围绕"可复现（reproducible）"与"易用（easy-to-use）"两大核心目标设计，涵盖真实世界机器人系统、标准化家具模型、演示数据集与模拟器四大模块。FurnitureBench is designed around two core goals, reproducible and easy-to-use, and covers four modules: the real-world robot system, standardized furniture models, the demonstration dataset and the simulator.
真实世界实验环境。系统由一台 7-DoF Franka Emika Panda 机械臂、三台 Intel RealSense D435 RGB-D 相机和 IKEA TOMMARYD 桌面组成；AprilTag 用于估计家具零件位姿以辅助任务初始化，不作为策略观测输入。Real-world experimental setup. The system consists of one 7-DoF Franka Emika Panda arm, three Intel RealSense D435 RGB-D cameras and an IKEA TOMMARYD table top; AprilTag is used to estimate the poses of furniture parts for task initialization and is not fed to the policy as an observation.
可复现系统设计Reproducible System Design
机器人系统使用学术界广泛采用的硬件（Franka Panda + RealSense D435），家具零件全部经由 3D 打印制作——研究者只需下载模型文件即可在自己实验室复现完全相同的家具零件。绿色摄影背景与受控照明（色温 4600K–6000K，亮度 ≤4000 lm）进一步降低环境变量。任务初始化 GUI 工具引导用户将零件摆放至从预定分布采样的目标位姿，确保不同实验室的评测结果可比较。The robot system uses hardware widely adopted in academia (Franka Panda + RealSense D435), and all furniture parts are 3D printed — researchers only need to download the model files to reproduce exactly the same parts in their own lab. A green photographic backdrop and controlled lighting (color temperature 4600K–6000K, brightness ≤4000 lm) further reduce environmental variation. A task-initialization GUI tool guides the user to place the parts at target poses sampled from a predefined distribution, making evaluation results comparable across laboratories.
演示数据集Demonstration Dataset
作者使用 Oculus Quest 2 VR 控制器与键盘共同完成了 219.6 小时的成功演示采集，覆盖所有 8 种家具与 3 个随机化等级（low / medium / high）。单条演示约 300–3000 步，属于典型的长时域轨迹。Using an Oculus Quest 2 VR controller together with a keyboard, the authors collected 219.6 hours of successful demonstrations, covering all 8 furniture models and 3 randomness levels (low / medium / high). A single demonstration spans roughly 300–3000 steps, a typical long-horizon trajectory.
FurnitureSim 模拟器FurnitureSim Simulator
FurnitureSim 基于 Isaac Gym 与 Factory 构建，支持快速在线渲染与离线光线追踪渲染，与真实世界共享相同的 3D 家具模型与机器人控制器。模拟实验结果与真实世界结果呈正相关，可作为算法快速迭代的代理指标；但仍存在 sim-to-real domain gap，策略不能直接迁移。FurnitureSim is built on Isaac Gym and Factory, supports both fast online rendering and offline ray-traced rendering, and shares the same 3D furniture models and robot controllers with the real world. Simulation results are positively correlated with real-world results and can serve as a proxy metric for fast algorithm iteration; a sim-to-real domain gap nevertheless remains, so policies cannot be transferred directly.
观测空间Observation Space
正面摄像头 RGB（1280×720 → 中心裁剪 224×224）Front camera RGB (1280×720 → center-cropped to 224×224)
腕部摄像头 RGB（直接下采样 224×224）Wrist camera RGB (directly downsampled to 224×224)
IQL（Implicit Q-Learning）+ R3M featuresIQL (Implicit Q-Learning) + R3M features
指标：技能成功率（10 次 trial 平均）、完成阶段数（phases）Metrics: skill success rate (averaged over 10 trials) and number of completed phases
03实验结果Experimental Results
基准评测分三部分：单技能基准（Single-Skill）、全装配基准（Full-Assembly）和模拟器基准（Simulation）。整体结论是：现有 IL 与离线 RL 算法在该基准上尚无法取得实质进展，特别是在"插接"和"螺旋固定"技能上。The benchmark evaluation has three parts: Single-Skill, Full-Assembly and Simulation. The overall conclusion is that existing IL and offline RL algorithms make no substantial progress on this benchmark, especially on the inserting and screwing skills.
IQL 在 square_table 的 Screwing 技能（low randomness）达到 90% 成功率，而 Inserting 技能在所有家具上普遍为 0–20%，证明"精准对齐插接"是当前算法的主要瓶颈。IQL reaches a 90% success rate on the Screwing skill of square_table (low randomness), whereas the Inserting skill generally stays at 0–20% across all furniture, showing that precisely aligned insertion is the main bottleneck of current algorithms.
全装配基准结果。BC 和 IQL 在所有 8 种家具、3 种随机化等级下的完成阶段数（phases）均值，误差条表示最小/最大完成数。BC 和 IQL 均无法完成任何一对家具零件的组装（完成阶段数约为 1–2 / 12）。Full-assembly benchmark results. Mean number of completed phases for BC and IQL over all 8 furniture models and 3 randomness levels; error bars denote the minimum and maximum. Neither BC nor IQL manages to assemble a single pair of furniture parts (about 1–2 of 12 phases completed).
one_leg 装配消融分析（IQL-R3M，平均完成阶段数）one_leg Assembly Ablation (IQL-R3M, mean completed phases)
实验设置Setting
low randomness
medium randomness
Original（低随机化数据）Original (low-randomness data)
3.8
—
Original（中随机化数据）Original (medium-randomness data)
3.0
3.0
Mixed data（混合数据）Mixed data
4.6
3.7
Front camera only（无腕部摄像头）Front camera only (no wrist camera)
2.0
1.3
No AprilTag
3.1
2.4
Random AprilTag
3.4
2.7
可复现性验证Reproducibility Validation
可复现性分析。10 名参与者各自独立搭建实验环境（约 3 小时），并在各自环境中评测同一 IQL-R3M 策略。新环境平均达到原始环境性能的 84%（范围 75%–93%），证明基准具备良好的跨实验室复现性。Reproducibility analysis. 10 participants each set up the experimental environment independently (about 3 hours) and evaluated the same IQL-R3M policy in their own setup. The new environments reach on average 84% of the original environment's performance (range 75%–93%), showing that the benchmark reproduces well across laboratories.
关键发现Key Findings
IQL 整体优于 BC，低随机化初始化带来更高成功率。IQL outperforms BC overall, and low-randomness initialization yields higher success rates.
Mixed data（2000 条演示）相比 Original（1000 条）显著提升：low 从 3.8 → 4.6，med 从 3.0 → 3.7。Mixed data (2000 demonstrations) improves markedly over Original (1000): low goes from 3.8 → 4.6, med from 3.0 → 3.7.
腕部摄像头至关重要：仅使用正面摄像头时成功率从 3.8 骤降至 2.0。The wrist camera is essential: with the front camera alone, performance drops sharply from 3.8 to 2.0.
视觉策略并非依靠 AprilTag 作弊：去除 AprilTag 后性能仅轻微下降（3.8 → 3.1）。The visual policy is not cheating via AprilTag: removing AprilTag degrades performance only slightly (3.8 → 3.1).
模拟与真实世界结果呈正相关，可用作快速代理指标。Simulation and real-world results are positively correlated, so simulation can serve as a fast proxy metric.
04局限性Limitations
Note:以下局限性均为作者在论文"Limitations and Future work"节中明确陈述。All limitations below are explicitly stated by the authors in the "Limitations and Future work" section of the paper.
家具模型经过简化，与真实 IKEA 家具存在差异The furniture models are simplified and differ from real IKEA furniture
"the 3D furniture models are still tailored to common robotic arms for research, e.g., all pieces have widths larger than 2 cm for easy grasping, which is larger than the tiny screws used in real-world IKEA furniture. Furthermore, our furniture models are much smaller in scale compared to the real-world furniture."尽管如此，作者认为基准仍涵盖大多数长时域操作挑战（temporal credit assignment、exploration、perception、dexterous manipulation）。"the 3D furniture models are still tailored to common robotic arms for research, e.g., all pieces have widths larger than 2 cm for easy grasping, which is larger than the tiny screws used in real-world IKEA furniture. Furthermore, our furniture models are much smaller in scale compared to the real-world furniture." Even so, the authors argue that the benchmark still covers most long-horizon manipulation challenges (temporal credit assignment, exploration, perception, dexterous manipulation).
单任务、单臂评测，缺乏多任务与多臂协作Single-task, single-arm evaluation, without multi-task or multi-arm collaboration
当前基准主要以单任务标准、单 Franka Panda 手臂评测。未来方向包括：多任务强化学习（利用不同家具类别的共享信息）、多臂/移动机器人协作，以提升操作灵巧性。The current benchmark evaluates mainly under a single-task protocol with a single Franka Panda arm. Future directions include multi-task reinforcement learning (exploiting information shared across furniture categories) and multi-arm / mobile-robot collaboration, so as to improve manipulation dexterity.
FurnitureSim 存在 sim-to-real domain gapFurnitureSim suffers from a sim-to-real domain gap
"system identification is very challenging. For example, the same torque command does not lead to the same robot trajectories in simulation and the real world due to inaccurate robot modeling (e.g., mass of each robot part, friction of joints)."视觉域与物理域均存在差异，直接 sim-to-real 策略迁移困难。作者将弥合该差距列为未来工作。"system identification is very challenging. For example, the same torque command does not lead to the same robot trajectories in simulation and the real world due to inaccurate robot modeling (e.g., mass of each robot part, friction of joints)." Both the visual and the physical domain differ, so directly transferring a sim-to-real policy is hard. The authors list closing this gap as future work.
