SAPIEN: A SimulAted Part-based Interactive ENvironment

01 动机Motivation

构建家庭辅助机器人一直是视觉与机器人领域的核心追求。机器人要完成开冰箱、拿取物品等日常任务，就必须能够感知并操作具有可动部件的关节体（如门、抽屉、水龙头）。然而现有仿真环境在物理真实性、关节体规模和向真实机器人的迁移能力上均存在明显不足。Building household assistive robots has long been a central pursuit of the vision and robotics communities. To accomplish everyday tasks such as opening a refrigerator or fetching an object, a robot must be able to perceive and manipulate articulated objects that have movable parts (e.g., doors, drawers and faucets). Existing simulation environments, however, fall clearly short in physical realism, in the scale of articulated objects, and in transferability to real robots.

"We take one step further in constructing an environment that supports household tasks for training robot learning algorithm. Our work, SAPIEN, is a realistic and physics-rich simulated environment that hosts a large-scale set for articulated objects."

为什么需要 SAPIEN？Why is SAPIEN needed?

现有仿真环境（OpenAI Gym、RLBench、DeepMind Control Suite 等）普遍存在以下问题：Existing simulation environments (OpenAI Gym, RLBench, DeepMind Control Suite, etc.) commonly suffer from the following problems:

• 物理真实性不足：大多简化物理，难以模拟关节体的复杂约束与运动。Insufficient physical realism: most of them simplify physics and can hardly simulate the complex constraints and motions of articulated objects.
• 物体种类稀少：仅支持少量预定义场景，缺乏大规模多样化关节体。Scarce object variety: only a small number of predefined scenes are supported, and large-scale, diverse articulated objects are missing.
• 缺乏部件级理解：无法支持需要精确部件感知的任务（如识别水龙头手柄并旋转）。Lack of part-level understanding: they cannot support tasks that require precise part perception (such as recognizing a faucet handle and rotating it).
• 与真实机器人的语义鸿沟：不提供 ROS 兼容接口，难以迁移到真实硬件。Semantic gap with real robots: no ROS-compatible interface is provided, which makes transfer to real hardware difficult.

如表 1 所示，与 Shape2Motion、RPM-Net、Hu et al. 等已有部件数据集相比，SAPIEN 以 46 类、2,346 个模型（14,068 个可动部件）显著领先，且提供丰富的纹理和真实渲染效果。As shown in Table 1, compared with existing part datasets such as Shape2Motion, RPM-Net and Hu et al., SAPIEN leads by a clear margin with 46 categories and 2,346 models (14,068 movable parts), and it also offers rich textures and realistic rendering.

02 方法Method

SAPIEN 围绕三个核心模块构建：SAPIEN Engine（基于 PhysX 的物理仿真）、SAPIEN Renderer（支持 OpenGL、光线追踪的渲染器）和 SAPIEN Asset（PartNet-Mobility 数据集及机器人模型库）。三者通过统一框架集成，提供 Python API 和 ROS 接口。SAPIEN is built around three core modules: SAPIEN Engine (PhysX-based physics simulation), SAPIEN Renderer (a renderer supporting OpenGL and ray tracing) and SAPIEN Asset (the PartNet-Mobility dataset together with a robot model library). The three are integrated in a unified framework that exposes a Python API and a ROS interface.

SAPIEN Engine — 物理仿真核心SAPIEN Engine — Physics Simulation Core

SAPIEN 使用开源的 Nvidia PhysX 4.1 提供精确的关节体物理仿真，支持机器人操作系统（ROS）。提供三种关节体类型：kinematic joint system（运动学关节，提供运动对象约束）、dynamic joint system（动态关节，模拟受力变化）以及 PhysX articulation（关节铰链）。仿真器可在单核上以约 5000Hz 运行，使用 OpenGL 渲染约 700Hz，支持同步与异步两种运行模式，后者允许仿真与客户端算法独立运行。SAPIEN uses the open-source Nvidia PhysX 4.1 to provide accurate physics simulation of articulated objects and supports the Robot Operating System (ROS). Three joint types are offered: kinematic joint system (kinematic joints, which impose constraints on moving objects), dynamic joint system (dynamic joints, which model force-driven changes) and PhysX articulation (articulated hinges). The simulator runs at about 5000Hz on a single core, and at about 700Hz when rendering with OpenGL; it supports both synchronous and asynchronous modes, the latter allowing the simulation and the client algorithm to run independently.

SAPIEN Renderer — 多模态渲染SAPIEN Renderer — Multi-modal Rendering

SAPIEN Renderer 支持 OpenGL 4.5 和 GLSL，提供 RGB 图像、深度图、法线图以及语义分割图的同步输出。还集成了基于 OmniNavi diffuse model 的 GGX specular model，并通过将 OpenGL 替换为 Nvidia OptiX 光线追踪引擎，以较低渲染速率（约 1 帧/步）生成高质量照片真实感图像（见图 1 上方）。SAPIEN Renderer supports OpenGL 4.5 and GLSL, providing synchronized output of RGB images, depth maps, normal maps and semantic segmentation maps. It also integrates a GGX specular model built on the OmniNavi diffuse model, and by replacing OpenGL with the Nvidia OptiX ray-tracing engine it produces high-quality photorealistic images at a lower rendering rate (about 1 frame per step; see the top of Figure 1).

PartNet-Mobility 数据集 — 大规模关节体资产PartNet-Mobility Dataset — Large-scale Articulated Object Assets

SAPIEN Asset 提供 PartNet-Mobility 数据集：涵盖 46 个物体大类的 2,346 个三维交互模型，每个模型均标注了部件级关节（运动类型：平移、转动）和关节极限。模型使用 URDF 格式描述，支持 PhysX 物理模拟；接触采用网格分解方案，确保在仿真中精确处理碰撞。针对每个可动部件，还提供了运动限制的相对位置标注（作为 part states），以及针对螺钉等特殊运动的处理（combined hinge+slider = screw）。SAPIEN Asset provides the PartNet-Mobility dataset: 2,346 3D interactive models covering 46 object categories, where every model is annotated with part-level joints (motion types: translation and rotation) and joint limits. The models are described in URDF format and support PhysX physics simulation; contacts adopt a mesh decomposition scheme that ensures collisions are handled precisely in simulation. For each movable part, relative-position annotations of the motion limits are also provided (as part states), together with handling of special motions such as screws (combined hinge+slider = screw).

机器人接口与控制Robot Interface and Control

SAPIEN 通过 ROS 接口提供传感器数据、控制指令和运动规划能力。在最低层，力和力矩可直接作用于关节，类似 OpenAI Gym；ROS 控制器提供关节空间和笛卡尔坐标空间控制；高层提供基于 MoveIt! 的轨迹规划和逆运动学求解。客户端 API 统一了接口，使仿真机器人与真实机器人使用完全相同的 ROS API，无需额外适配即可部署。SAPIEN exposes sensor data, control commands and motion planning capabilities through a ROS interface. At the lowest level, forces and torques can be applied directly to joints, similar to OpenAI Gym; ROS controllers provide joint-space and Cartesian-space control; at the high level, MoveIt!-based trajectory planning and inverse kinematics solving are available. The client API unifies these interfaces, so that a simulated robot and a real robot use exactly the same ROS API and deployment requires no extra adaptation.

03 实验Experiments

论文在两类核心视觉任务上评估 SAPIEN：Movable Part Detection（可动部件检测）和 Motion Attributes Estimation（运动属性估计），并展示两类机器人交互任务（door/drawer 开合）的启发式方法与强化学习结果。The paper evaluates SAPIEN on two core vision tasks, Movable Part Detection and Motion Attributes Estimation, and presents heuristic and reinforcement-learning results on two robotic interaction tasks (opening and closing doors/drawers).

任务一：可动部件检测Task 1: Movable Part Detection

使用 Mask R-CNN（以 2D RGB 图像为输入）和 PartNet-InsSeg（以 3D RGB-D partial scans 为输入）两种算法在 PartNet-Mobility 数据集上评估。数据集：46 个类别共 2,346 个物体，使用 75%（1,772 个形状）训练、25%（574 个形状）测试；从随机视角采样 20 张 512×512 图像，在仿真中用简单照明渲染（RGB、深度、法线、语义分割图）。评估指标为 per-part-category Average Precision（AP）及全类平均 mAP。Two algorithms are evaluated on the PartNet-Mobility dataset: Mask R-CNN (taking 2D RGB images as input) and PartNet-InsSeg (taking 3D RGB-D partial scans as input). Dataset: 2,346 objects across 46 categories, with 75% (1,772 shapes) for training and 25% (574 shapes) for testing; 20 images of 512×512 are sampled from random viewpoints and rendered in simulation with simple lighting (RGB, depth, normal and semantic segmentation maps). The evaluation metrics are per-part-category Average Precision (AP) and the mean mAP over all categories.

结果表明：两种方法在检测小型可动部件（如桌脚轮 table wheel 和桌子脚轮 table caster）上表现较差；对具有相对均衡尺寸分布的物体类别（如风扇 fan 和水龙头 faucet）效果更好。The results show that both methods perform poorly on detecting small movable parts (such as the table wheel and the table caster), and work better on object categories with a relatively balanced size distribution (such as the fan and the faucet).

任务二：运动属性估计Task 2: Motion Attributes Estimation

针对关节体的运动属性估计任务，论文考虑两类刚性运动：3D rotation（3D 旋转）和 3D translation（3D 平移）。对平移运动用 3 维向量表示方向；对旋转运动用两个 3 维向量分别表示旋转轴方向和轴上一点的相对位置。评估指标包括：方向差（direction diff）、旋转轴起点差（origin diff）、以及旋转角误差与平移距离误差。For motion attribute estimation on articulated objects, the paper considers two kinds of rigid motion: 3D rotation and 3D translation. Translational motion is represented by a 3-dimensional vector giving its direction; rotational motion is represented by two 3-dimensional vectors giving the direction of the rotation axis and the relative position of a point on the axis. The evaluation metrics include direction diff, origin diff, and the errors of rotation angle and translation distance.

实验表明加入深度信息对方向估计和旋转轴定位有明显帮助；增加视角数量（从 2 到 5）显著提升各项指标。The experiments show that adding depth information clearly helps direction estimation and rotation-axis localization, and that increasing the number of views (from 2 to 5) significantly improves every metric.

机器人交互任务Robotic Interaction Tasks

论文在 SAPIEN 中演示了两类机器人操作任务的执行：The paper demonstrates the execution of two kinds of robotic manipulation task in SAPIEN:

• Door Opening and Drawer-pulling（启发式方法）：给定抓取点，智能体通过 Position-Based Control 沿圆弧轨迹开门，或沿直线轨迹拉抽屉。使用 Franka Gripper 进行抓取实验，在门/抽屉任务上分别取得 81.7% 和 84.3% 的成功率（成功标准：关节打开距离超过目标长度 80%）。Door Opening and Drawer-pulling (heuristic method): given a grasp point, the agent opens a door along an arc trajectory, or pulls a drawer along a straight trajectory, using Position-Based Control. Grasping experiments with the Franka Gripper achieve success rates of 81.7% and 84.3% on the door and drawer tasks respectively (success criterion: the joint is opened beyond 80% of the target length).
• 强化学习（RL）：使用 PPO 训练智能体在给定 2D 图像观测下执行开门任务，在 1M 步训练后达到 81.7% success rate（收敛值），展示了 SAPIEN 作为 RL 训练环境的可用性。Reinforcement learning (RL): PPO is used to train an agent to perform the door-opening task from 2D image observations, reaching 81.7% success rate (converged value) after 1M training steps, which demonstrates the usability of SAPIEN as an RL training environment.

04 局限性Limitations