Evaluating Real-World Robot Manipulation Policies in Simulation (SIMPLER)

01 动机Motivation

随着通用机器人操作策略（如 RT-1、RT-2、Octo）的迅速发展，对其进行全面评估的代价也在急剧上升——真实世界评估需要大量人力、时间，且难以复现。本文核心问题是：能否将在真实数据上训练的策略放入仿真环境中评估，并使仿真结果与真实表现保持高度相关？As generalist robot manipulation policies (e.g., RT-1, RT-2, Octo) advance rapidly, the cost of evaluating them comprehensively rises sharply: real-world evaluation demands substantial manpower and time, and is hard to reproduce. The central question of this paper is: can policies trained on real data be evaluated in a simulated environment such that the simulated results stay highly correlated with real-world performance?

"Real-world evaluation of such policies is not scalable and faces reproducibility challenges, which are likely to worsen as policies broaden the spectrum of tasks they can perform."

SIMPLER teaser — real vs. simulated evaluations correlation — **图0：SIMPLER 框架概览。**左侧展示真实机器人评估的繁重流程（上）与 SIMPLER 仿真评估流程（下）；右侧散点图显示在 Google Robot 与 WidowX 机器人上，约 1500 条评估轨迹中，仿真成功率与真实成功率呈现出强相关性，验证了仿真评估作为可靠代理的有效性。**Figure 0: Overview of the SIMPLER framework.** Left: the laborious real-robot evaluation pipeline (top) versus the SIMPLER simulated evaluation pipeline (bottom). Right: scatter plots show that, across about 1500 evaluation trajectories on the Google Robot and WidowX robots, simulated success rates correlate strongly with real success rates, validating simulated evaluation as a reliable proxy.

~1500配对 sim-and-real 评估轨迹paired sim-and-real evaluation trajectories

0.924SIMPLER-VisMatch Pearson r（Google Robot 平均）SIMPLER-VisMatch Pearson r (Google Robot average)

0.056SIMPLER-VisMatch MMRV（越低越好）SIMPLER-VisMatch MMRV (lower is better)

2支持的机器人形态：Google Robot & WidowXSupported robot embodiments: Google Robot & WidowX

现有数字孪生（digital twin）方案需要大量手工建模，不具备规模化潜力。本文的关键洞察是：不需要精确复刻真实环境，只需仿真策略排名与真实排名高度一致即可。为此，需解决两个核心间隙：控制间隙（control gap）和视觉间隙（visual gap）。Existing digital twin approaches require extensive manual modeling and do not scale. The key insight of this paper is that an exact replica of the real environment is unnecessary; it suffices that the policy ranking in simulation agrees closely with the real ranking. To this end, two core gaps must be closed: the control gap and the visual gap.

SIMPLER simulated environments — **图1：SIMPLER 仿真环境套件。**涵盖 Google Robot（RT 系列评估任务：Pick Coke Can、Move Near、Open/Close Drawer）和 BridgeData V2（WidowX）对应的操作任务。所有环境均可通过标准 Gym 接口一行代码导入。**Figure 1: The SIMPLER simulated environment suite.** It covers the Google Robot (RT-series evaluation tasks: Pick Coke Can, Move Near, Open/Close Drawer) and the corresponding BridgeData V2 (WidowX) manipulation tasks. All environments can be imported in a single line of code through the standard Gym interface.

02 方法Method

SIMPLER 通过两个主要技术消除真实与仿真之间的差距：（1）系统辨识（System Identification, SysID）消除控制间隙；（2）Visual Matching（绿幕背景 + 纹理匹配）消除视觉间隙。SIMPLER closes the gap between the real world and simulation with two main techniques: (1) system identification (SysID) removes the control gap; (2) Visual Matching (green-screen background + texture matching) removes the visual gap.

消除控制间隙：System Identification (SysID)Closing the control gap: System Identification (SysID)

控制间隙的核心问题是：在仿真中回放真实动作序列时，机器人末端执行器轨迹与真实不符，导致抓取失败。本文通过最小化如下损失来优化仿真控制器的 PD 参数（刚度 p 与阻尼 d）：The core issue of the control gap is that, when replaying real action sequences in simulation, the robot end-effector trajectory deviates from the real one and grasping fails. This paper optimizes the PD parameters of the simulated controller (stiffness p and damping d) by minimizing the following losses:

平移损失 L_transl：真实与仿真末端位置的 L2 距离均值Translation loss L_transl: the mean L2 distance between real and simulated end-effector positions
旋转损失 L_rot：基于 Frobenius 范数的旋转误差均值Rotation loss L_rot: the mean rotation error based on the Frobenius norm
优化方法：模拟退火（Simulated Annealing），共 3 轮，利用现有开源数据集中的演示轨迹，无需采集新数据Optimization method: simulated annealing, 3 rounds in total, using demonstration trajectories from existing open-source datasets, with no new data collection required

利用离线数据集（如 RT-1 或 BridgeData V2 演示），无需进行额外真实数据采集，即可完成系统辨识。System identification can be completed with offline datasets (e.g., RT-1 or BridgeData V2 demonstrations), without any additional real-world data collection.

System identification results — control gap — **图2：系统辨识效果。**对 Google Robot "Pick Coke Can" 任务，SysID 前（左）仿真轨迹严重偏离真实，导致抓取失败；SysID 后（右）仿真机器人能准确跟踪真实运动，成功复现抓取动作。**Figure 2: Effect of system identification.** On the Google Robot "Pick Coke Can" task, before SysID (left) the simulated trajectory deviates severely from the real one and grasping fails; after SysID (right) the simulated robot tracks the real motion accurately and reproduces the grasp successfully.

消除视觉间隙：Visual MatchingClosing the visual gap: Visual Matching

视觉差异会导致策略在仿真中表现失真。Visual Matching 包含两个步骤：Visual discrepancies distort policy behavior in simulation. Visual Matching consists of two steps:

绿幕背景（Green Screening）：从真实评估视频第一帧中抠除前景（使用在线图像修复工具），将仿真前景（机械臂和操作对象）叠加到真实背景图像上。合成公式：I' = M ⊙ I_sim + (1-M) ⊙ I_realGreen Screening: the foreground is removed from the first frame of the real evaluation video (using an online image inpainting tool), and the simulated foreground (the arm and the manipulated objects) is overlaid onto the real background image. Compositing formula: I' = M ⊙ I_sim + (1-M) ⊙ I_real
纹理匹配（Texture Matching）：对外观差异较大的对象，通过（1）从真实图像中分割对象，（2）对齐仿真物体位姿，（3）将真实纹理"反投影"到仿真网格，实现精准纹理迁移；对机器人手臂则使用 GIMP 等工具直接涂色匹配。Texture Matching: for objects whose appearance differs substantially, precise texture transfer is achieved by (1) segmenting the object from the real image, (2) aligning the pose of the simulated object, and (3) "back-projecting" the real texture onto the simulated mesh; for the robot arm, tools such as GIMP are used to recolor it to match directly.

作为对比，本文还探索了另一种方法：Variant Aggregation——对场景视觉属性（背景、灯光、桌面纹理等）进行大量随机化，通过在多个变体上平均结果来获得鲁棒估计。实验表明 Visual Matching 效果更优。For comparison, the paper also explores an alternative approach: Variant Aggregation — heavily randomizing the visual attributes of the scene (background, lighting, table texture, etc.) and averaging results over many variants to obtain a robust estimate. Experiments show that Visual Matching works better.

**图3：Visual Matching 方法示意。**左列为原始仿真渲染，中列为绿幕合成（真实背景 + 仿真前景），右列为额外进行纹理匹配后的效果，大幅缩小了与真实图像的视觉差距。**Figure 3: Illustration of the Visual Matching approach.** The left column is the raw simulated rendering, the middle column the green-screen composite (real background + simulated foreground), and the right column the result after additional texture matching, which greatly narrows the visual gap to real images.

评估指标：MMRV 与 Pearson rEvaluation metrics: MMRV and Pearson r

传统 Pearson 相关系数存在两个缺陷：（1）只关注线性关系；（2）对真实性能接近的策略极为敏感。本文提出新指标 Mean Maximum Rank Violation (MMRV)（范围 [0,1]，越低越好），通过加权排名违反幅度来更准确地刻画仿真评估质量：The conventional Pearson correlation coefficient has two shortcomings: (1) it captures only linear relationships; (2) it is extremely sensitive to policies whose real performance is close. This paper proposes a new metric, Mean Maximum Rank Violation (MMRV) (range [0,1], lower is better), which characterizes the quality of simulated evaluation more accurately by weighting the magnitude of rank violations:

RankViolation(i, j) = |R_i − R_j| × 𝟙[排名与真实不一致ranking disagrees with the real one]
MMRV = 每个策略最大 RankViolation 的平均值the average of the maximum RankViolation of each policy

03 实验Experiments

在 Google Robot（RT-1、RT-1-X、RT-2-X、Octo-Base）和 WidowX (BridgeData V2) 上进行配对 sim-and-real 评估，对比 SIMPLER 与验证集 MSE 等基线方法的策略排名质量。Paired sim-and-real evaluations are run on the Google Robot (RT-1, RT-1-X, RT-2-X, Octo-Base) and WidowX (BridgeData V2), comparing the policy ranking quality of SIMPLER against baselines such as validation-set MSE.

主要结果：与真实评估的强相关性Main results: strong correlation with real-world evaluation

Real vs SIMPLER success rates on Google Robot tasks — **图4：Google Robot 任务上的真实 vs. SIMPLER 成功率对比。**SIMPLER-VisMatch 各任务均呈现出低 MMRV、高 Pearson r，表明仿真策略排名与真实排名高度一致。**Figure 4: Real vs. SIMPLER success rates on Google Robot tasks.** SIMPLER-VisMatch attains low MMRV and high Pearson r on every task, indicating that the simulated policy ranking agrees closely with the real ranking.

核心数值结果（Google Robot，6个策略检查点对比）：Key quantitative results (Google Robot, comparison over 6 policy checkpoints):

评估方法Evaluation method	Pick Coke Can MMRV ↓	Drawer MMRV ↓	平均 MMRV ↓Mean MMRV ↓	平均 Pearson r ↑Mean Pearson r ↑
Validation MSE	0.412	0.306	0.375	0.308
SIMPLER-VarAgg（ours）SIMPLER-VarAgg (ours)	0.084	0.235	0.143	0.778
SIMPLER-VisMatch（ours）SIMPLER-VisMatch (ours)	0.031	0.027	0.056	0.924

"Simulated manipulation policy evaluation with SIMPLER leads to strong correlation with real-world policy performance, and we recommend SIMPLER-'Visual Matching' as the default approach since it directly minimizes visual discrepancies between real and simulated environments."

分布偏移鲁棒性预测Predicting robustness to distribution shift

本文进一步验证 SIMPLER 能否准确预测策略对分布偏移的敏感性，对比了背景、灯光、干扰物、桌面纹理、相机位姿等 5 类分布偏移轴。关键发现：The paper further verifies whether SIMPLER can accurately predict how sensitive a policy is to distribution shift, comparing 5 axes of distribution shift: background, lighting, distractors, table texture and camera pose. Key findings:

相机位姿变化和桌面纹理变化对策略影响最大，灯光和干扰物影响较小——这一规律在仿真和真实世界中高度一致Changes in camera pose and in table texture affect policies the most, while lighting and distractors matter less — a pattern that is highly consistent between simulation and the real world
固体颜色变化平均导致成功率下降 4%，图案纹理变化导致下降 25%——仿真对应数值为 2% 和 24%，极为吻合Solid color changes lower the success rate by 4% on average and patterned texture changes by 25%; the corresponding simulated numbers are 2% and 24%, an extremely close match
预测新型分布偏移：仿真中 Octo-Base 对机械臂纹理极为敏感（成功率从 0% 到 29.3%），真实实验（用礼品包装纸包裹机械臂）证实了这一预测，而 RT-1-X 则更具鲁棒性Predicting novel distribution shifts: in simulation, Octo-Base is extremely sensitive to the robot arm texture (success rate ranging from 0% to 29.3%), and a real experiment (wrapping the arm in gift wrapping paper) confirmed this prediction, whereas RT-1-X is more robust

消融研究Ablation studies

视觉匹配消融（Google Robot Drawer 任务，3 个策略）：Visual matching ablation (Google Robot Drawer task, 3 policies):

Green Screen	Drawer Matching	Robot Matching	MMRV ↓	Real-Sim Success Gap ↓
✗	✗	✗	0.087	0.272
✓	✗	✗	0.087	0.198
✓	✓	✗	0.142	0.253
✓	✓	✓	0.050	0.136

仅部分应用 Visual Matching（如只匹配抽屉而不匹配机械臂）反而会导致 MMRV 变差，说明场景各部分的视觉一致性至关重要，需整体应用。Applying Visual Matching only partially (e.g., matching the drawer but not the robot arm) actually worsens MMRV, showing that visual consistency across every part of the scene is essential and must be applied as a whole.

控制消融：SIMPLER SysID 方案的控制损失（0.131）和 MMRV（0.031）均优于两个替代参数设置（0.267/0.070 和 0.432/0.100）。Control ablation: the control loss (0.131) and MMRV (0.031) of the SIMPLER SysID solution are both better than those of two alternative parameter settings (0.267/0.070 and 0.432/0.100).

物理属性敏感性（质量、摩擦系数等）：在合理参数范围内，SIMPLER 评估保持稳定（低 MMRV、高 Pearson r）。Sensitivity to physical properties (mass, friction coefficient, etc.): within a reasonable parameter range, SIMPLER evaluation stays stable (low MMRV, high Pearson r).

物理仿真器无关性：在 Isaac Sim 上复现 SIMPLER（Variant Aggregation），同样获得与 SAPIEN 相近的强相关性，说明框架不依赖特定物理引擎。Independence from the physics simulator: reproducing SIMPLER (Variant Aggregation) in Isaac Sim likewise yields a strong correlation close to that of SAPIEN, showing that the framework does not depend on a specific physics engine.

04 局限性Limitations

Note：以下局限性均为论文作者在 Conclusion 部分明确陈述（stated）。Note: all limitations below are explicitly stated by the authors in the Conclusion.

仅针对刚体操作任务（Rigid-Object Manipulation）Restricted to rigid-object manipulation tasks

当前 SIMPLER 环境专注于物理仿真相对简单的刚体抓取与操作任务。许多近期研究已展示了布料、流体等软体对象的操作能力，但这类任务的仿真物理远未成熟，将 SIMPLER 扩展到柔性/软体对象操作（如利用 IPC 等软体物理仿真）是未来的重要研究方向。Current SIMPLER environments focus on rigid-object grasping and manipulation tasks, whose physical simulation is relatively simple. Many recent works have demonstrated manipulation of soft objects such as cloth and fluids, but the simulated physics of such tasks is far from mature; extending SIMPLER to deformable / soft-object manipulation (for example with soft-body physics simulation such as IPC) is an important direction for future research.

绿幕方法仅适用于固定相机，且无法准确模拟阴影等视觉细节Green screening applies only to fixed cameras and cannot accurately simulate visual details such as shadows

"Green-screening" 方法要求场景中的相机位置固定（fixed cameras），无法处理动态视角变化，且在合成图像时难以还原物体阴影、反射等精细视觉特征，可能对部分对视觉敏感的策略造成影响。The "Green-screening" approach requires the camera positions in the scene to be fixed, cannot handle dynamic viewpoint changes, and struggles to restore fine visual features such as object shadows and reflections when compositing images, which may affect some visually sensitive policies.

环境创建流程仍需一定人工干预Environment creation still needs a degree of manual intervention

当前构建 SIMPLER 评估环境的流程（资产整理、场景组装、纹理匹配等）仍需要人工参与，尚未实现全自动化。实现从真实场景图像全自动生成大规模高保真仿真评估环境是雄心勃勃的未来目标，需要场景重建、材质估计等多方面技术的进一步突破。The current pipeline for building SIMPLER evaluation environments (asset curation, scene assembly, texture matching, etc.) still requires human involvement and is not yet fully automated. Fully automatic generation of large-scale, high-fidelity simulated evaluation environments from real scene images is an ambitious future goal that requires further breakthroughs in scene reconstruction, material estimation and other techniques.