SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation
一个通用策略，零样本迁移至真实世界的灵巧工具操控A single general policy that transfers zero-shot to real-world dexterous tool manipulation
Kushal Kedia · Tyler Ga Wei Lum · Jeannette Bohg · C. Karen Liu | Stanford University · 2026.02
工具操控要求机器人同时做到：从平面上抓取细薄物体、在手内将其转向功能姿态、以及在施力交互中保持稳定抓持——这对并联夹爪而言几乎无解，对灵巧多指手而言也极具挑战。SimToolReal 的核心洞见是将工具使用抽象为将工具运动至任意目标位姿的 object-centric 问题，在仿真中训练一个单一的 goal-conditioned RL policy，再零样本部署至真实工具，无需任何特定工具或任务的额外训练。Tool manipulation requires a robot to do three things at once: pick a thin, flat object off a surface, reorient it in-hand into a functional pose, and hold a stable grasp under force-rich interaction — nearly hopeless for a parallel-jaw gripper and highly challenging even for a dexterous multi-finger hand. The key insight of SimToolReal is to abstract tool use as the object-centric problem of moving the tool to an arbitrary goal pose, training a single goal-conditioned RL policy in simulation and then deploying it zero-shot on real tools, without any tool- or task-specific additional training.
工具是人类能力的放大器——锤子、刷子、画笔、螺丝刀……每件工具都显著扩展了机器人可执行任务的范围。但工具操控同时要求三项高难度技能并存：抓取薄而扁平的物体、手内旋转至功能构型、在施力过程中维持稳定抓持。遥操作数据采集对此极为困难，而现有的 sim-to-real RL 方案通常需要为每个物体和任务单独建模、单独调节 reward，工程量巨大且无法跨任务泛化。Tools are amplifiers of human capability — hammers, brushes, markers, screwdrivers... each one substantially widens the range of tasks a robot can perform. But tool manipulation demands three difficult skills simultaneously: grasping thin, flat objects, rotating them in-hand into a functional configuration, and maintaining a stable grasp while exerting force. Collecting teleoperation data for this is extremely hard, while existing sim-to-real RL approaches usually require per-object and per-task modeling and reward tuning — an enormous engineering effort that still fails to generalize across tasks.
"Instead of focusing on a single object and task, we procedurally generate a large variety of tool-like object primitives in simulation and train a single RL policy with the universal goal of manipulating each object to random goal poses. This approach enables SimToolReal to perform general dexterous tool manipulation at test-time without any object or task-specific training."
图 1 · SimToolReal 框架概览。顶部展示在真实环境中对从未见过的工具（从细薄的画笔到厚重的锤子）进行零样本部署；底部展示工具使用的三个典型阶段：从平面抓取、手内重定向、执行任务动作。Figure 1 · Overview of the SimToolReal framework.Top: zero-shot deployment in the real world on tools never seen before, from a slender marker to a heavy hammer; bottom: the three canonical stages of tool use — grasping from a surface, in-hand reorientation, and executing the task motion.
1个统一策略（无任何物体/任务特定训练）Unified policy (no object- or task-specific training)
02方法Method
SimToolReal 将工具操控统一建模为 object-centric goal-reaching：给定当前物体 6D 位姿与目标位姿，策略输出关节角度目标，驱动机器人将物体从当前位姿运动至目标位姿。训练完全在仿真中对程序生成的 primitive objects 进行，推理阶段从人类示范视频中提取工具轨迹作为目标位姿序列，直接零样本部署。SimToolReal casts tool manipulation uniformly as object-centric goal-reaching: given the object's current 6D pose and a goal pose, the policy outputs joint angle targets that drive the robot to move the object from its current pose to the goal pose. Training takes place entirely in simulation on procedurally generated primitive objects; at inference time, tool trajectories extracted from human demonstration videos serve as the goal pose sequence, and the policy is deployed zero-shot.
图 2 · 整体框架。训练阶段（上）：在仿真中对程序生成的各种 primitive objects 训练 goal-conditioned RL policy，目标为将物体操纵至随机采样的目标位姿。推理阶段（下）：将此策略零样本部署至 DexToolBench 真实工具，跟随从人类视频中提取的工具轨迹。Figure 2 · Overall framework.Training (top): a goal-conditioned RL policy is trained in simulation on a wide variety of procedurally generated primitive objects, with the objective of manipulating each object to randomly sampled goal poses. Inference (bottom): the same policy is deployed zero-shot on the real DexToolBench tools, following tool trajectories extracted from human videos.
o_t ∈ SE(3)：当前物体位姿o_t ∈ SE(3): the object's current pose
φ：物体描述符（6D 位姿 + 3D 抓取 bounding box 的中心与尺寸，均在物体坐标系下）φ: object descriptor (6D pose plus the center and dimensions of the 3D grasp bounding box, all expressed in the object frame)
g ∈ SE(3)：目标位姿g ∈ SE(3): the goal pose
Reward 设计为：r = r_smooth + r_grasp + I_grasped × r_goal，其中 r_goal = max(d* - d(o_t, g), 0) + B_succ × I[d(o_t, g) < ε]。这一设计使 reward 无需针对具体工具或任务进行调整，具有高度通用性。The reward is defined as r = r_smooth + r_grasp + I_grasped × r_goal, where r_goal = max(d* - d(o_t, g), 0) + B_succ × I[d(o_t, g) < ε]. This design makes the reward highly general: it needs no tuning for a specific tool or task.
仿真中的训练物体为大量程序生成的"handle + head"组合体，使用圆柱体与长方体搭建，变化维度包括尺寸、质量分布等。这些 primitive objects 无需与真实工具完全对齐，其多样性足以激发策略学习到在真实任务中所需的核心操控技能。The training objects in simulation are a large set of procedurally generated "handle + head" assemblies built from cylinders and cuboids, varied along dimensions such as size and mass distribution. These primitive objects need not align exactly with real tools; their diversity is enough to elicit the core manipulation skills required in real tasks.
SAPG 优化与 Asymmetric CriticSAPG Optimization and the Asymmetric Critic
训练采用 SAPG（Scalable Actor-Prior Guided）优化器替代标准 PPO，其基于种群的探索机制对于灵巧操控所需的多模态动作空间探索至关重要。同时使用 Asymmetric Critic：Critic 访问仿真中的特权信息（如物体真实质量、接触力），Actor 仅使用可在真实世界获取的观测——这一设计显著提升了 value function 的估计精度而不依赖不可观测的状态。此外，策略采用 LSTM 主干网络以整合时序信息，并在训练中注入 domain randomization（观测延迟、动作延迟、力/力矩扰动等）。Training uses the SAPG (Scalable Actor-Prior Guided) optimizer in place of standard PPO; its population-based exploration is critical for exploring the multimodal action space that dexterous manipulation requires. An Asymmetric Critic is also used: the critic accesses privileged simulation information (such as the object's true mass and contact forces) while the actor uses only observations obtainable in the real world — this markedly improves the accuracy of the value function without depending on unobservable state. In addition, the policy uses an LSTM backbone to integrate temporal information, and domain randomization (observation delay, action delay, force/torque perturbations, etc.) is injected during training.
图 3 · 真实世界部署流水线。左侧：从 RGB-D 人类示范视频出发，经 SAM 3D 生成物体 3D mesh 并分割 3D 抓取 bounding box，再经 FoundationPose 提取 6D 目标位姿序列（3 Hz 下采样）。右侧：LSTM 策略接收本体感知、物体位姿、抓取 bounding box 与目标位姿，输出 29-DoF 机器人（手臂 + 手部）的关节位置目标。Figure 3 · Real-world deployment pipeline.Left: starting from an RGB-D human demonstration video, SAM 3D generates the object's 3D mesh and segments the 3D grasp bounding box, after which FoundationPose extracts the 6D goal pose sequence (downsampled to 3 Hz). Right: the LSTM policy takes proprioception, object pose, grasp bounding box and goal pose, and outputs joint position targets for the 29-DoF robot (arm + hand).
DexToolBench
论文同时提出 DexToolBench——一个针对灵巧工具操控的评测基准，包含：The paper also introduces DexToolBench — a benchmark for dexterous tool manipulation, comprising:
每个任务配有 RGB-D 人类示范视频和数字孪生仿真环境。评测指标为 Task Progress：策略达到示范目标位姿的百分比，成功阈值 ε = 2 cm。Every task comes with an RGB-D human demonstration video and a digital-twin simulation environment. The evaluation metric is Task Progress: the percentage of demonstration goal poses the policy reaches, with a success threshold of ε = 2 cm.
03实验Experiments
在 DexToolBench 上进行全面评测：120 次真实世界 rollouts（每个任务 5 次），覆盖 24 个任务、12 个物体实例、6 种工具类别。SimToolReal 与三类基线对比：Kinematic Retargeting（运动学重定向）、Fixed Grasp（固定抓取）以及 Specialist RL policies（每个类别单独训练的专家策略）。A comprehensive evaluation is run on DexToolBench: 120 real-world rollouts (5 per task), covering 24 tasks, 12 object instances and 6 tool categories. SimToolReal is compared against three families of baselines: Kinematic Retargeting, Fixed Grasp, and Specialist RL policies trained separately for each category.
真实世界零样本泛化Real-World Zero-Shot Generalization
图 4 · DexToolBench 真实世界泛化结果。每个柱子对应一个任务轨迹在一个物体实例上、5 次 rollouts 的平均 Task Progress。策略在未见过的工具和轨迹上均表现出强泛化能力。橡皮擦类（eraser）以平移为主，Task Progress 最高；细薄的画笔（marker）因姿态追踪难度较大，性能有所下降。Figure 4 · Real-world generalization results on DexToolBench.Each bar is the mean Task Progress over 5 rollouts of one task trajectory on one object instance. The policy generalizes strongly to unseen tools and trajectories. The eraser category is dominated by translation and reaches the highest Task Progress; the slender marker degrades somewhat because its pose is harder to track.
论文原文："SimToolReal outperforms prior retargeting and fixed-grasp methods by 37%"。Kinematic Retargeting 因忽略接触力而无法完成抓取；Fixed Grasp 在无需旋转的变体中可以成功，但一旦要求手内旋转，强制固定抓取会导致手臂与桌面碰撞。From the paper: "SimToolReal outperforms prior retargeting and fixed-grasp methods by 37%". Kinematic Retargeting ignores contact forces and therefore fails to grasp at all; Fixed Grasp succeeds on the variant that needs no rotation, but as soon as in-hand rotation is required, forcing a fixed grasp drives the arm into the table.
图 5 · 基线对比（刷子扫桌任务，两种变体）。左：无需旋转变体；右：需要手内旋转变体。SimToolReal 在两种变体均成功，展现出流畅的手内旋转动作；Fixed Grasp 在需旋转时导致碰撞；Kinematic Retargeting 在两种变体均无法完成抓取。Figure 5 · Baseline comparison (brush sweeping task, two variants).Left: the variant without rotation; right: the variant requiring in-hand rotation. SimToolReal succeeds on both variants and exhibits a fluent in-hand rotation; Fixed Grasp collides when rotation is required; Kinematic Retargeting fails to grasp in either variant.
与专家策略对比（仿真）Comparison with Specialist Policies (Simulation)
图 6 · SimToolReal 与专家策略对比（仿真）。6 个专家策略各自在单一物体（Obj A）+ 单一轨迹（Traj A）上训练。在训练配置（Obj A / Traj A）下专家策略与 SimToolReal 持平；切换至新轨迹（Obj A / Traj B）或新物体（Obj B / Traj A）后，专家策略 Task Progress 显著下降，而 SimToolReal 在所有变体上保持高水准的零样本 Task Progress。Figure 6 · SimToolReal versus specialist policies (simulation).6 specialist policies are each trained on a single object (Obj A) and a single trajectory (Traj A). Under the training configuration (Obj A / Traj A) the specialists match SimToolReal; once switched to a new trajectory (Obj A / Traj B) or a new object (Obj B / Traj A), their Task Progress drops sharply, whereas SimToolReal sustains high zero-shot Task Progress across all variants.
训练目标与泛化的相关性Correlation Between the Training Objective and Generalization
图 7 · 训练目标驱动泛化。左：训练过程中对程序生成物体的 episode reward；右：对未见 DexToolBench 工具的零样本 Task Progress。两条曲线高度同步，验证了核心假设：在多样 primitive objects 上提升随机目标位姿达成能力，可直接驱动对未见工具使用行为的泛化。Figure 7 · The training objective drives generalization.Left: episode reward on procedurally generated objects during training; right: zero-shot Task Progress on unseen DexToolBench tools. The two curves track each other closely, confirming the core hypothesis: improving the ability to reach random goal poses on diverse primitive objects directly drives generalization to unseen tool-use behavior.
Ablation 研究Ablation Studies
论文在 5 个随机种子上进行 ablation，比较以下关键设计决策对训练 reward 的影响：The paper ablates over 5 random seeds, comparing the effect of the following key design decisions on training reward:
SAPG vs. PPO：将 SAPG 替换为标准 PPO 导致"significant drop in performance"——SAPG 的种群式探索机制对灵巧操控的多模态动作空间至关重要。SAPG vs. PPO: replacing SAPG with standard PPO causes a "significant drop in performance" — SAPG's population-based exploration is critical for the multimodal action space of dexterous manipulation.
Asymmetric Critic：移除该设计"severely hinders learning"——特权信息对 value function 的准确估计不可或缺。Asymmetric Critic: removing this design "severely hinders learning" — privileged information is indispensable for accurately estimating the value function.
失败模式分析Failure Mode Analysis
真实世界失败原因分布：Distribution of real-world failure causes:
注：以下局限性部分来自论文显式陈述（标注"作者指出"），部分为从设计中推断（标注"设计推断"）。Note:Some of the limitations below are stated explicitly in the paper (marked "stated by the authors"); others are inferred from its design (marked "inferred from design").
无法保证高力交互任务的功能完成（作者指出）No guarantee of functional task completion in high-force interaction (stated by the authors)
策略以 Task Progress（目标位姿到达率）为优化目标，而非任务功能的最终完成。对于需要精确施力（如钉钉子、拧螺丝到底）的场景，达到目标轨迹位姿不等同于任务成功。The policy optimizes Task Progress (the rate of reaching goal poses) rather than the eventual functional completion of the task. In scenarios that require precise force application — driving a nail home, tightening a screw all the way — reaching the goal trajectory poses is not equivalent to task success.
以物体位姿为目标的环境盲问题（作者指出）Object-pose goals alone are environment-blind (stated by the authors)
"Conditioning on object pose goals alone is environment-blind, which can lead to collisions in cluttered scenes."——策略仅感知物体位姿与目标位姿，不感知周围障碍物，在复杂场景中可能发生碰撞。"Conditioning on object pose goals alone is environment-blind, which can lead to collisions in cluttered scenes." — the policy perceives only the object pose and the goal pose, not the surrounding obstacles, so collisions may occur in cluttered scenes.
不支持非刚体工具（设计推断）Non-rigid tools are not supported (inferred from design)
物体表示依赖 6D 刚体位姿，无法处理几何形变（如弹性刷毛、软管等非刚体工具），从根本上限制了可泛化的工具范围。The object representation relies on a 6D rigid-body pose and cannot handle geometric deformation (elastic bristles, flexible tubing and other non-rigid tools), which fundamentally limits the range of tools it can generalize to.
固定目标序列，无动态重规划（设计推断）Fixed goal sequence, no dynamic replanning (inferred from design)
目标位姿序列从人类示范视频中离线提取，推理阶段按固定顺序条件化执行，无法根据实时感知动态调整策略或跳过不可达目标。The goal pose sequence is extracted offline from human demonstration videos and executed by conditioning on it in a fixed order at inference time; the policy cannot adapt dynamically to real-time perception or skip unreachable goals.
姿态追踪失败是最主要的真实世界失败模式（作者指出）Pose tracking failure is the dominant real-world failure mode (stated by the authors)
真实世界失败中 43.7% 源于 FoundationPose 的姿态追踪丢失。策略的泛化上限在一定程度上受制于感知模块的鲁棒性，而非策略本身的控制能力。43.7% of real-world failures stem from FoundationPose losing pose tracking. The ceiling on the policy's generalization is set in part by the robustness of the perception module rather than by the control capability of the policy itself.
