Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement

01 Motivation

布料操作的挑战在于「无限维配置空间、持续变化的局部几何、严重自遮挡、大量摩擦接触」，这使得状态估计和动作推理都远比刚体操作困难。当前主流的模仿学习 / 机器人基础模型能拟合专家演示，但在部署阶段缺乏自我纠错能力——感知、抓取点选择、时序或动作预测上的小偏差会在接触密集的布料动力学中不断累积，把系统带离目标状态。simulator-in-the-loop 优化此前已在刚体操作中被证明有效（用物理仿真器作为后端 rollout 引擎并行评估候选轨迹、在线精修名义动作），但布料这种可变形物体的 state 和 contact 更难处理，此前少有工作直接把仿真器本身当作推理时的前向动力学模型。The challenge of cloth manipulation lies in its "infinite-dimensional configuration space, continuously changing local geometry, severe self-occlusion, and abundant frictional contact", which makes both state estimation and action reasoning far harder than in rigid-body manipulation. Mainstream imitation learning and robot foundation models can fit expert demonstrations, but lack the ability to self-correct at deployment time: small deviations in perception, grasp-point selection, timing, or action prediction keep accumulating in contact-rich cloth dynamics and carry the system away from the target state. Simulator-in-the-loop optimization has previously been shown effective in rigid-body manipulation (using a physics simulator as a backend rollout engine to evaluate candidate trajectories in parallel and refine nominal actions online), but the state and contact of deformable objects such as cloth are far harder to handle, and little prior work has directly treated the simulator itself as the forward dynamics model at inference time.

Can we directly use deformable-object simulators as inference-time forward models for real-world cloth manipulation?

representative rollouts of the framework on generalization and robustness tasks — Figure 3: Representative rollouts for generalization and robustness evaluation. Time progresses from left to right. 四行分别展示 long-sleeve shirt folding、shorts folding、mid-fold human disturbance 恢复、以及 reverse-diagonal towel folding。The four rows show long-sleeve shirt folding, shorts folding, recovery from mid-fold human disturbance, and reverse-diagonal towel folding.

9/10Single-arm diagonal folding 完整 pipeline 成功率 (Table 3)Single-arm diagonal folding, full-pipeline success rate (Table 3)

8/10Dual-arm symmetric folding 完整 pipeline 成功率 (Table 3)Dual-arm symmetric folding, full-pipeline success rate (Table 3)

95–100%FLASH 在各 K 下的 MPPI 控制成功率 (Table 1)MPPI control success rate of FLASH under each K (Table 1)

2.33 mm²towel real-to-sim per-vertex MSE，最低 (Table 2)towel real-to-sim per-vertex MSE, the lowest (Table 2)

02 Method

系统围绕三根支柱构建一个闭环的 real-to-sim-to-real pipeline：可扩展的合成数据生成与在线 rollout 流水线（基于可变形仿真器 FLASH）；一个只用合成数据训练、把单张 RGB 观测映射为仿真兼容布料状态的 real-to-sim 模块；以及把稀疏网格 rollout 后端与 prior-guided MPPI 耦合起来的在线规划器。The system builds a closed-loop real-to-sim-to-real pipeline around three pillars: a scalable synthetic data generation and online rollout pipeline (based on the deformable simulator FLASH); a real-to-sim module trained only on synthetic data that maps a single RGB observation to a simulation-compatible cloth state; and an online planner that couples a sparse-mesh rollout backend with prior-guided MPPI.

system overview of the simulator-in-the-loop framework — Figure 1: Overview of the proposed framework. Offline, we use FLASH to generate synthetic data for real-to-sim training. Online, RGB observations initialize physics rollouts that refine the prior policy through MPPI for closed-loop hardware execution.

Simulator backbone — FLASH

FLASH 把布料表示为三角网格，顶点是被仿真的自由度，网格连接编码局部形变与表面几何；对接触密集的布料动力学，FLASH 通过在 Signorini–Coulomb 条件上做 non-smooth Newton 迭代来求解摩擦接触，在大变形下保留 stick–slip 行为。其 local-global integrator 和接触公式把主要计算归约为 GPU 友好的稀疏矩阵运算，从而支持在线控制所需的、可重复的稳定 rollout。系统状态 x_t=(q_t, q̇_t) 为网格顶点位置和速度，仿真器按 x_{t+1}=F(x_t, a_t) 推进动力学，在线控制时对候选动作序列并行 rollout 后交给 MPPI 做动作精修。FLASH represents cloth as a triangle mesh, whose vertices are the simulated degrees of freedom and whose mesh connectivity encodes local deformation and surface geometry; for contact-rich cloth dynamics, FLASH resolves frictional contact through non-smooth Newton iterations on the Signorini–Coulomb conditions, preserving stick–slip behavior under large deformation. Its local-global integrator and contact formulation reduce the dominant computation to GPU-friendly sparse matrix operations, thereby supporting the repeatable and stable rollouts required by online control. The system state x_t=(q_t, q̇_t) consists of the mesh vertex positions and velocities, the simulator advances the dynamics as x_{t+1}=F(x_t, a_t), and during online control the candidate action sequences are rolled out in parallel and then handed to MPPI for action refinement.

RGB-native real-to-sim reconstruction

状态估计器 f_θ 从单张 RGB 图像直接预测服装资产上 N 个材料顶点的形变位置，且只用 FLASH 生成的合成数据训练——离线重建每件服装的固定拓扑 rest-shape mesh（SAM 2 分割 + 深度点云 + ball pivoting 三角化 + isotropic remeshing），再在仿真中随机化布料姿态生成 (image, ground-truth vertex) 配对。网络用冻结的 DINOv2 encoder 提取稠密 patch 特征，并维护 N 个可学习的 canonical tokens（每个网格顶点一个），用于编码每个材料点的持久身份与 canonical 位置，二者拼接后通过 self-attention 联合处理。The state estimator f_θ predicts, directly from a single RGB image, the deformed positions of the N material vertices on a garment asset, and is trained only on synthetic data generated by FLASH: offline, a fixed-topology rest-shape mesh is reconstructed for each garment (SAM 2 segmentation + depth point cloud + ball pivoting triangulation + isotropic remeshing), and cloth poses are then randomized in simulation to generate (image, ground-truth vertex) pairs. The network extracts dense patch features with a frozen DINOv2 encoder and maintains N learnable canonical tokens (one per mesh vertex), which encode the persistent identity and canonical position of each material point; the two are concatenated and then processed jointly through self-attention.

Prior-guided MPPI refinement

给定 base policy 产生的名义动作序列 Ū_t，系统把它当作 task-level prior，在其周围采样 K 个候选动作序列 U_t^(k)=Ū_t+ε^(k)，每个候选在 FLASH 中做 rollout 并按 folding 任务代价（含 workspace 限制、桌面穿透、动作平滑度等正则项）打分，再用 MPPI 把 rollout 代价转换为归一化权重、加权得到精修后的动作序列，并以 receding-horizon 方式执行：机器人执行第一个动作、获得新的 RGB 观测，循环往复地纠正 base policy 的局部执行误差与布料状态偏差。Given the nominal action sequence Ū_t produced by the base policy, the system treats it as a task-level prior and samples K candidate action sequences U_t^(k)=Ū_t+ε^(k) around it; each candidate is rolled out in FLASH and scored by the folding task cost (including regularization terms such as workspace limits, table penetration, and action smoothness), and MPPI then converts the rollout costs into normalized weights whose weighted combination yields the refined action sequence, executed in a receding-horizon manner: the robot executes the first action, obtains a new RGB observation, and iteratively corrects the local execution errors of the base policy and the deviations of the cloth state.

03 Experiments

实验分四部分：仿真器骨干对比（Genesis / Newton / Isaac Sim / FLASH，在同一 MPPI 控制回路和 cloth asset 上）、状态估计对比（towel 与 long-sleeve shirt 两种资产，对比 DPM / DeFM 基线）、真机 pipeline 组件消融（single-arm diagonal folding 与 dual-arm symmetric folding 两个真实任务），以及跨服装、跨折叠方式和扰动恢复的泛化鲁棒性评估。The experiments consist of four parts: a simulator backbone comparison (Genesis / Newton / Isaac Sim / FLASH, on the same MPPI control loop and cloth asset), a state estimation comparison (two assets, towel and long-sleeve shirt, against the DPM / DeFM baselines), an ablation of the real-robot pipeline components (two real tasks, single-arm diagonal folding and dual-arm symmetric folding), and an evaluation of generalization and robustness across garments, folding variants, and disturbance recovery.

Simulator backbone (Table 1)

在相同布料操作任务和 MPPI 控制器下，FLASH 在所有并行环境数 K∈{64,128,256,512} 上取得 95–100% 的成功率，而 Isaac Sim 和 Newton 均低于 60%；per-vertex MSE 与 EMD 也比另外两者低约一个数量级。原文："FLASH attains 95–100% success across all K while Isaac Sim and Newton stay below 60%, with per-vertex MSE and EMD roughly an order [of magnitude lower]." Isaac Sim 单步略快（1.5–1.9×），但整体轨迹质量明显落后于 FLASH。Under the same cloth manipulation task and MPPI controller, FLASH attains a 95–100% success rate at every parallel-environment count K∈{64,128,256,512}, whereas Isaac Sim and Newton both stay below 60%; its per-vertex MSE and EMD are also roughly an order of magnitude lower than those of the other two. The original text: "FLASH attains 95–100% success across all K while Isaac Sim and Newton stay below 60%, with per-vertex MSE and EMD roughly an order [of magnitude lower]." Isaac Sim is slightly faster per step (1.5–1.9×), but its overall trajectory quality clearly lags behind FLASH.

State estimation (Table 2)

Asset	Method	MSE (mm²) ↓	CD (mm) ↓	EMD (mm) ↓	Latency (ms) ↓
Towel	Ours (RGB)	2.33 ± 1.21	1.94 ± 0.41	1.98 ± 0.45	7.4 ± 0.6
Towel	DPM	7.17 ± 2.22	3.89 ± 0.67	4.01 ± 0.72	318.5 ± 3.7
Towel	DeFM	43.36 ± 34.66	5.51 ± 0.99	6.53 ± 1.52	12.6 ± 0.5
Long-sleeve	Ours (RGB)	24.23 ± 11.49	3.65 ± 0.78	4.67 ± 1.24	12.4 ± 0.4
Long-sleeve	DPM	45.74 ± 14.19	5.93 ± 0.27	8.21 ± 0.65	512.9 ± 6.0
Long-sleeve	DeFM	43.52 ± 22.02	4.05 ± 0.38	5.93 ± 0.91	18.0 ± 0.6

real-world real-to-sim reconstruction comparison against DPM — Figure 2: Real-world real-to-sim reconstruction. Panel (a) towel and (b) long-sleeve top, each comparing our RGB-native estimation with DPM at different folding stages. From top to bottom, the real observation, predicted vertices overlaid on the image, and the reconstructed cloth.

Real-world pipeline-variant evaluation (Table 3)

Task	Real-to-Sim	Backend	Controller	Real SR ↑
Single-arm diagonal	Ours (RGB)	FLASH	MPPI	9/10
Single-arm diagonal	DPM	FLASH	MPPI	6/10
Single-arm diagonal	Ours (RGB)	Isaac Sim	MPPI	5/10
Single-arm diagonal	Ours (RGB)	FLASH	Base-policy	3/10
Dual-arm symmetric	Ours (RGB)	FLASH	MPPI	8/10
Dual-arm symmetric	DPM	FLASH	MPPI	3/10
Dual-arm symmetric	Ours (RGB)	Isaac Sim	MPPI	4/10
Dual-arm symmetric	Ours (RGB)	FLASH	Base-policy	1/10

完整 pipeline 在两个任务上都取得最高成功率，相对 direct base-policy execution 有显著的闭环提升；替换任一组件（state estimator、simulator backbone 或 controller）都会导致成功率下降，说明状态同步、仿真器保真度与在线精修三者是紧密耦合的。The full pipeline achieves the highest success rate on both tasks, a marked closed-loop improvement over direct base-policy execution; replacing any single component (state estimator, simulator backbone, or controller) lowers the success rate, indicating that state synchronization, simulator fidelity, and online refinement are tightly coupled.

Generalization and robustness (Table 4)

Task	Real SR ↑
Long-sleeve shirt folding	8/10
Shorts folding	7/10
Towel, reverse-diagonal folding	8/10
Towel, mid-fold disturbance recovery	8/10

框架在新的服装资产、折叠变体以及 mid-fold 扰动下依然保持较高成功率，且不需要额外的动力学再训练，展示了对任务与资产变化的泛化能力以及从执行扰动中恢复的鲁棒性。The framework still maintains a relatively high success rate on new garment assets, folding variants, and mid-fold disturbances, without any additional dynamics retraining, demonstrating generalization to task and asset variation as well as robustness in recovering from execution disturbances.

04 Limitations

Note: 以下两点均为论文 Section 6 "Limitations and Future Work" 中原文陈述（stated），非本文推断。Both points below are stated in the paper's Section 6, "Limitations and Future Work", and are not inferences of this note.

可变形动力学的并行度受限，闭环控制率被 simulation backend 卡住Parallelism of deformable dynamics is limited; the closed-loop control rate is bottlenecked by the simulation backend

论文指出，尽管 FLASH 在 rollout 效率与物理保真度之间取得了较好平衡，但求解可变形动力学远不如刚体情形那样容易并行化，rollout 很容易占满单 GPU 的算力预算，可达到的 batch size（上界了闭环控制率）仍然受限于仿真后端。作者计划未来通过跨多 GPU 的分布式 rollout 与后端优化来扩展推理规模。The paper states that although FLASH strikes a good balance between rollout efficiency and physical fidelity, solving deformable dynamics is far less readily parallelized than the rigid-body case: rollouts easily saturate the compute budget of a single GPU, and the attainable batch size (which upper-bounds the closed-loop control rate) remains limited by the simulation backend. The authors plan to scale up inference in the future through distributed rollouts across multiple GPUs and backend optimization.

对新服装的泛化受限于两个 per-asset 阶段Generalization to new garments is limited by two per-asset stages

论文指出泛化到新服装仍受限于离线的 cloth-mesh 重建，以及目前按每件服装单独训练的 real-to-sim 模块。作者提出一个有前景的方向：用 image-conditioned mesh foundation model 从单张观测直接生成仿真兼容资产，并配合 category-level 的 real-to-sim 模块，从而去掉 per-garment 训练这一步。The paper states that generalization to new garments is still limited by the offline cloth-mesh reconstruction and by the real-to-sim module, which is currently trained separately for each garment. The authors propose a promising direction: using an image-conditioned mesh foundation model to generate simulation-compatible assets directly from a single observation, together with a category-level real-to-sim module, thereby removing the per-garment training step.