Dexterous Visuotactile Policy with Generative Contact Grounding
Zhengtong Xu, Yeping Wang, Ben Abbatematteo, Jom Preechayasomboon, Sonny Chan, Nick Colonnese, Amirhossein H. Memar Purdue University · Meta Reality Labs Research · University of Wisconsin–Madison
本文提出 Contact-Grounded Policy（CGP）——一种面向灵巧接触丰富操作任务的视触觉策略学习框架。
CGP 通过联合预测机器人实际状态与触觉反馈的耦合轨迹，并利用学习到的 contact-consistency mapping 将预测结果转化为兼容柔顺控制器的可执行目标状态，
从而实现对演化中多点接触的显式"锚定"，而非仅将触觉信号用作辅助观测。This paper introduces Contact-Grounded Policy (CGP)—a visuotactile policy learning framework for dexterous, contact-rich manipulation tasks. By jointly predicting coupled trajectories of the robot's proprioceptive state and tactile feedback, and by using a learned contact-consistency mapping to convert the predictions into executable target states compatible with a compliance controller, CGP explicitly "grounds" the evolving multi-point contact instead of merely using tactile signals as auxiliary observations.
arXiv 2603.05687仿真：UR5 + DG-5F 五指手（20-DoF）Simulation: UR5 + DG-5F five-fingered hand (20-DoF)实体：Franka Panda + Allegro V5（16-DoF）Real robot: Franka Panda + Allegro V5 (16-DoF)论文链接 →Paper link →
灵巧操作需要对高维、动态变化的多点接触进行持续调控。现有方法在处理接触丰富任务时均存在明显短板：
以抓取为中心的流水线在完成抓取后限制了手指运动；强化学习面临繁琐的奖励工程与 sim-to-real 迁移难题；
模仿学习虽可扩展，但在接触丰富任务上表现不佳。Dexterous manipulation requires continuous regulation of high-dimensional, dynamically evolving multi-point contact. Existing approaches all show clear shortcomings on contact-rich tasks: grasp-centric pipelines restrict finger motion once the grasp is established; reinforcement learning suffers from tedious reward engineering and sim-to-real transfer difficulties; imitation learning is scalable but performs poorly on contact-rich tasks.
"Policies must go beyond using tactile signals as additional observations and instead model contact state and how action outputs interact with low-level controller dynamics."
—— 论文第 I 节，Introduction— Paper, Section I, Introduction
图 2：CGP 整体流程。
CGP 通过预测机器人实际状态与触觉反馈的耦合轨迹来锚定多点接触，
并使用学习到的 contact-consistency mapping 将预测结果转化为柔顺控制器可执行的目标机器人状态。Figure 2: Overall CGP pipeline. CGP grounds multi-point contact by predicting coupled trajectories of the robot's proprioceptive state and tactile feedback, and uses a learned contact-consistency mapping to convert the predictions into target robot states executable by the compliance controller.
93.3%真实机器人 Jar Opening 成功率（CGP）Real-robot Jar Opening success rate (CGP)
+13.3~26.7%真实任务相对 Visuotactile DP 的提升幅度Improvement over Visuotactile DP on the real-robot tasks
5 Hz策略推理频率（8步 DDIM 降噪）Policy inference frequency (8-step DDIM denoising)
三种现有范式的对比Comparison of Three Existing Paradigms
策略范式Policy paradigm
可执行接触建模Actionable contact modeling
多指手支持Multi-fingered hand support
分布式接触可扩展性Distributed-contact scalability
Adaptive Compliance Policies
✓
受限（单末端执行器）Limited (single end-effector)
✗
Sparse Fingertip Force Policies
✗
✓
有限Limited
Contact-Grounded Policy (CGP)
✓
✓
✓
02方法Method
CGP 将灵巧操作建模为一个接触锚定问题。核心思路是：在特定触觉传感器与柔顺控制器配置下，
接触状态可由三元组 (实际机器人状态 x，触觉反馈 u，控制器参考目标 a) 隐式表达，
无需显式建模接触位置或接触模式。
策略由两个耦合模块组成：contact-consistency mapping 与条件轨迹生成器（diffusion-based）。CGP formulates dexterous manipulation as a contact-grounding problem. The core idea is that, under a given tactile sensor and compliance controller configuration, the contact state can be expressed implicitly by the triplet (proprioceptive robot state x, tactile feedback u, controller reference target a), without explicitly modeling contact locations or contact modes. The policy consists of two coupled modules: the contact-consistency mapping and a conditional trajectory generator (diffusion-based).
图 3：遥操作数据采集流程。
仿真环境使用 Meta Quest 3 头显进行基于 VR 的手部追踪；真实机器人使用 OptiTrack 动捕系统与仪器化手套。
两个场景共享相同的 retargeting 与控制栈架构。Figure 3: Teleoperation data collection pipeline. The simulation environment uses a Meta Quest 3 headset for VR-based hand tracking; the real robot uses an OptiTrack motion capture system together with an instrumented glove. Both settings share the same retargeting and control-stack architecture.
Contact-Consistency Mapping Mφ
该映射将（实际机器人状态 xt，触觉反馈 ut）映射为柔顺控制器可执行的目标状态 at：The mapping converts (proprioceptive robot state xt, tactile feedback ut) into a target state at executable by the compliance controller:
at = Mφ(xt, ut)
采用残差映射（输出当前实际状态的偏移量而非绝对目标）以改善条件化效果与鲁棒性。
触觉编码器使用 ResNet 风格架构（优于 MLP 和 Transformer 变体）。
该映射以纯数据驱动方式学习，灵活适配分布式演化多点接触。A residual mapping is adopted (outputting an offset from the current proprioceptive state rather than an absolute target) to improve conditioning and robustness. The tactile encoder uses a ResNet-style architecture (better than MLP and Transformer variants). The mapping is learned in a purely data-driven manner and flexibly accommodates distributed, evolving multi-point contact.
使用带 KL 正则化的变分自编码器（VAE）将原始触觉观测压缩为紧凑的潜在表示：
仿真中触觉阵列压缩至 32 维，真实硬件的 Digit360 传感器压缩至 80 维（每传感器 20 维）。A variational autoencoder (VAE) with KL regularization compresses raw tactile observations into a compact latent representation: the tactile array is compressed to 32 dimensions in simulation, and the Digit360 sensors on the real hardware to 80 dimensions (20 dimensions per sensor).
耦合扩散模型（Coupled Diffusion）在潜在空间中联合预测未来的机器人状态轨迹与触觉潜在状态轨迹，
采用 DDPM/DDIM 训练与采样。预测视野 T=16 步，执行视野 Ta=8 步，以滚动时域方式执行。Coupled Diffusion jointly predicts future robot state trajectories and tactile latent trajectories in the latent space, trained and sampled with DDPM/DDIM. The prediction horizon is T=16 steps and the execution horizon Ta=8 steps, executed in a receding-horizon fashion.
KL 正则化虽略微提升重建误差，但能产生更紧凑、更结构化的潜在空间，
显著改善基于扩散的预测稳定性和下游策略性能。KL regularization slightly increases the reconstruction error, yet yields a more compact and more structured latent space, markedly improving the stability of diffusion-based prediction and downstream policy performance.
实验涵盖 5 个接触丰富的灵巧操作任务（3 个仿真 + 2 个真实），并与两条基线进行对比：
Visuotactile DP（视触觉 diffusion policy）和 Visuomotor DP（纯视觉 diffusion policy）。
仿真任务在最后 5 个 checkpoint 上各评估 250 条轨迹（取均值），真实任务各评估 15 条连续轨迹。The experiments cover 5 contact-rich dexterous manipulation tasks (3 in simulation + 2 on real robots) and compare against two baselines: Visuotactile DP (visuotactile diffusion policy) and Visuomotor DP (vision-only diffusion policy). Each simulation task is evaluated over 250 rollouts on each of the last 5 checkpoints (averaged); each real-robot task is evaluated over 15 consecutive rollouts.
DP = diffusion policy。仿真任务报告最后 5 个 checkpoint 在 250 条轨迹上的平均成功率；真实任务报告 15 条连续轨迹的成功率。DP = diffusion policy. Simulation tasks report the mean success rate over 250 rollouts across the last 5 checkpoints; real-robot tasks report the success rate over 15 consecutive rollouts.
图 4：CGP 在三个仿真任务上的推理快照，展示时间对齐的预测触觉与观测触觉。
在每个推理步，扩散模型预测未来 16 步的触觉反馈与实际状态，映射为目标状态后执行 8 步，
再进入下一轮推理。预测触觉与后续实际执行结果高度吻合，说明 CGP 确实执行了接触锚定的目标，
实现了预测中的接触演化。Figure 4: Inference snapshots of CGP on the three simulation tasks, showing time-aligned predicted and observed tactile signals. At each inference step, the diffusion model predicts tactile feedback and proprioceptive states 16 steps into the future, maps them into target states, executes 8 steps, and then proceeds to the next inference round. The predicted tactile signals agree closely with what is subsequently executed, indicating that CGP does execute contact-grounded targets and realizes the predicted contact evolution.
接触一致性映射验证（Hand Configuration Prediction）Validating the Contact-Consistency Mapping (Hand Configuration Prediction)
使用 150 条遥操作抓取演示（4,114 帧，11 个物体）在仿真中进行独立测试，
数据以 1:1 的激进划分策略在 episode 层面切分以避免数据泄漏。
结果显示：同时输入机器人实际状态与触觉反馈的预测精度远优于任一单模态输入，
验证了接触锚定假设。ResNet 风格触觉编码器优于 MLP 和 Transformer 变体；残差映射优于绝对位置预测。A standalone test in simulation uses 150 teleoperated grasping demonstrations (4,114 frames, 11 objects), with the data split at the episode level under an aggressive 1:1 partition to avoid data leakage. The results show that feeding both the proprioceptive robot state and tactile feedback yields far better prediction accuracy than either single modality alone, validating the contact-grounding hypothesis. The ResNet-style tactile encoder outperforms MLP and Transformer variants; the residual mapping outperforms absolute position prediction.
图 5：Contact-consistency mapping 对未见过抓取构型的手部配置预测。
该可视化提供了高层次证据：接触可以通过该映射以在多样化接触配置间泛化的方式被一致地表达。Figure 5: Hand configuration prediction of the contact-consistency mapping on unseen grasp configurations. This visualization provides high-level evidence that contact can be consistently expressed through the mapping in a way that generalizes across diverse contact configurations.
触觉压缩与消融分析Tactile Compression and Ablation Analysis
图 6：KL 正则化对触觉压缩的消融结果。
移除 KL 正则化虽略微降低重建误差，但使潜在空间失去结构性（KL 散度大幅增加），
并在后续策略推理中一致地降低滚出成功率。MAE = mean absolute error；Recon. = reconstruction。Figure 6: Ablation of KL regularization for tactile compression. Removing KL regularization slightly lowers the reconstruction error, but leaves the latent space unstructured (the KL divergence increases substantially) and consistently reduces rollout success rates in the subsequent policy inference. MAE = mean absolute error; Recon. = reconstruction.
推理效率Inference Efficiency
图 7 报告了在 NVIDIA A100 80GB GPU 上，50 次推理运行的平均时延。
尽管 CGP 需要额外建模未来触觉反馈和接触一致目标，
其推理延迟与 visuomotor 和 visuotactile diffusion-policy 基线相当，
满足 5 Hz 实时推理要求（8步 DDIM 降噪）。Figure 7 reports the average latency over 50 inference runs on an NVIDIA A100 80GB GPU. Although CGP has to additionally model future tactile feedback and contact-consistent targets, its inference latency is comparable to the visuomotor and visuotactile diffusion-policy baselines, meeting the 5 Hz real-time inference requirement (8-step DDIM denoising).
图 7：推理时延对比。
三种方法（visuomotor DP 基线、visuotactile DP 基线、CGP）的平均推理时间对比。
所有共享网络模块使用相同的架构与参数量以保证公平性。Figure 7: Inference latency comparison. Average inference time of the three methods (visuomotor DP baseline, visuotactile DP baseline, CGP). All shared network modules use the same architecture and parameter count to ensure a fair comparison.
04局限性Limitations
说明：以下局限性均来自论文第 VI 节（Limitations and Future Work）中作者的明确陈述（stated），并非推断。Note: All limitations below are explicitly stated by the authors in Section VI of the paper (Limitations and Future Work); none are inferred.
"The contact-consistency mapping relies on tactile observations and is learned under a particular compliance controller,
so it does not readily transfer across sensor types or controller configurations."
具体而言，CGP 为每种传感器从头训练，更换传感器类型需要重新训练或适配，
跨传感器和跨控制器的接触锚定仍具挑战性。"The contact-consistency mapping relies on tactile observations and is learned under a particular compliance controller, so it does not readily transfer across sensor types or controller configurations." Concretely, CGP is trained from scratch for each sensor; switching sensor type requires retraining or adaptation, and cross-sensor, cross-controller contact grounding remains challenging.
有限的 Sim-to-Real 迁移（Limited Sim-to-Real Transfer）Limited Sim-to-Real Transfer
当前评估通过独立的仿真实验和真实部署分别进行，而非直接的 sim-to-real 迁移。
真实触觉传感器（Digit360 视觉触觉传感器）与仿真中的力阵列在传感原理上存在根本差异，
使得直接迁移困难。论文将仿真与真实训练分离，未验证仿真预训练是否能有效迁移至真实。The current evaluation is carried out through separate simulation experiments and real-robot deployments rather than direct sim-to-real transfer. The real tactile sensors (Digit360 vision-based tactile sensors) differ fundamentally in sensing principle from the force arrays used in simulation, which makes direct transfer difficult. The paper keeps simulation and real-robot training separate and does not verify whether simulation pretraining transfers effectively to the real robot.
单任务训练（Single-Task Training）Single-Task Training
当前方法验证了接触锚定在单个任务上的有效性，
但未探索跨具有不同目标和接触模式的任务之间的迁移能力。
论文建议未来方向之一是通过跨任务联合训练扩展到更广泛的任务分布。The current method validates the effectiveness of contact grounding on a single task, but does not explore transfer across tasks with different goals and contact modes. The paper suggests that one future direction is to scale to a broader task distribution through cross-task joint training.
未来方向（Future Work）Future Work
通过控制器参数条件化实现跨传感器与跨控制器的联合训练Joint training across sensors and controllers by conditioning on controller parameters
结合领域随机化的仿真+真实联合训练（sim-and-real co-training）Sim-and-real co-training combined with domain randomization
通过跨任务联合训练扩展到更广泛的任务分布Scaling to a broader task distribution through cross-task joint training
