Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalization
将 VLA 模型的隐式物理知识解锁，用于接触密集型操作任务的触觉泛化Unlocking the implicit physical knowledge of VLA models for tactile generalization in contact-rich manipulation tasks
Jialei Huang · Shuo Wang · Fanqi Lin · Yihang Hu · Chuan Wen · Yang Gao | 清华大学 · UESTC · 上海交通大学Tsinghua University · UESTC · Shanghai Jiao Tong University
Tactile-VLA 将视觉、语言、动作与触觉感知深度融合，发现 VLA 模型已潜在地编码了物理交互的语义知识——
只需少量示范，即可将其激活，在接触密集型操作任务中实现零样本力控泛化。Tactile-VLA deeply fuses vision, language, action and tactile sensing, and finds that VLA models already latently encode semantic knowledge of physical interaction — a small number of demonstrations suffices to activate it, achieving zero-shot force-control generalization in contact-rich manipulation tasks.
Vision-Language-Action (VLA) 模型凭借强大的视觉-语言先验在机器人操作中展现出令人印象深刻的泛化能力，
但在接触密集型任务中，它们仍无法将抽象的语义意图精确地落地为细粒度的力控交互。
力感知（tactile sensing）是弥补这一鸿沟的关键缺失环节。Vision-Language-Action (VLA) models show impressive generalization in robot manipulation thanks to strong vision-language priors, yet in contact-rich tasks they still cannot ground abstract semantic intent precisely into fine-grained force-controlled interaction. Tactile sensing is the key missing link for closing this gap.
"We advance VLAs' implicit knowledge beyond identifying what to do, towards guiding how to physically interact with real world."
Tactile-VLA 的三大核心能力。(a) 基于语言的力控泛化：将在 USB 插拔任务中学到的力相关副词（如 "gently"）零样本迁移至充电器插拔任务——后者训练时仅有运动数据，无力指令。
(b) 隐式常识推断：无需显式指令，模型自动为铁球施加坚实抓握力，为火龙果施加轻柔抓握力。
(c) 基于触觉推理的自适应恢复：在白板擦拭任务中训练的推理能力，零样本迁移至黑板场景——初次擦拭因力度不足失败后，模型通过 Chain-of-Thought 分析触觉反馈，自主增大力度成功完成任务。The three core capabilities of Tactile-VLA. (a) Language-conditioned force-control generalization: force-related adverbs learned in the USB plugging task (e.g. "gently") transfer zero-shot to the charger plugging task, which was trained with motion data only and no force instructions. (b) Implicit common-sense inference: without any explicit instruction, the model automatically applies a firm grasp to an iron ball and a gentle grasp to a dragon fruit. (c) Tactile-reasoning-based adaptive recovery: the reasoning ability trained on the whiteboard wiping task transfers zero-shot to a blackboard scene — after the first wipe fails for lack of force, the model analyses tactile feedback through Chain-of-Thought and autonomously increases the force to complete the task.
9.13 N"hard" 零样本力控 基线无法区分指令Zero-shot force control for "hard" Baselines cannot tell the instructions apart
80%黑板擦拭成功率（Tactile-VLA-CoT） 所有基线: 0%Blackboard wiping success rate (Tactile-VLA-CoT) All baselines: 0%
少量示范A few demonstrations即可激活 VLM 的物理先验 实现零样本泛化suffice to activate the physical prior of the VLM and achieve zero-shot generalization
问题背景Problem background
当前 VLA 模型（如 π₀、π₀-fast）擅长高层次规划，但在接触密集型场景中缺乏对力控制的精细感知与调节能力。
现有将触觉/力觉引入机器人框架的工作通常将其作为附加感知模态，而非直接参与动作生成的核心要素。
Tactile-VLA 的核心洞见是：VLA 模型的语言骨干已隐式编码了丰富的物理交互知识（例如"softly"与"firmly"对应的力度差异），
只需将触觉传感器与少量示范数据"桥接"进来，便可将这种先验激活并泛化至全新场景。Current VLA models (such as π₀ and π₀-fast) excel at high-level planning, but in contact-rich settings they lack fine-grained perception and regulation of force control. Existing work that brings touch or force sensing into robot frameworks usually treats it as an additional perceptual modality rather than a core element that directly participates in action generation. The key insight of Tactile-VLA is that the language backbone of VLA models already implicitly encodes rich knowledge of physical interaction (for example the difference in force implied by "softly" versus "firmly"); one only needs to "bridge" in a tactile sensor together with a small amount of demonstration data to activate this prior and generalize it to entirely new scenes.
论文聚焦三类泛化能力，分别对应三个研究问题：The paper focuses on three kinds of generalization ability, corresponding to three research questions:
RQ1 (Tactile-Aware Instruction Following)：模型能否从一个任务中学习力相关语言的语义，并零样本迁移至新任务？RQ1 (Tactile-Aware Instruction Following): can the model learn the semantics of force-related language from one task and transfer it zero-shot to a new task?
RQ2 (Tactile-Relevant Common Sense)：模型能否利用 VLM 的常识推断未见物体所需的交互力？RQ2 (Tactile-Relevant Common Sense): can the model use the common sense of the VLM to infer the interaction force required by unseen objects?
RQ3 (Tactile-Involved Reasoning)：触觉反馈能否驱动模型识别任务失败并自主调整策略？RQ3 (Tactile-Involved Reasoning): can tactile feedback drive the model to recognize task failure and adjust its strategy autonomously?
02方法Method
Tactile-VLA 由三个核心模块构成：多模态 Token 级融合策略网络、
混合位置-力控制器（Hybrid Position-Force Controller），
以及用于自适应推理的 Tactile-VLA-CoT 变体。Tactile-VLA is built from three core modules: a token-level multimodal fusion policy network, a Hybrid Position-Force Controller, and the Tactile-VLA-CoT variant for adaptive reasoning.
Tactile-VLA 架构总览。视觉（Vision）、语言（Language）、触觉（Tactile）和本体感受（Proprioception）输入分别经独立编码器处理后，以 token 序列形式拼接为统一前缀 S_t，送入预训练 Vision-Language Model 的 Transformer 主干。非因果注意力机制允许各模态 token 自由交叉注意，生成深度融合的上下文表示。Tactile-Aware Action Expert 在此基础上输出增强动作向量，同时预测目标位置 P_target 与目标接触力 F_target。虚线框为 Tactile-VLA-CoT 变体，通过 Chain-of-Thought 推理模块根据触觉反馈生成修正指令。Overview of the Tactile-VLA architecture. Vision, Language, Tactile and Proprioception inputs are each processed by a separate encoder and then concatenated as a token sequence into a unified prefix S_t, which is fed to the Transformer backbone of a pretrained Vision-Language Model. A non-causal attention mechanism lets tokens of all modalities attend to one another freely, producing a deeply fused contextual representation. On top of this, the Tactile-Aware Action Expert outputs an augmented action vector and predicts both the target position P_target and the target contact force F_target. The dashed box marks the Tactile-VLA-CoT variant, which uses a Chain-of-Thought reasoning module to generate corrective instructions from tactile feedback.
策略架构与学习（Policy Architecture and Learning）Policy Architecture and Learning
模型基于 π₀ 的预训练参数初始化共享组件（ViT 视觉编码器 E'_vis、语言 tokenizer E_lang），
新增一个轻量级 MLP 触觉编码器 E'_ψ，将历史 H 步触觉测量值压缩为单一 token，
与视觉和语言 token 拼接形成前缀序列：Shared components are initialized from the pretrained parameters of π₀ (the ViT visual encoder E'_vis and the language tokenizer E_lang). A lightweight MLP tactile encoder E'_ψ is added, compressing the past H steps of tactile measurements into a single token, which is concatenated with the vision and language tokens to form the prefix sequence:
动作空间扩展为增强向量 a_t，显式包含目标位置 P_target 和目标接触力 F_target。
端到端采用 Conditional Flow Matching (CFM) 目标函数微调，
损失同时惩罚运动维度和力维度的预测偏差，迫使模型将语言细微差别（如"gently"）映射至对应物理力幅（如 0.5 N）。The action space is extended into an augmented vector a_t that explicitly contains the target position P_target and the target contact force F_target. The model is fine-tuned end to end with a Conditional Flow Matching (CFM) objective whose loss penalizes prediction error along both the motion and the force dimensions, forcing the model to map subtle linguistic nuances (such as "gently") to the corresponding physical force magnitude (such as 0.5 N).
策略网络的输出由低层控制器执行。该控制器遵循"位置主导"策略，采用类似阻抗控制的间接力控方法，
将力目标转化为位置命令的自适应调整：The output of the policy network is executed by a low-level controller. This controller follows a "position-dominant" strategy and adopts an indirect force-control method akin to impedance control, turning force targets into adaptive adjustments of the position command:
P_hybrid = P_target + K · ΔF (当 ‖ΔF‖ > τ 时); 否则无调整P_hybrid = P_target + K · ΔF (when ‖ΔF‖ > τ); otherwise no adjustment
其中 ΔF = F_target − F_measured 为力误差，K 为增益矩阵，τ 为平滑阈值。
控制器将外部净力（通过末端执行器笛卡尔位置调节）与内部抓握力（通过夹爪宽度调节）
解耦为两条独立控制通道，从而同时精确管理接触力和抓握力。Here ΔF = F_target − F_measured is the force error, K the gain matrix and τ a smoothing threshold. The controller decouples the external net force (regulated through the Cartesian position of the end-effector) and the internal grasping force (regulated through the gripper width) into two independent control channels, so that contact force and grasping force can be managed precisely at the same time.
Tactile-VLA-CoT 变体激活 VLM 自身预训练解码器的推理能力，以 Chain-of-Thought 方式
将触觉反馈转化为显式内部独白（explicit internal monologue）。
当任务失败时，模型分析失败原因（如"grasping force is sufficient, but normal force is too low"）
并生成修正指令（如"wipe the board again, but apply more downward force"）。
CoT 在固定时间间隔触发，首先判断任务是否成功，若失败则分析原因并输出修正动作指令。The Tactile-VLA-CoT variant activates the reasoning ability of the VLM's own pretrained decoder, turning tactile feedback into an explicit internal monologue in a Chain-of-Thought manner. When a task fails, the model analyses the cause of failure (e.g. "grasping force is sufficient, but normal force is too low") and generates a corrective instruction (e.g. "wipe the board again, but apply more downward force"). The CoT fires at fixed time intervals: it first judges whether the task succeeded, and if it failed, analyses the cause and outputs a corrective action instruction.
Tactile-VLA-CoT 在黑板擦拭任务中的工作流程：感知失败 → 分析触觉反馈 → 生成修正指令 → 重新执行。Workflow of Tactile-VLA-CoT on the blackboard wiping task: perceive failure → analyse tactile feedback → generate a corrective instruction → execute again.
数据采集（Data Collection）Data Collection
传统遥操作因缺乏真实力反馈，导致采集策略无法依赖触觉信息。
作者基于 Universal Manipulation Interface (UMI) 构建专用数据采集装置，
在 UMI 夹爪上增配双高分辨率触觉传感器，可捕获法向力和剪切力，使操作员能够直接感知接触动力学。
数据以 100 Hz 采集触觉反馈、20 Hz 采集视觉数据，并进行时间戳对齐，
最终形成精确同步的多模态 VLA-T 训练数据集。Conventional teleoperation lacks genuine force feedback, so the collected policy cannot rely on tactile information. The authors build a dedicated data-collection rig on top of Universal Manipulation Interface (UMI), adding dual high-resolution tactile sensors to the UMI gripper that capture normal and shear forces, letting the operator perceive contact dynamics directly. Tactile feedback is recorded at 100 Hz and visual data at 20 Hz, and the streams are timestamp-aligned, finally yielding a precisely synchronized multimodal VLA-T training dataset.
03实验Experiments
实验聚焦三类接触密集型操作任务：充电器/USB 插拔与抽取、
桌面抓取（Tabletop Grasping）、擦板（Wiping the Board）。
基线包括 π₀-base 和 π₀-fast，均不具备触觉融合架构。The experiments focus on three kinds of contact-rich manipulation task: charger/USB plugging and unplugging, Tabletop Grasping and Wiping the Board. The baselines are π₀-base and π₀-fast, neither of which has a tactile fusion architecture.
实验一：触觉感知的指令跟随与力控泛化（RQ1）Experiment 1: tactile-aware instruction following and force-control generalization (RQ1)
模型在 USB 插拔任务（Task A）中以含力副词的指令（"softly" / "hard"）训练，
随后零样本迁移至充电器插拔任务（Task B，仅学过运动，无力指令）。
评估指标为成功率（%）和施加插入力（N）。The model is trained on the USB plugging task (Task A) with instructions containing force adverbs ("softly" / "hard"), and is then transferred zero-shot to the charger plugging task (Task B, where only the motion was learned and no force instruction was given). The evaluation metrics are success rate (%) and applied insertion force (N).
模型Model
USB 成功率 (%)USB success rate (%)
充电器成功率 (%)Charger success rate (%)
π₀-base
5
40
π₀-fast
0
25
Tactile-VLA
35
90
力控泛化结果（Table 2）更直接展示了语义-力映射的泛化性：The force-control generalization results (Table 2) demonstrate the generalization of the semantics-to-force mapping even more directly:
Tactile-VLA 正确区分已训练词汇的力度（"softly": 0.51 N vs "hard": 2.57 N），
并对未见副词（"gently" 0.75 N、"firmly" 1.98 N）和超范围指令（"harder" 2.94 N，超过 "hard" 的 2.57 N）做出合理外推。
基线模型在所有条件下施力均无显著差异——说明其缺乏将语言与力相关联的机制。Tactile-VLA correctly distinguishes the force levels of the trained words ("softly": 0.51 N vs "hard": 2.57 N), and extrapolates sensibly to unseen adverbs ("gently" 0.75 N, "firmly" 1.98 N) and to out-of-range instructions ("harder" 2.94 N, exceeding the 2.57 N of "hard"). The baseline models show no significant difference in applied force across any condition, indicating that they lack a mechanism for associating language with force.
(a) 充电器插拔与抽取任务场景。(b) 桌面抓取任务中的训练集（in-domain）物体，按物理属性分为三类：Solid & Heavy（铁方块、电池）、Solid & Light（木块、充电器、塑料件、玩具）、Fragile & Light（火龙果、甜瓜、蓝莓、纸盒）。(a) The charger plugging and unplugging task setting. (b) The in-domain training objects of the tabletop grasping task, split by physical property into three categories: Solid & Heavy (iron block, battery), Solid & Light (wooden block, charger, plastic part, toy), Fragile & Light (dragon fruit, melon, blueberry, paper box).
桌面抓取实验要求机器人根据物体外观自动推断合适的抓握力，
不提供显式力指令。训练集覆盖 6 个 in-domain 物体，测试时引入额外 6 个 out-of-domain 物体。
成功标准：单次抓取，无明显形变。The tabletop grasping experiment requires the robot to infer an appropriate grasping force automatically from the appearance of the object, with no explicit force instruction provided. The training set covers 6 in-domain objects, and 6 additional out-of-domain objects are introduced at test time. Success criterion: grasped in a single attempt without visible deformation.
模型Model
重/坚硬 IDHeavy/Hard ID
重/坚硬 OODHeavy/Hard OOD
轻/坚实 IDLight/Solid ID
轻/坚实 OODLight/Solid OOD
轻/易碎 IDLight/Fragile ID
轻/易碎 OODLight/Fragile OOD
π₀-base
90%
45%
65%
35%
25%
0%
π₀-fast
65%
40%
60%
35%
25%
0%
Tactile-VLA
95%
95%
95%
85%
85%
95%
（注：表中数值为论文 Table 2 各列均值简化呈现，原始数据按单件物体 10 次试验报告，详见原文。）(Note: the values in the table are a simplified presentation of the column means of Table 2 in the paper; the original data are reported per individual object over 10 trials — see the paper for details.)
各类别物体的平均抓握力（5 次试验均值，误差棒为标准差）。Tactile-VLA 对重型物体施加较大力，
对易碎物体施加轻柔力，且对 OOD 物体的力度推断与 ID 物体同样准确——说明 VLM 的常识先验成功转移至触觉模态。Average grasping force for each object category (mean of 5 trials, error bars are standard deviations). Tactile-VLA applies larger forces to heavy objects and gentle forces to fragile ones, and its force inference on OOD objects is as accurate as on ID objects, showing that the common-sense prior of the VLM transfers successfully to the tactile modality.
实验三：触觉推理与自适应控制（RQ3）Experiment 3: tactile reasoning and adaptive control (RQ3)
模型在白板擦拭场景（marker ink）训练，零样本迁移至黑板擦拭（chalk）——后者需要显著更大的力。The model is trained on the whiteboard wiping scene (marker ink) and transferred zero-shot to blackboard wiping (chalk), which requires markedly larger forces.
模型Model
白板（In-Domain）Whiteboard (In-Domain)
黑板（OOD，零样本）Blackboard (OOD, zero-shot)
π₀-base
40%
0%
π₀-fast
45%
0%
Tactile-VLA
80%
15%
Tactile-VLA-CoT
75%
80%
黑板擦拭任务的零样本推理过程。(a) 模型在白板任务上训练。
(b) 零样本迁移至黑板，初次以默认力 3.5 N 擦拭失败。
(c) Tactile-VLA-CoT 通过 Chain-of-Thought 分析触觉反馈，自主将力提升至 6.7 N
（比白板训练数据中的 5 N 高 34%），成功擦除粉笔痕迹。The zero-shot reasoning process on the blackboard wiping task. (a) The model is trained on the whiteboard task. (b) Transferred zero-shot to the blackboard, the first wipe with the default force of 3.5 N fails. (c) Tactile-VLA-CoT analyses the tactile feedback through Chain-of-Thought and autonomously raises the force to 6.7 N (34% higher than the 5 N seen in the whiteboard training data), successfully erasing the chalk marks.
在零样本黑板场景中，Tactile-VLA-CoT 的成功率达 80%，
而所有基线为 0%。
关键推理链路：初次施力 3.5 N → 识别失败（触觉信号显示法向力不足）→ 输出修正指令 → 增力至 6.7 N → 任务成功。In the zero-shot blackboard scene the success rate of Tactile-VLA-CoT reaches 80%, while all baselines score 0%. The key reasoning chain: apply an initial force of 3.5 N → recognize the failure (tactile signals show insufficient normal force) → output a corrective instruction → raise the force to 6.7 N → task succeeds.
04局限性Limitations
注意：论文正文无独立 Limitations 章节。以下各点部分来自结论或方法设计的直接推断（标注为inferred），部分来自论文明确陈述（标注为stated）。Note: The paper has no standalone Limitations section. Some of the points below are direct inferences from the conclusion or the method design (marked inferred), others are explicitly stated in the paper (marked stated).
数据采集依赖专用硬件（stated）Data collection depends on dedicated hardware (stated)
论文明确指出，传统遥操作因缺乏真实力反馈而不适合此类任务。
作者基于 UMI 搭建了专用数据采集装置，配备双高分辨率触觉传感器。
这意味着数据采集门槛较高，需要特定硬件支持，限制了方法在标准遥操作平台上的直接可复现性。The paper states explicitly that conventional teleoperation is unsuitable for such tasks because it lacks genuine force feedback. The authors built a dedicated data-collection rig on top of UMI, equipped with dual high-resolution tactile sensors. This means the barrier to data collection is high and specific hardware is required, which limits the direct reproducibility of the method on standard teleoperation platforms.
评估任务种类有限（inferred）Limited variety of evaluation tasks (inferred)
当前实验仅涵盖三类任务（插拔、抓取、擦板），均为相对受控的桌面场景。
在更复杂、非结构化或需要多步骤接触的真实环境中，
VLM 的物理先验能否持续有效激活仍有待验证。The current experiments cover only three task types (plugging, grasping, board wiping), all of them relatively controlled tabletop scenes. Whether the physical prior of the VLM can still be activated effectively in more complex, unstructured real environments, or ones requiring multi-step contact, remains to be verified.
CoT 推理依赖固定时间间隔触发（inferred）CoT reasoning relies on fixed-interval triggering (inferred)
Tactile-VLA-CoT 的推理在"固定时间间隔"触发，属于简单有效但非最优的触发策略。
对于需要即时响应的快速失败场景，固定间隔可能引入不必要的延迟；
对于慢速稳定操作，频繁触发则可能浪费计算资源。
更智能的事件驱动触发机制（如力突变检测）有望进一步提升系统表现。The reasoning of Tactile-VLA-CoT is triggered at "fixed time intervals", a simple and effective but suboptimal triggering strategy. For fast-failure scenes that need an immediate response, a fixed interval may introduce unnecessary latency; for slow, stable manipulation, frequent triggering may waste computation. A smarter event-driven trigger (such as detecting abrupt force changes) could further improve the performance of the system.
基础 Tactile-VLA 对 OOD 黑板场景泛化有限（stated）The base Tactile-VLA generalizes poorly to the OOD blackboard scene (stated)
在零样本黑板擦拭任务中，基础 Tactile-VLA（无 CoT）的成功率仅为 15%，
而 Tactile-VLA-CoT 达到 80%。这说明仅靠端到端力控学习，
在物理属性显著不同的新场景中，自适应能力受限；
CoT 推理模块是实现跨场景鲁棒泛化的关键补丁，而非基础架构本身就能覆盖。In the zero-shot blackboard wiping task the success rate of the base Tactile-VLA (without CoT) is only 15%, whereas Tactile-VLA-CoT reaches 80%. This shows that end-to-end force-control learning alone has limited adaptive capability in new scenes whose physical properties differ substantially; the CoT reasoning module is the key patch for robust cross-scene generalization, rather than something the base architecture already covers.
