Do You Need Proprioceptive States in Visuomotor Policies?
State-free 策略：仅凭视觉实现跨高度、跨位置的空间泛化机器人操控State-free policies: vision-only spatial generalization across heights and positions for robot manipulation
Juntu Zhao*, Wenbo Lu*, Di Zhang, Yufeng Liu, Yushen Liang, Tianluo Zhang, Yifeng Cao, Junyuan Xie, Yingdong Hu, Shengjie Wang, Junliang Guo, Dequan Wang, Yang Gao 上海交通大学 · Spirit AI · 清华大学 · 纽约大学上海 · 同济大学Shanghai Jiao Tong University · Spirit AI · Tsinghua University · NYU Shanghai · Tongji University
本体感知（关节角度、末端位姿等状态输入）是机器人视觉运动策略的默认输入，但作者发现它实际上是泛化的"绊脚石"——策略会以状态为"捷径"记忆训练轨迹，而非学会真正的视觉推理。去掉状态输入、改用相对末端执行器动作空间与双广角腕部相机，即可在多机器人平台上大幅提升空间泛化，同时保持域内性能不变。Proprioception (state inputs such as joint angles and end-effector pose) is the default input of robot visuomotor policies, but the authors find that it is in fact a “stumbling block” for generalization — policies take states as a “shortcut” to memorize training trajectories rather than learning genuine visual reasoning. Removing state inputs and switching to a relative end-effector action space with dual wide-angle wrist cameras substantially improves spatial generalization across multiple robot platforms while keeping in-domain performance unchanged.
本体感知状态（关节角度、末端执行器绝对位姿等）长期以来被视为视觉运动策略的必要输入——但它真的有帮助吗？作者通过系统实验发现，状态输入反而会让策略"走捷径"：在固定高度、固定位置训练后，只要物体位置稍有变动，成功率就从98%骤降至0%。Proprioceptive states (joint angles, absolute end-effector pose, etc.) have long been treated as necessary inputs for visuomotor policies — but do they really help? Through systematic experiments the authors find that state inputs instead let the policy “take a shortcut”: after training at a fixed height and a fixed position, the success rate plunges from 98% to 0% as soon as the object position changes slightly.
"State inputs may act as shortcuts that enable policies to memorize training trajectories tied to specific states, rather than developing true visual reasoning for task completion."
图1：空间泛化对比。 State-based 策略（蓝）在高度/水平方向泛化时成功率接近0%；State-free 策略（绿）在多个任务和机器人平台上均保持高成功率。x轴为物体偏移量，y轴为任务成功率。Figure 1: Spatial generalization comparison. State-based policies (blue) drop to near 0% success under height / horizontal generalization; State-free policies (green) keep a high success rate across multiple tasks and robot platforms. The x-axis is the object offset, the y-axis the task success rate.
0% → 98.4%Pick Pen 高度泛化提升Pick Pen height-generalization gain
6% → 58.4%Pick Pen 水平泛化提升Pick Pen horizontal-generalization gain
18.3% → 83.4%Fold Shirt 水平泛化提升Fold Shirt horizontal-generalization gain
11.7% → 78.4%Fetch Bottle 全身机器人泛化提升Fetch Bottle whole-body-robot generalization gain
传统的 state-based 策略在域内（in-domain）表现良好，但当桌子高度变化±10cm 或物体水平偏移5~10cm 时，性能急剧下降。这种脆弱性在真实部署中代价极高。作者验证了该现象在三种不同机器人平台（双臂类人机器人、Arx5 系统、26自由度全身机器人）上均普遍存在。Conventional state-based policies perform well in-domain, but once the table height changes by ±10cm or the object shifts horizontally by 5~10cm, performance collapses. Such brittleness is extremely costly in real deployment. The authors verify that the phenomenon is pervasive on three different robot platforms: a dual-arm humanoid, an Arx5 system, and a 26-DoF whole-body robot.
02方法Method
State-free Policy 由三个核心设计组成：去除所有状态输入、采用相对末端执行器动作空间、以及使用双广角腕部相机提供充分的视觉覆盖。三者协同，使策略的决策完全依赖于视觉观测，从而具备内在的位姿无关性。The State-free Policy consists of three core designs: removing all state inputs, adopting a relative end-effector action space, and using dual wide-angle wrist cameras to provide sufficient visual coverage. Together they make the policy’s decisions depend entirely on visual observation, giving it intrinsic pose-independence.
图2：相机配置对比。 (a) 单个普通腕部相机视角受限，任务目标可能移出视野；(b) 顶部+底部双广角相机（各120°×120°视角）安装于末端执行器，覆盖完整任务空间，确保目标始终可见。Figure 2: Camera configuration comparison. (a) A single ordinary wrist camera has a limited field of view and the task target may move out of sight; (b) top plus bottom dual wide-angle cameras (each with a 120°×120° field of view) mounted on the end-effector cover the whole task space and keep the target always visible.
相对末端执行器动作空间（Relative EEF Action Space）Relative EEF Action Space
策略预测相对位移 Δpt = [Δxt, Δqt]，而非绝对位姿。由于相机固定在末端执行器上，相同的视觉观测对应相同的相对位移，无论机器人的绝对姿态如何——这是空间泛化的核心机制。与之对比，绝对动作空间下策略必须感知自身绝对位置（即需要状态输入），而关节角度空间则因逆运动学高度非线性而难以泛化。The policy predicts a relative displacement Δpt = [Δxt, Δqt] rather than an absolute pose. Because the cameras are fixed on the end-effector, the same visual observation corresponds to the same relative displacement no matter what the absolute configuration of the robot is — this is the core mechanism of spatial generalization. By contrast, under an absolute action space the policy must perceive its own absolute position (i.e. it needs state inputs), while a joint-angle space generalizes poorly because inverse kinematics is highly nonlinear.
在末端执行器顶部和底部各安装一个120°×120° 广角相机。单个普通腕部相机在抓取不同高度物体时，目标很容易滑出视野；双广角方案保证了在所有操作阶段任务相关区域均在视场内。实验还发现，移除俯视（overhead）相机反而提升了性能——俯视视角在机器人姿态变化时会引入分布偏移，拖累泛化。One 120°×120° wide-angle camera is mounted on the top and one on the bottom of the end-effector. A single ordinary wrist camera easily loses the target out of view when grasping objects at different heights; the dual wide-angle design guarantees that the task-relevant region stays inside the field of view at every stage of manipulation. The experiments further find that removing the overhead camera actually improves performance — the overhead view introduces a distribution shift when the robot configuration changes, dragging generalization down.
图3：三种机器人平台与代表性任务。 从左至右：2×8 DoF 类人双臂机器人、2×7 DoF Arx5 系统、26 DoF 全身机器人，以及 pick-and-place、shirt-folding、bottle-fetching 等代表性任务。所提方法在三种平台上均有效。Figure 3: Three robot platforms and representative tasks. From left to right: a 2×8 DoF humanoid dual-arm robot, a 2×7 DoF Arx5 system and a 26 DoF whole-body robot, together with representative tasks such as pick-and-place, shirt-folding and bottle-fetching. The proposed method is effective on all three platforms.
与策略架构无关Independent of the Policy Architecture
State-free 设计并非针对某一特定策略架构，而是一种通用的输入/动作空间改造方案。实验验证了其在 π₀、ACT、Diffusion Policy 三种架构上均一致有效，其中 π₀ 表现最优。The State-free design does not target one particular policy architecture; it is a general redesign of the input / action space. Experiments verify that it is consistently effective on three architectures — π₀, ACT and Diffusion Policy — with π₀ performing best.
03实验Experiments
在三种真实机器人平台和 LIBERO 仿真基准上进行验证，涵盖高度泛化（±10cm）、水平泛化（5~10cm偏移）、域内性能、数据效率和跨实体迁移五个维度。使用 π₀ 作为主要策略架构，对比 state-based 变体（相同视觉输入+本体感知状态）。Validation is carried out on three real robot platforms and the LIBERO simulation benchmark, covering five dimensions: height generalization (±10cm), horizontal generalization (5~10cm offset), in-domain performance, data efficiency and cross-embodiment transfer. π₀ serves as the main policy architecture, compared against a state-based variant (same visual inputs plus proprioceptive states).
图5：三项 pick-and-place 任务的高度与水平泛化成功率柱状图。 绿色（State-free）在各任务和偏移量下均显著优于蓝色（State-based）。域内性能（最左侧）两者相当，证明 State-free 在不损失训练集性能的前提下大幅提升泛化。Figure 5: Bar charts of height and horizontal generalization success rates on three pick-and-place tasks. Green (State-free) is significantly better than blue (State-based) under every task and every offset. In-domain performance (leftmost) is comparable for the two, showing that State-free greatly improves generalization without losing performance on the training distribution.
消融实验：动作表示Ablation: Action Representation
Table III 对比四种动作表示：Table III compares four action representations:
动作空间Action space
高度泛化Height generalization
水平泛化Horizontal generalization
Relative EEF（本文）Relative EEF (ours)
98.4%
58.4%
Absolute EEF
0%
0%
Relative Joint-angle
0%
0%
Absolute Joint-angle
0%
0%
结果清晰表明：相对末端执行器动作空间是泛化的必要条件，其他三种表示方式均完全失效。The results show clearly that the relative end-effector action space is a necessary condition for generalization; the other three representations fail completely.
消融实验：相机配置Ablation: Camera Configuration
Table IV 对比五种相机方案（Pick Pen 任务）：Table IV compares five camera setups (Pick Pen task):
相机配置Camera configuration
高度泛化Height generalization
水平泛化Horizontal generalization
仅 OverheadOverhead only
21.7%
13.3%
单个普通腕部Single ordinary wrist
86.7%
26.7%
双普通腕部Dual ordinary wrist
92.0%
40.0%
双广角腕部（本文）Dual wide-angle wrist (ours)
98.3%
58.3%
数据效率与跨实体迁移Data Efficiency and Cross-Embodiment Transfer
图6：数据效率对比。 仅使用50个演示时，State-based 策略域内成功率约40%；State-free 策略约85%。随微调步数增加（10k步时），跨实体迁移（Arx5 → 类人机器人）成功率：State-based 76.7%，State-free 96.7%（Table VI）。State-free 策略在数据稀缺场景下表现出更强的鲁棒性。Figure 6: Data-efficiency comparison. With only 50 demonstrations, the in-domain success rate of the State-based policy is about 40%, that of the State-free policy about 85%. As the number of fine-tuning steps grows (at 10k steps), cross-embodiment transfer (Arx5 → humanoid robot) reaches 76.7% for State-based and 96.7% for State-free (Table VI). The State-free policy is markedly more robust in data-scarce settings.
04局限性Limitations
Note:以下第1条为作者在论文中明确陈述的局限；第2条为作者提及的观察现象；第3条为从设计出发的推断（inferred）。Item 1 below is a limitation explicitly stated by the authors in the paper; item 2 is an observed phenomenon they mention; item 3 is inferred from the design.
背景敏感性（stated）Background sensitivity (stated)
论文明确指出："Vision-only policies might exhibit sensitivity to the background: changing the background (e.g., relocating the robot and table) may require additional fine-tuning to restore performance." 纯视觉策略对环境外观变化较为敏感，实际部署时若更换场地、光照或背景，需要额外采集数据并微调。The paper states explicitly: “Vision-only policies might exhibit sensitivity to the background: changing the background (e.g., relocating the robot and table) may require additional fine-tuning to restore performance.” Vision-only policies are fairly sensitive to changes in environment appearance; in real deployment, changing the site, the lighting or the background calls for collecting extra data and fine-tuning.
双臂场景中非活动手臂引发误动（stated）The inactive arm triggers spurious motion in bimanual settings (stated)
作者观察到：在双臂操作中，当一侧手臂静止时，另一侧手臂的视觉移动有时会触发非预期的运动。这是宽视场腕部相机会捕捉到对侧手臂运动的副作用，目前尚无系统性解决方案。The authors observe that in bimanual manipulation, when one arm stays still, visual motion of the other arm sometimes triggers unintended movement. This is a side effect of the wide-field-of-view wrist cameras capturing the motion of the contralateral arm, and there is so far no systematic solution.
动作空间局限于末端执行器控制（inferred）Action space limited to end-effector control (inferred)
Relative EEF 动作空间要求机器人支持末端执行器笛卡尔控制接口；对于仅支持关节级控制、或动力学特性复杂（如柔性/软体机器人）的平台，该方法的适用性有待验证。此为从设计推断，论文未显式讨论。The Relative EEF action space requires the robot to expose a Cartesian end-effector control interface; for platforms that only support joint-level control, or whose dynamics are complex (e.g. flexible / soft robots), the applicability of the method remains to be verified. This is inferred from the design and is not explicitly discussed in the paper.
