现有视觉编码器缺乏"动作感知"——它们理解语义，却不关注与物理交互相关的区域。CAIP 提出以大规模人类第一视角视频中的 3D 手部姿态为末端执行器动作的代理，通过 SigLIP 风格的对比损失将"动作嵌入"与"文本条件图像嵌入"对齐，在只用 88 小时机器人数据的前提下，于真实灵巧操纵任务上达到 76% 平均成功率，超越最强基线 SigLIP 2（43.4%）逾 30 个百分点。Existing visual encoders lack "action awareness" — they understand semantics, yet pay no attention to the regions relevant to physical interaction. CAIP proposes using 3D hand poses from large-scale egocentric human video as a proxy for end-effector actions, aligning the "action embedding" with the "text-conditioned image embedding" through a SigLIP-style contrastive loss. With only 88 hours of robot data, it reaches a 76% average success rate on real-world dexterous manipulation tasks, surpassing the strongest baseline SigLIP 2 (43.4%) by more than 30 percentage points.
现有机器人视觉编码器面临双重困境：一方面，机器人轨迹数据规模远不及互联网图文数据；另一方面，用图文对比（CLIP/SigLIP）或掩码自编码（MAE/DINOv2）预训练的编码器虽捕获语义或空间结构，却从未接触过配对的视觉-动作信号，因此在策略学习中产生根本性的表示错位。Existing robot visual encoders face a twofold dilemma: on the one hand, robot trajectory data is far smaller in scale than internet image-text data; on the other hand, encoders pre-trained with image-text contrastive learning (CLIP/SigLIP) or masked autoencoding (MAE/DINOv2) capture semantics or spatial structure, yet have never been exposed to paired visual-action signals, and therefore induce a fundamental representational misalignment in policy learning.
"robot trajectories, the most direct source of this paired signal, are not available at pre-training scale, motivating us to extract action signals from abundant human video instead."
CAIP 的核心洞察：人类手部姿态（3D 关键点）在形式上类似于机器人末端执行器轨迹，可以从海量第一视角视频中廉价获取，从而弥合人类示范与稀缺机器人数据之间的鸿沟。The core insight of CAIP: human hand poses (3D keypoints) are formally similar to robot end-effector trajectories and can be obtained cheaply from massive egocentric video, thereby bridging the gap between human demonstrations and scarce robot data.
Figure 1（左）：显著图对比——SigLIP 关注语义区域，DINOv2 关注空间结构，而 CAIP 的注意力集中于手与操纵相关物体。（中）：CAIP 预训练框架概览：手部姿态动作与图文对作为正负样本输入对比损失。（右）：在六项真实操纵任务上，CAIP（76%）大幅领先 R3M（17%）、MVP（42%）、DINOv2（42%）与 SigLIP 2（43%）。Figure 1 (left): saliency map comparison — SigLIP attends to semantic regions, DINOv2 to spatial structure, whereas CAIP's attention concentrates on the hand and the objects involved in manipulation. (middle): overview of the CAIP pre-training framework, where hand-pose actions and image-text pairs enter the contrastive loss as positive and negative samples. (right): on six real-world manipulation tasks, CAIP (76%) leads R3M (17%), MVP (42%), DINOv2 (42%) and SigLIP 2 (43%) by a wide margin.
76%CAIP 平均成功率CAIP average success rate
43.4%最强基线 SigLIP 2Strongest baseline SigLIP 2
>30pp超越最强基线的提升幅度Gain over the strongest baseline
32,041 h自我中心预训练视频Egocentric pre-training video
02方法Method
CAIP 包含三个编码器（视觉、语言、动作），通过两阶段注意力池化生成文本条件图像嵌入，再与动作嵌入用 SigLIP 风格 sigmoid 对比损失对齐。整套系统基于大规模自我中心人类视频预训练，下游策略使用冻结的视觉编码器。CAIP comprises three encoders (vision, language, action). Two-stage attention pooling produces a text-conditioned image embedding, which is then aligned with the action embedding by a SigLIP-style sigmoid contrastive loss. The whole system is pre-trained on large-scale egocentric human video, and the downstream policy uses the frozen visual encoder.
Figure 2 — CAIP architecture。 ViT-L/16 视觉主干（初始化自 SigLIP 2）将图像切成 N 个 patch token；24 层文本 Transformer 编码语言指令为 L 个 token。第一阶段注意力池化：以文本 token 为 query、patch 特征为 key/value，产生文本对齐视觉特征（B×L×C）。第二阶段注意力池化：可学习 query 进一步压缩为单一文本条件图像嵌入（B×1×C）。动作编码器为 4 层 Transformer，接收 T 步 42 关键点动作块（A_d=378），通过 [CLS] token 输出动作嵌入（B×1×C），从头训练。两路嵌入由 SigLIP 对比损失对齐。Figure 2 — CAIP architecture. A ViT-L/16 visual backbone (initialized from SigLIP 2) splits the image into N patch tokens; a 24-layer text Transformer encodes the language instruction into L tokens. First-stage attention pooling: text tokens act as queries and patch features as key/value, yielding text-aligned visual features (B×L×C). Second-stage attention pooling: learnable queries further compress them into a single text-conditioned image embedding (B×1×C). The action encoder is a 4-layer Transformer that receives a T-step, 42-keypoint action chunk (A_d=378) and emits the action embedding (B×1×C) through a [CLS] token, trained from scratch. The two embeddings are aligned by the SigLIP contrastive loss.
预训练目标：SigLIP 风格 sigmoid 对比损失Pre-training objective: SigLIP-style sigmoid contrastive loss
与 CLIP 的 softmax InfoNCE 不同，SigLIP 的 sigmoid 损失将每对图-动作视为独立二分类问题，无需跨 batch 全局归一化，在大 batch 下训练更稳定。正样本对为同一场景帧对应的"文本条件图像嵌入 + 动作嵌入"，batch 内其余配对均为负样本。Unlike the softmax InfoNCE of CLIP, the sigmoid loss of SigLIP treats every image-action pair as an independent binary classification problem and requires no global normalization across the batch, making training more stable at large batch sizes. A positive pair is the "text-conditioned image embedding + action embedding" of the same scene frame; all remaining pairings within the batch serve as negatives.
每个训练样本包含从当前帧起的 T 步未来手部动作块。每时刻手部姿态用 42 个关键点（双手各 21 个，含手腕）的 SE(3) 变换表示（MANO 手型约定），A_d = 378（= 42 × 9）。设 T = 64，约覆盖 30 Hz 下 2 秒的未来手部运动。相对变换定义为：时刻 t 相对于基准帧的 SE(3) 变换——与下游机器人策略输出的 delta 控制形式完全对应。Each training sample contains a chunk of T future hand actions starting from the current frame. The hand pose at each timestep is represented by the SE(3) transforms of 42 keypoints (21 per hand, including the wrist) under the MANO hand convention, giving A_d = 378 (= 42 × 9). Setting T = 64 covers roughly 2 seconds of future hand motion at 30 Hz. The relative transform is defined as the SE(3) transform of timestep t with respect to a reference frame — exactly matching the delta control format emitted by the downstream robot policy.
下游策略架构Downstream policy architecture
视觉编码器冻结后输出 per-patch 视觉 token 与文本 token，投影后送入 Qwen3.5-0.8B decoder-only Transformer（从头训练），最终由 flow-matching 动作头预测动作块。评估平台为 Dexmate Vega 双臂机器人 + 22-DoF Sharpa Wave 灵巧手，配三路摄像头（立体头部 + 双腕）。Once frozen, the visual encoder outputs per-patch visual tokens and text tokens, which after projection are fed into a Qwen3.5-0.8B decoder-only Transformer (trained from scratch), and a flow-matching action head finally predicts the action chunk. The evaluation platform is a Dexmate Vega dual-arm robot with a 22-DoF Sharpa Wave dexterous hand and three cameras (a stereo head camera plus two wrist cameras).
03实验Experiments
在六项真实世界灵巧操纵任务上（每任务 12 次试验，成功率 %），与 R3M、VideoMAE、VC-1、MVP、DINOv2、SigLIP 和 SigLIP 2 对比；同时在保留自我中心数据集上做 zero-shot 动作检索，以及光照 / 干扰物环境鲁棒性分析。每项任务约 200 次示范（pour 为 150 次）。On six real-world dexterous manipulation tasks (12 trials per task, success rate in %), CAIP is compared against R3M, VideoMAE, VC-1, MVP, DINOv2, SigLIP and SigLIP 2; in addition, zero-shot action retrieval is run on a held-out egocentric dataset, together with a robustness analysis under lighting and distractor conditions. Each task uses about 200 demonstrations (150 for pour).
Table 1。六项操纵任务成功率（%），每任务 12 次试验。CAIP 在五项任务中排名第一，平均成功率 76.04%，比最强基线 SigLIP 2（43.40%）高出逾 30 个百分点。MVP 在 Dispense Soap 上达 93.75%，但在其他任务严重退化（Turn On Lamp 仅 8.33%），表明非动作感知表示缺乏跨任务一致性。Table 1. Success rates (%) on six manipulation tasks, 12 trials per task. CAIP ranks first on five of them, with an average success rate of 76.04%, exceeding the strongest baseline SigLIP 2 (43.40%) by more than 30 percentage points. MVP reaches 93.75% on Dispense Soap but degrades severely on the other tasks (only 8.33% on Turn On Lamp), indicating that representations without action awareness lack cross-task consistency.
Method
Fold Shorts
Pour
Pick Fruits
Dispense Soap
Turn On Lamp
Pull Tissue
Avg.
R3M
14.58
12.50
2.08
29.17
8.33
37.50
17.36
Qwen3.5 ViT
27.08
22.92
60.42
72.92
8.33
12.50
34.03
VideoMAE
22.92
52.08
0.00
37.50
25.00
18.75
26.04
VC-1
18.75
56.25
0.00
62.50
0.00
22.92
26.74
MVP
54.17
62.50
2.08
93.75
8.33
31.25
42.01
DINOv2
22.92
81.25
52.08
50.00
25.00
20.83
42.01
SigLIP
12.50
70.83
37.50
83.33
25.00
25.00
42.36
SigLIP 2
4.17
35.42
52.08
93.75
50.00
25.00
43.40
CAIP (Ours)
68.75
83.33
56.25
100.00
75.00
72.92
76.04
Zero-Shot 动作检索与线性探针Zero-shot action retrieval and linear probing
在保留自我中心数据集上（K=50 K-means 动作类别）：CAIP 的 zero-shot 检索（无任何域内监督）在仅 16 样本/类时就超越所有基线的线性探针上界，说明其表示空间已内化了可迁移的动作语义结构（Figure 3）。On a held-out egocentric dataset (K=50 K-means action categories): the zero-shot retrieval of CAIP, without any in-domain supervision, already surpasses the linear-probe upper bound of every baseline with as few as 16 samples per class, showing that its representation space has internalized transferable action semantics (Figure 3).
Figure 3（左）：线性探针与 zero-shot 动作检索准确率随标注样本数变化曲线。CAIP (zero-shot) 在 16 样本/类以内持续优于所有基线线性探针。（Tables 2&3）：光照扰动与干扰物测试下，CAIP 在全部条件中均保持最高成功率，相比 MVP 和 Qwen3.5 ViT 退化幅度最小。Figure 3 (left): linear-probe and zero-shot action-retrieval accuracy as a function of the number of labeled samples. CAIP (zero-shot) consistently outperforms the linear probes of all baselines within 16 samples per class. (Tables 2&3): under lighting perturbations and distractor tests, CAIP retains the highest success rate in every condition, degrading the least compared with MVP and Qwen3.5 ViT.
光照扰动（额外灯泡 / 减光）与两个干扰物（红色书 + 河内塔玩具）场景下，CAIP 在所有条件下保持最高成功率：光照平均 81.25% / 51.39% / 43.06%，干扰物平均 81.25% / 52.78%。而 MVP 在干扰物条件下从原始 52.08% 骤降至 9.72%，Qwen3.5 ViT 从 36.11% 降至 28.47%。CAIP 的动作感知表示对场景级干扰更具鲁棒性。Under lighting perturbations (an extra bulb / dimmed light) and two distractor scenarios (a red book + a Tower of Hanoi toy), CAIP keeps the highest success rate in all conditions: lighting averages of 81.25% / 51.39% / 43.06%, and distractor averages of 81.25% / 52.78%. MVP, by contrast, plunges from an original 52.08% to 9.72% under distractors, and Qwen3.5 ViT falls from 36.11% to 28.47%. The action-aware representation of CAIP is more robust to scene-level distraction.
Scaling Ablations
对视觉主干从 ViT-B → ViT-L → ViT-SO400M 进行 scaling 消融：ViT-B 到 ViT-L 的过渡带来最大的平均提升（>30%），ViT-L 提供最佳性能/参数量/推理速度权衡，因此选为主实验编码器。A scaling ablation on the visual backbone from ViT-B → ViT-L → ViT-SO400M: the transition from ViT-B to ViT-L brings the largest average gain (>30%), and ViT-L offers the best trade-off among performance, parameter count and inference speed, hence it is chosen as the encoder for the main experiments.
04局限性Limitations
Note：以下两点均为作者在"Limitations and Future Work"一节明确陈述的局限。Note: Both points below are limitations explicitly stated by the authors in the "Limitations and Future Work" section.
负样本歧义：连续动作空间中的假负样本问题Negative-sample ambiguity: false negatives in a continuous action space
对比目标将 batch 内所有非配对的图-动作对均视为负样本，忽略动作间的物理相似性。"distinct trajectories drawn from different timesteps or scenes may feature similar hand motions (e.g., two pouring actions or two reaches toward similar targets), yet the loss will actively push their representations apart." 这一假设可能弱化学习信号、限制表示质量。未来可探索依动作空间距离对负样本加权的 soft contrastive 目标。The contrastive objective treats every non-paired image-action combination within the batch as a negative, ignoring the physical similarity between actions. "distinct trajectories drawn from different timesteps or scenes may feature similar hand motions (e.g., two pouring actions or two reaches toward similar targets), yet the loss will actively push their representations apart." This assumption may weaken the learning signal and limit representation quality. Future work could explore a soft contrastive objective that weights negatives by their distance in action space.
人体形态偏差：MANO 骨骼对非五指末端执行器的迁移性未知Human morphology bias: transfer of the MANO skeleton to non-five-fingered end-effectors is unknown
动作表示以 42 关键点 MANO 手型骨骼为中心，"biases the learned features toward human hands." 对 Sharpa Wave 五指手迁移良好，但对平行夹爪或三指爪等平行形态的可迁移性是开放问题。"Future work should evaluate CAIP across a broader range of end-effector morphologies."The action representation is centered on a MANO hand skeleton of 42 keypoints, which "biases the learned features toward human hands." It transfers well to the five-fingered Sharpa Wave hand, but its transferability to parallel morphologies such as parallel-jaw or three-finger grippers is an open question. As the authors put it, "Future work should evaluate CAIP across a broader range of end-effector morphologies."
