3D CAVLA: 利用深度与三维上下文提升 VLA 模型对未见任务的泛化能力3D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Tasks
通过链式推理 + 点云深度感知 + 任务感知 ROI 池化，使机器人操纵的零样本泛化能力大幅提升Chain-of-thought reasoning + point-cloud depth perception + task-aware ROI pooling substantially improve zero-shot generalization in robotic manipulation
Vineet Bhat · Yu-Hsiang Lan · Prashanth Krishnamurthy · Ramesh Karri · Farshad Khorrami · New York University
3D CAVLA 是一个基于 VLA 模型（OpenVLA-OFT）的微调框架，将 chain-of-thought 推理、基于点云的深度嵌入与任务感知区域兴趣（TA-ROI）池化三种机制有机融合，在 LIBERO 仿真基准上达到 98.1% 的平均成功率，并在未见任务上比基线提升 8.8%，同时在真实 Franka 机器臂实验中实现 25% 的成功率提升，且收敛速度快 3 倍。3D CAVLA is a fine-tuning framework built on a VLA model (OpenVLA-OFT) that integrates three mechanisms: chain-of-thought reasoning, point-cloud-based depth embeddings, and task-aware region-of-interest (TA-ROI) pooling. It reaches a 98.1% average success rate on the LIBERO simulation benchmark, improves over the baseline by 8.8% on unseen tasks, delivers a 25% success-rate gain on a real Franka arm, and converges 3× faster.
VLAVision-Language-Action机器人操作Robot Manipulationchain-of-thought深度感知点云Depth-Aware Point CloudTA-ROI 池化Pooling零样本泛化Zero-Shot Generalizationsim-to-realLIBEROOpenVLA-OFT
01动机Motivation
VLA 模型通过端到端训练将视觉感知、语言理解和动作生成统一起来，在分布内任务上表现优秀，但面对新场景时泛化能力不足。现有方法主要依赖二维 RGB 感知，缺乏三维空间理解，导致机器人在复杂操作任务中难以推广到未见过的任务组合。VLA models unify visual perception, language understanding and action generation through end-to-end training and perform well on in-distribution tasks, but generalize poorly to novel scenes. Existing methods rely mainly on 2D RGB perception and lack 3D spatial understanding, which makes it hard for robots to extend to unseen task combinations in complex manipulation settings.
"Integrating reasoning within VLA training objectives can improve out-of-domain performance." — 论文核心主张the core claim of the paper
作者提出三个核心问题：（1）如何将结构化推理引入 VLA 训练目标？（2）如何利用深度信息增强三维空间感知？（3）如何让模型聚焦于任务相关的视觉区域？3D CAVLA 通过 chain-of-thought 推理分解、深度点云嵌入和 TA-ROI 池化三管齐下，系统性地解决上述挑战，且无需重设计基础架构，以 LoRA 微调方式即可叠加到现有 VLA 模型上。The authors pose three core questions: (1) how can structured reasoning be introduced into the VLA training objective? (2) how can depth information strengthen 3D spatial perception? (3) how can the model be made to focus on task-relevant visual regions? 3D CAVLA attacks all three at once with chain-of-thought reasoning decomposition, point-cloud depth embeddings and TA-ROI pooling, systematically addressing these challenges without redesigning the base architecture: it can be stacked onto an existing VLA model by LoRA fine-tuning.
图 1：3D CAVLA 整体架构。Figure 1: Overall architecture of 3D CAVLA."3D-CAVLA integrates chain-of-thought style narrative task descriptions, depth embeddings and region of interest pooling to improve scene awareness." 系统以 RGB-D 图像和语言指令为输入，同时生成 CoT 推理步骤、点云深度嵌入和 TA-ROI 掩码，三路特征融合后送入 OpenVLA-OFT 生成机器人动作。The system takes RGB-D images and language instructions as input and simultaneously produces CoT reasoning steps, point-cloud depth embeddings and TA-ROI masks; the three feature streams are fused and fed into OpenVLA-OFT to generate robot actions.
98.1%LIBERO 分布内任务平均成功率（双摄像头 + 深度）Average success rate on in-distribution LIBERO tasks (dual camera + depth)
+8.8%未见任务绝对成功率提升（vs. OpenVLA-OFT）Absolute success-rate gain on unseen tasks (vs. OpenVLA-OFT)
+25%真实机器人未见任务成功率提升Success-rate gain on unseen real-robot tasks
3×训练收敛速度提升（3K vs. 10K epochs）Faster training convergence (3K vs. 10K epochs)
02方法Method
3D CAVLA 在 OpenVLA-OFT 基础上叠加三个正交模块：chain-of-thought 任务分解、深度点云编码器和任务感知 ROI 池化（TA-ROI）。三个组件均以离线预计算方式融入训练，不改变基础 VLA 的推理接口，LoRA 微调使参数量增加极小。3D CAVLA stacks three orthogonal modules on top of OpenVLA-OFT: chain-of-thought task decomposition, a depth point-cloud encoder and task-aware ROI pooling (TA-ROI). All three components are folded into training by offline pre-computation, leaving the inference interface of the base VLA unchanged, and LoRA fine-tuning adds only a marginal number of parameters.
1. Chain-of-Thought 链式推理指令Reasoning Instructions
利用冻结的大语言模型（GPT）将任务描述分解为可逐步执行的子步骤序列。例如，"Grab the ball and place it in the basket" 被改写为 "Locate ball → grasp at center → move over basket → release"。这种结构化分解帮助模型在未见任务中进行组合式推理，冻结 LLM 防止过拟合，生成的 CoT 指令在训练时与原始任务描述拼接后输入到 VLA。A frozen large language model (GPT) decomposes the task description into a sequence of step-by-step executable sub-steps. For example, "Grab the ball and place it in the basket" is rewritten as "Locate ball → grasp at center → move over basket → release". Such structured decomposition helps the model reason compositionally on unseen tasks; freezing the LLM prevents overfitting, and the generated CoT instruction is concatenated with the original task description as input to the VLA during training.
2. 深度点云嵌入Depth Point-Cloud Embedding
将 RGB-D 输入通过相机内参反投影为三维点云，公式为：The RGB-D input is back-projected into a 3D point cloud with the camera intrinsics:
Zb,h,w = Db,h,w
Xb,h,w = [(Uh,w − cx) / fx] × Zb,h,w
Yb,h,w = [(Vh,w − cy) / fy] × Zb,h,w
再通过轻量级 PointNet 风格编码器（约 1M 参数）提取深度嵌入 dt，与视觉语言特征拼接后融合。推理开销极小（4.3 Hz vs. 基线 4.4 Hz）。A lightweight PointNet-style encoder (about 1M parameters) then extracts the depth embedding dt, which is concatenated and fused with the vision-language features. The inference overhead is negligible (4.3 Hz vs. 4.4 Hz for the baseline).
3. 任务感知 ROI 池化（TA-ROI）Task-Aware ROI Pooling (TA-ROI)
图 2：TA-ROI 检测流程。Figure 2: The TA-ROI detection pipeline."Task-Aware ROI detection pipeline. Task-relevant entities are identified via named-entity recognition, localized with object detection, and tracked across frames." 依次通过 NER 提取任务相关实体 → Molmo 目标检测 → SAMURAI 跨帧追踪 → 生成二值掩码 M ∈ {0,1}H×W，池化公式：ṽtROI = Pool(ṽt, M)。训练时随机丢弃 ROI 30% 的样本以增强零样本泛化。Task-relevant entities are first extracted by NER → localized by Molmo object detection → tracked across frames by SAMURAI → yielding a binary mask M ∈ {0,1}H×W, with the pooling formula ṽtROI = Pool(ṽt, M). During training the ROI is randomly dropped for 30% of the samples to strengthen zero-shot generalization.
4. 特征融合与训练Feature Fusion and Training
深度嵌入 dt 和 TA-ROI 特征 ṽtROI 在视觉语言特征拼接之前注入，保持 OpenVLA-OFT 整体架构不变。LoRA 微调应用于全模型，CoT 分解和 ROI 掩码均离线预计算，不增加实时推理复杂度。The depth embedding dt and the TA-ROI feature ṽtROI are injected before the vision-language features are concatenated, leaving the overall OpenVLA-OFT architecture unchanged. LoRA fine-tuning is applied to the whole model, while the CoT decomposition and the ROI masks are pre-computed offline, adding no real-time inference complexity.
03实验Experiments
实验在 LIBERO 仿真基准（分布内 + 10 个未见任务）和真实 Franka 桌面操作（10 个任务，5 个物体，2 个目标区域）上进行。基线包括 OpenVLA-OFT、Diffusion Policy（DP）、ECoT* 和 π₀，评价指标为任务成功率。Experiments are run on the LIBERO simulation benchmark (in-distribution + 10 unseen tasks) and on real Franka tabletop manipulation (10 tasks, 5 objects, 2 target regions). Baselines include OpenVLA-OFT, Diffusion Policy (DP), ECoT* and π₀, with task success rate as the metric.
图 3：真实机器人实验。Figure 3: Real-robot experiments."Real robot trials. 3D-CAVLA transforms vision-language observations to SE(3) waypoints and binary gripper states for task execution." 实验在 Franka 桌面操作场景中进行，涵盖 5 个物体和 2 个目标区域，已见任务成功率 90.0%，未见任务提升至 38.0%（基线 30.2%）。The trials are carried out in a Franka tabletop manipulation setting covering 5 objects and 2 target regions; the success rate on seen tasks is 90.0%, while unseen tasks improve to 38.0% (baseline 30.2%).
消融实验Ablation Studies
消融实验在 LIBERO 分布内和未见任务两个维度进行，验证各组件独立贡献：The ablations are run along two axes, LIBERO in-distribution tasks and unseen tasks, to verify the individual contribution of each component:
配置Configuration
LIBERO 已见任务Seen Tasks (%)
LIBERO 未见任务Unseen Tasks (%)
3D-CAVLA（完整）(full)
98.1
45.2
w/o CoT
97.4
42.4（−2.8）
w/o Depth
97.0
41.0（−4.2）
w/o TA-ROI
98.2
41.4（−3.8）
消融结果表明三个组件对已见任务影响有限，但均显著提升未见任务成功率，尤其深度特征移除后未见任务下降最多（−4.2%），说明三维空间感知是泛化的核心驱动力。The ablations show that the three components have only a limited effect on seen tasks but all markedly raise the success rate on unseen tasks; removing the depth features causes the largest drop on unseen tasks (−4.2%), indicating that 3D spatial perception is the core driver of generalization.
图 4：基线（OpenVLA-OFT）在未见任务上的失败案例。Figure 4: Failure cases of the baseline (OpenVLA-OFT) on unseen tasks.在组合式新任务中，基线模型的动作轨迹偏离目标区域，无法完成操作，而 3D CAVLA 借助深度感知和 CoT 推理能够正确执行。On compositional new tasks the baseline model drifts away from the target region and fails to complete the manipulation, whereas 3D CAVLA executes correctly with the help of depth perception and CoT reasoning.
04局限性Limitations
说明：Note:以下限制均由作者在论文中明确陈述（stated），部分推断性限制已标注为 inferred。All limitations below are explicitly stated by the authors in the paper (stated); those that are partly inferential are marked as inferred.
过拟合于小规模真实机器人数据集（stated）Overfitting to the small real-robot dataset (stated)
"In several trials, the policy reverted to executing trajectories resembling previously seen tasks, indicating overfitting to the relatively small dataset used for fine-tuning." 在有限真实数据下微调时，策略容易回退到已见任务的轨迹模式，导致泛化受限。When fine-tuned on limited real-world data, the policy tends to fall back to the trajectory patterns of previously seen tasks, which limits generalization.
抓取点附近的策略震荡（stated）Policy oscillation near the grasp point (stated)
"When approaching the target object, the robot frequently oscillated near the grasp point without completing the action, due to low variation in training images near contact and insufficient cues to trigger grasp closure." 接近目标时，训练数据在接触阶段多样性不足，导致机械臂在抓取点附近来回抖动而无法闭合夹爪。As the arm approaches the target, the training data offers too little variation in the contact phase, so the manipulator jitters back and forth near the grasp point without closing the gripper.
依赖外部 VLM 的检测精度（stated）Reliance on the detection accuracy of external VLMs (stated)
TA-ROI 流程依赖 Molmo 目标检测和 SAMURAI 追踪的准确性。外部模型的误检或漏检会直接影响 ROI 掩码质量，进而影响策略性能。The TA-ROI pipeline depends on the accuracy of Molmo object detection and SAMURAI tracking. False or missed detections by these external models directly degrade the quality of the ROI mask, and in turn the performance of the policy.
抓取角度多样性有限（stated）Limited diversity of grasp angles (stated)
训练数据中抓取角度多样性不足，导致模型对具有新颖方向的物体抓取失败，真实世界泛化仍存在瓶颈。The training data contains too little diversity of grasp angles, so the model fails to grasp objects with novel orientations; real-world generalization remains bottlenecked.
系统依赖多个冻结外部组件（GPT、Molmo、SAMURAI），增加了工程复杂度和部署难度，与端到端方案相比模块间耦合风险更高。The system relies on several frozen external components (GPT, Molmo, SAMURAI), which raises engineering complexity and deployment cost and carries a higher risk of inter-module coupling than an end-to-end solution.
