几何基础模型对视觉-语言-动作模型的影响研究Understanding the Impact of Geometric Foundation Models on Vision-Language-Action Models
Understanding the Impact of Geometric Foundation Models on Vision-Language-Action Models
Yurou Yang, Muyuan Lin, Roberto Martin-Martin, Martin Labrie, Shreekant Gayaka, Cheng-Hao Kuo, Luca Carlone · Amazon Personal Robotics Group · UT Austin · MIT
近期工作探索将几何基础模型（GFM，如 VGGT）与机器人操作用的视觉-语言-动作模型（VLA，如 GR00T-N1.5）相结合的可能性。
然而，现有 VLA 本身是否具备足够的几何理解能力？注入几何信息的最佳架构是什么？其他设计选择又如何影响效果？
本文针对上述三个问题进行了系统性实验分析，并提供了迄今为止最严格的量化评估。Recent work explores the possibility of combining geometric foundation models (GFM, such as VGGT) with vision-language-action models for robot manipulation (VLA, such as GR00T-N1.5). However, do existing VLAs themselves already possess sufficient geometric understanding? What is the best architecture for injecting geometric information? And how do other design choices affect the outcome? This paper carries out a systematic experimental analysis of these three questions and provides the most rigorous quantitative evaluation to date.
关键词Keywordsgeometric foundation modelVision-Language-ActionVLA机器人操作robot manipulation几何感知融合geometry-aware fusionVGGTEarly Fusion深度估计depth estimation线性探针linear probingGR00T
01动机Motivation
现代 VLA 在多项操作任务中表现出色，但它们的视觉编码器（通常为 CLIP 风格）主要针对语义理解优化，
对于机器人精准操作所需的深度、法线等几何信息建模能力有限。
与此同时，以 VGGT 为代表的几何基础模型（GFM）具备卓越的 3D 重建能力，
自然引发了"将 GFM 的几何感知能力注入 VLA"的研究思路。
然而，此前工作缺乏对以下问题的严格回答：Modern VLAs perform strongly on many manipulation tasks, but their visual encoders (usually CLIP-style) are mainly optimized for semantic understanding and have limited capacity to model the depth, surface normals and other geometric information required for precise robot manipulation. At the same time, geometric foundation models (GFM) such as VGGT possess outstanding 3D reconstruction ability, which naturally motivates the idea of "injecting the geometric perception of a GFM into a VLA". Prior work, however, lacks rigorous answers to the following questions:
"(i) 现代 VLA 本身是否已具备足够的几何理解能力？(ii) 将几何信息注入 VLA 的最佳架构是什么？(iii) 影响几何 VLA 性能的其他设计选择（训练数据规模、摄像头数量、重建质量）各有怎样的作用？""(i) Do modern VLAs already possess sufficient geometric understanding? (ii) What is the best architecture for injecting geometric information into a VLA? (iii) What roles do the other design choices that affect the performance of geometric VLAs (training data scale, number of cameras, reconstruction quality) play?"
图 1： (a) 标准 VLA 架构；(b)-(d) 将几何基础模型（如 VGGT）的 token 注入 VLA 的三种关键策略：
Early Fusion（在 LLM 前融合）、Late Fusion（在 LLM 后融合）、Spatial Forcing（训练时对齐损失）。Figure 1: (a) The standard VLA architecture; (b)-(d) three key strategies for injecting the tokens of a geometric foundation model (such as VGGT) into a VLA: Early Fusion (fusing before the LLM), Late Fusion (fusing after the LLM), and Spatial Forcing (an alignment loss at training time).
几何差距（Geometric Gap）的量化Quantifying the Geometric Gap
作者通过线性探针（linear probing）方法量化 VLA 与 GFM 之间的几何差距——
在固定预训练权重的基础上，训练一个线性层预测稠密深度图，以此评估各模型内部 token 所携带的几何信息量。The authors quantify the geometric gap between VLAs and GFMs through linear probing—on top of frozen pretrained weights, a linear layer is trained to predict a dense depth map, which measures how much geometric information the internal tokens of each model carry.
标准 VLA（GR00T-N1.5）的深度预测 RMSE 约为 VGGT 的两倍，δ₁ 分数也明显更低，
证明 VLA 确实存在显著的"几何差距"。论文进一步发现：
"深度信息在视觉编码器之后便已丢失（the depth information is already lost after the vision encoder）"。The depth-prediction RMSE of the standard VLA (GR00T-N1.5) is about twice that of VGGT and its δ₁ score is markedly lower, showing that the VLA does suffer from a significant "geometric gap". The paper further finds that "the depth information is already lost after the vision encoder".
图 2（a）： 线性探针深度预测对比中的 RGB 输入图像。
标准 VLA（GR00T 视觉编码器及 VLM）输出的深度图质量明显低于 VGGT 探针，
而几何 VLA（Early/Late Fusion）借助 VGGT token 能大幅恢复几何信息。Figure 2(a): The RGB input images of the linear-probe depth-prediction comparison. The depth maps produced by the standard VLA (the GR00T visual encoder and VLM) are clearly of lower quality than those of the VGGT probe, whereas the geometric VLAs (Early/Late Fusion) recover geometric information to a large extent with the help of VGGT tokens.
02方法Method
本文以 GR00T-N1.5 为基础 VLA，以 VGGT 为几何基础模型，设计并对比了三种将几何 token 注入 VLA 的策略，
同时保持底层实现细节尽可能一致，以确保"苹果对苹果"的公平比较。
融合模块统一采用带门控的交叉注意力机制。This paper takes GR00T-N1.5 as the base VLA and VGGT as the geometric foundation model, designing and comparing three strategies for injecting geometric tokens into a VLA while keeping the underlying implementation details as consistent as possible, so as to guarantee an "apples-to-apples" comparison. The fusion module uniformly adopts a gated cross-attention mechanism.
图 1（b-c）细节： Early Fusion 将 VGGT 几何 token 与视觉编码器输出融合后送入 LLM；
Late Fusion 则在 LLM 输出之后、动作专家之前完成融合。两种策略均采用带注意力门控的交叉注意力模块。Figure 1(b-c) in detail: Early Fusion merges the VGGT geometric tokens with the output of the visual encoder before feeding the LLM; Late Fusion instead performs the fusion after the LLM output and before the action expert. Both strategies use a cross-attention module with attention gating.
Early Fusion（早期融合）Early Fusion
VGGT 与 VLA 接收相同图像输入，生成几何 token G。
随后，VLA 视觉编码器输出 Ve 与 G 通过融合模块合并，
得到的融合 token 替代原始视觉 token 送入 LLM。
直觉上，Early Fusion 将 GFM 视为一个额外的视觉编码器。VGGT and the VLA receive the same image input and produce geometric tokens G. The visual-encoder output of the VLA, Ve, is then merged with G through the fusion module, and the resulting fused tokens replace the original visual tokens fed into the LLM. Intuitively, Early Fusion treats the GFM as an additional visual encoder.
Late Fusion（晚期融合）Late Fusion
几何 token G 与 LLM 的视觉输出 Vl 融合，
融合后的 token 替代 Vl 送入动作专家（action expert）。
这种策略尝试在动作生成前用几何信息丰富 VLM 的输出。The geometric tokens G are fused with the visual output of the LLM, Vl, and the fused tokens replace Vl as the input of the action expert. This strategy tries to enrich the VLM output with geometric information before action generation.
Spatial Forcing（空间强制）Spatial Forcing
架构与标准 VLA 相同，不做结构修改。仅在训练时加入对齐损失（alignment loss），
以余弦相似度衡量，鼓励 LLM 内部 token 与 GFM token 对齐，
促使 VLM 在内部保留更多几何信息以供动作专家使用。The architecture is identical to the standard VLA, with no structural modification. An alignment loss is added at training time only, measured by cosine similarity, encouraging the internal tokens of the LLM to align with the GFM tokens and thus pushing the VLM to retain more geometric information internally for the action expert to use.
交叉注意力融合模块Cross-Attention Fusion Module
Early Fusion 与 Late Fusion 使用统一的融合实现：标准缩放点积交叉注意力，
加上一个注意力门控（attention gating）残差项：
X̃ = X + A ⊙ Z，门控矩阵 A 初始化接近零，
避免融合信号在训练初期使动作专家"偏离分布"。
LoRA rank = 8，仅训练交叉注意力层参数。Early Fusion and Late Fusion share one fusion implementation: standard scaled dot-product cross-attention plus a residual term with attention gating: X̃ = X + A ⊙ Z, where the gating matrix A is initialized close to zero so that the fusion signal does not drive the action expert "out of distribution" early in training. LoRA rank = 8, and only the parameters of the cross-attention layers are trained.
融合模块注意力掩码： Early Fusion（左）与 Late Fusion（右）中，
VLA token 对 VGGT 几何 token 的注意力权重可视化，
展示了跨模态几何信息的传递模式。Attention masks of the fusion module: Visualization of the attention weights from VLA tokens to VGGT geometric tokens in Early Fusion (left) and Late Fusion (right), showing how geometric information is passed across modalities.
训练设置Training Setup
硬件：NVIDIA A100（320 GB 显存）。微调（finetuning）训练 100 个 epoch，
中间训练（mid-training）协议为 10 + 50 个 epoch。
学习率 1×10⁻⁴，批大小 12。
VLA 视觉 token 使用 2D 位置编码，VGGT token 使用 3D 位置编码，
以区分模态。每次训练约需 2–3 天。Hardware: NVIDIA A100 (320 GB of memory). Finetuning runs for 100 epochs, and the mid-training protocol uses 10 + 50 epochs. Learning rate 1×10⁻⁴, batch size 12. VLA visual tokens use 2D positional encodings and VGGT tokens use 3D positional encodings, so that the modalities can be told apart. Each training run takes about 2–3 days.
03实验Experiments
在 RoboCasa 仿真基准（8 个厨房操作任务，每任务 5 个演示，每演示 15 次测试，共 600 次试验）、
LIBERO 基准（LIBERO-SPATIAL/OBJECT/GOAL/100），以及 Unitree G1 人形机器人真实场景（90 次试验，pick-and-place）上进行评测。
统计显著性检验采用 McNemar 检验（报告 p 值）。Evaluation is carried out on the RoboCasa simulation benchmark (8 kitchen manipulation tasks, 5 demonstrations per task, 15 rollouts per demonstration, 600 trials in total), the LIBERO benchmark (LIBERO-SPATIAL/OBJECT/GOAL/100), and real-world scenes on the Unitree G1 humanoid robot (90 trials, pick-and-place). Statistical significance is assessed with the McNemar test (p-values reported).
关键发现： 标准微调（finetuning）下，三种几何 VLA 策略均未能在 RoboCasa 上取得统计显著的性能提升（所有 p 值 > 0.1）。但引入中间训练（mid-training）协议后，Early Fusion 平均成功率提升至 75.2%，高于基线的 71.7%，且差距更为稳健。论文指出部分仿真基准已趋饱和，限制了提升空间。Key finding: Under standard finetuning, none of the three geometric VLA strategies achieves a statistically significant improvement on RoboCasa (all p-values > 0.1). Once the mid-training protocol is introduced, however, Early Fusion raises the average success rate to 75.2%, above the baseline's 71.7%, and the gap becomes more robust. The paper notes that some simulation benchmarks are already saturated, which limits the room for improvement.
单摄像头配置（RoboCasa，单摄像头）Single-Camera Setting (RoboCasa, single camera)
在仅使用单个摄像头的配置下，GR00T-N1.5 基线平均成功率仅为 17.2%，
而 Early Fusion 提升至 21.5%（p=0.030），达到统计显著水平。
这说明在视觉信息不足时，几何融合带来的收益更为明显。With only a single camera, the average success rate of the GR00T-N1.5 baseline is merely 17.2%, while Early Fusion lifts it to 21.5% (p=0.030), reaching statistical significance. This shows that the benefit of geometric fusion is more pronounced when visual information is insufficient.
图：Early Fusion 在 RoboCasa 不同训练 checkpoint 的成功率动态。
训练曲线因任务而异，存在较大波动，说明针对特定任务的微调动态高度依赖于任务本身。Success-rate dynamics of Early Fusion across RoboCasa training checkpoints. The training curves vary from task to task and fluctuate considerably, showing that task-specific finetuning dynamics depend heavily on the task itself.
在真实机器人测试中，Early Fusion 的抓取接近（approach/grasp）阶段成功率从基线的 57.78% 大幅提升至 84.44%（p<0.001），达到统计显著，
表明几何信息对需要精准空间定位的操作阶段确实有益。In the real-robot tests the success rate of Early Fusion in the approach/grasp stage rises sharply from the baseline's 57.78% to 84.44% (p<0.001), which is statistically significant and indicates that geometric information really does help the manipulation stages that require precise spatial localization.
图：Early Fusion 融合模块消融实验。
对 Early Fusion 策略内部组件进行消融，分析不同设计选择（如注意力门控、位置编码类型等）的贡献。
注意力门控（attention gating）被证明对策略有效性至关重要。Ablation of the Early Fusion fusion module. The internal components of the Early Fusion strategy are ablated to analyze the contribution of different design choices (attention gating, type of positional encoding, and so on). Attention gating turns out to be crucial for the effectiveness of the policy.
VGGT 重建质量的影响Impact of VGGT Reconstruction Quality
分析 VGGT 深度重建误差与操作成功率之间的关系，
Spearman 相关系数 ρ = −0.202，呈温和的负相关——
重建质量越好，成功率略有提升，但相关性并不强烈。
这说明 GFM 重建质量对最终性能的影响是存在的，但并非决定性因素。Analyzing the relation between VGGT depth-reconstruction error and manipulation success rate gives a Spearman correlation coefficient of ρ = −0.202, a mild negative correlation—the better the reconstruction quality, the slightly higher the success rate, though the correlation is not strong. This suggests that the reconstruction quality of the GFM does affect final performance, but is not the decisive factor.
图：VGGT 深度误差与 RoboCasa 操作成功率的散点图。
Spearman 相关系数 ρ = −0.202，呈温和负相关，说明更好的几何重建质量与更高的任务成功率相关联，
但关系并不强烈。Scatter plot of VGGT depth error against RoboCasa manipulation success rate. The Spearman correlation coefficient ρ = −0.202 indicates a mild negative correlation: better geometric reconstruction quality is associated with higher task success rates, but the relationship is not strong.
消融分析Ablation Analysis
论文对多项设计选择进行了系统消融：The paper systematically ablates a number of design choices:
注意力门控（Attention Gating）： 对 Early Fusion 效果至关重要。若残差连接引入过于突然，会使动作专家偏离原有分布。Attention Gating: Crucial for the effectiveness of Early Fusion. If the residual connection is introduced too abruptly, it drives the action expert away from its original distribution.
训练数据规模（Data Scaling）： Early Fusion 加入中间训练（mid-training）后，性能明显优于直接微调，表明更大规模的预训练数据可充分发挥几何信息的价值。Data Scaling: With mid-training added, Early Fusion performs clearly better than direct finetuning, indicating that larger-scale pretraining data lets the value of geometric information be fully realized.
摄像头数量： 单摄像头配置下几何融合带来的提升更显著（Early Fusion: 21.5% vs. 基线 17.2%，p=0.030），多摄像头下提升不显著。Number of cameras: The gain from geometric fusion is more significant in the single-camera setting (Early Fusion: 21.5% vs. baseline 17.2%, p=0.030), whereas with multiple cameras the improvement is not significant.
位置编码： 为 VGGT token 使用专属的 3D 位置编码，与 VLA 的 2D 编码区分，有助于模型区分不同模态特征。Positional encoding: Giving VGGT tokens a dedicated 3D positional encoding, distinct from the 2D encoding of the VLA, helps the model tell the features of the different modalities apart.
04局限性Limitations
说明： 以下局限性部分为论文明确陈述（stated），部分为根据实验设计推断（推断/inferred）。Note: Some of the limitations below are explicitly stated in the paper, while others are inferred from the experimental design.
实验局限于单一 VLA + GFM 组合（stated）Experiments are limited to a single VLA + GFM combination (stated)
"我们的分析局限于所选择的 VLA（GR00T-N1.5）和 GFM（VGGT）的组合。
尽管存在许多其他 VLA 和 GFM，我们无法将结论推广至其他未经测试的组合。"
不同 VLA 架构（如 Pi0、OpenVLA 等）或不同 GFM 可能产生不同结论。"Our analysis is limited to the chosen combination of VLA (GR00T-N1.5) and GFM (VGGT). Although many other VLAs and GFMs exist, we cannot generalize our conclusions to other untested combinations." Different VLA architectures (such as Pi0 or OpenVLA) or different GFMs may well lead to different conclusions.
仿真基准趋于饱和，限制了提升空间（stated）Simulation benchmarks are saturated, which limits the room for improvement (stated)
"部分仿真基准已趋饱和（some of the simulation benchmarks are saturated），
因此限制了改进空间（limiting the margin of improvement）。"
例如 RoboCasa 中某些任务基线成功率已达 88–93%，几何信息几乎无法带来额外提升。"Some of the simulation benchmarks are saturated," and therefore "limiting the margin of improvement." For instance, the baseline success rate on certain RoboCasa tasks already reaches 88–93%, leaving almost no room for geometric information to add anything.
真实机器人验证规模有限（stated）Real-robot validation is limited in scale (stated)
真实机器人实验仅在 Unitree G1 上进行了 90 次试验，涵盖 3 类物体的 pick-and-place 任务，
场景多样性有限。大多数结论依赖于仿真基准，真实世界的泛化能力有待进一步验证。The real-robot experiments were run only on the Unitree G1, with 90 trials covering pick-and-place of 3 object categories, so scene diversity is limited. Most conclusions rely on simulation benchmarks, and real-world generalization remains to be verified further.
计算资源制约了设计空间探索（stated）Compute resources constrained the exploration of the design space (stated)
"由于计算资源限制，我们未对所有设计选择进行完整消融。"
每个模型的训练耗时 2–3 天（NVIDIA A100），导致超参数搜索空间受限，
部分潜在更优的架构变体（如多层融合、不同融合位置等）未能充分探索。"Because of limited computational resources, we did not run a complete ablation over all design choices." Training each model takes 2–3 days (NVIDIA A100), which restricts the hyperparameter search space, so potentially better architectural variants (multi-layer fusion, different fusion positions, and so on) could not be explored thoroughly.
统计显著性检验的局限（推断/inferred）Limitations of the statistical significance tests (inferred)
虽然使用 McNemar 检验提供了严格的统计基础，但每个任务的试验次数（15 次/episode）有限，
检验功效（statistical power）可能不足以检测出较小的性能差异。
部分结论（如 Early Fusion 中间训练 p=0.104，刚好超过 0.1 显著性阈值）仍处于边界区域。Although the McNemar test provides a rigorous statistical basis, the number of trials per task (15 per episode) is limited, so the statistical power may be insufficient to detect smaller performance differences. Some conclusions (for example Early Fusion with mid-training at p=0.104, just above the 0.1 significance threshold) therefore remain borderline.
GFM 推理开销（推断/inferred）Inference overhead of the GFM (inferred)
引入 VGGT 作为并行编码器会显著增加推理时的计算量和显存需求，
论文未详细分析该开销对实时机器人控制（通常要求 >10 Hz 控制频率）的影响，
这对实际部署构成潜在障碍。Introducing VGGT as a parallel encoder markedly increases the computation and memory needed at inference time. The paper does not analyze in detail how this overhead affects real-time robot control (which usually requires a control frequency >10 Hz), which is a potential obstacle to practical deployment.
