G2VLM 是首个将视觉几何重建与高层空间语义推理统一到单一模型的视觉语言系统。模型采用"双流假说"启发的 Mixture-of-Transformer-Experts 架构，在仅使用 2D 图像输入的前提下，同时实现了竞争性的 3D 点云重建、相机位姿估计，以及多个空间推理基准上的最优性能。G2VLM is the first vision language system that unifies visual geometry reconstruction and high-level spatial semantic reasoning within a single model. It adopts a Mixture-of-Transformer-Experts architecture inspired by the "two-streams hypothesis" and, using only 2D image inputs, simultaneously achieves competitive 3D point cloud reconstruction, camera pose estimation, and state-of-the-art performance on several spatial reasoning benchmarks.
arXiv 2511.21688模型规模：基于 Qwen2-VL-2BModel scale: built on Qwen2-VL-2BSPAR-Bench 均值 54.87（vs GPT-4o 38.81）SPAR-Bench avg. 54.87 (vs GPT-4o 38.81)arXiv 论文arXiv paper
现有视觉语言模型（VLMs）将图像视为"扁平"的 2D 数据处理，缺乏对三维空间的几何理解能力，导致在需要 3D 空间推理的任务上表现受限。另一方面，专门的视觉几何模型虽能进行精确的 3D 重建，但不具备高层语义理解和自然语言交互能力——两类模型之间存在鸿沟。Existing vision language models (VLMs) treat images as "flat" 2D data and lack any geometric understanding of 3D space, which limits their performance on tasks requiring 3D spatial reasoning. Dedicated visual geometry models, on the other hand, can perform accurate 3D reconstruction but possess no high-level semantic understanding or natural language interaction — a gap separates the two families of models.
"By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G2VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing."
图 1：G2VLM 综合演示。模型能够同时处理空间 3D 重建任务（点图估计、相机位姿估计、单目深度估计）和空间理解推理任务。对于空间推理问题，G2VLM 可直接预测 3D 几何并通过交叉推理给出答案。图片来源：原论文 Figure 1。Figure 1: G2VLM overall demonstration.The model simultaneously handles spatial 3D reconstruction tasks (point map estimation, camera pose estimation, monocular depth estimation) and spatial understanding and reasoning tasks. For spatial reasoning questions, G2VLM directly predicts 3D geometry and produces the answer through cross reasoning. Image source: Figure 1 of the original paper.
54.87SPAR-Bench 均值 （本文 G2VLM-SR）SPAR-Bench average (ours, G2VLM-SR)
38.81GPT-4o 在 SPAR-Bench 的均值（对比基线）GPT-4o average on SPAR-Bench (comparison baseline)
48.33MindCube 均值 （本文最优）MindCube average (ours, best)
2B参数量，媲美 更大规模的模型parameters, on par with far larger models
为何现有方法不足？Why are existing methods insufficient?
空间推理 VLMs：将图像视为纯 2D 输入，缺乏几何归纳偏置，空间推理能力上限受限。Spatial reasoning VLMs: treat images as purely 2D inputs, lack geometric inductive bias, and are therefore capped in spatial reasoning ability.
前馈视觉几何模型（如 VGGT、π³）：擅长 3D 重建，但不具备语义理解和自然语言交互能力。Feed-forward visual geometry models (e.g. VGGT, π³): excel at 3D reconstruction but have no semantic understanding or natural language interaction ability.
大型专有模型（如 GPT-4o）：虽具备语言能力，但空间推理分数（SPAR-Bench 38.81）远低于本文 2B 参数的专用模型（54.87）。Large proprietary models (e.g. GPT-4o): they have strong language ability, but their spatial reasoning score (SPAR-Bench 38.81) falls far below that of our 2B-parameter dedicated model (54.87).
02方法Method
G2VLM 以"两流假说"（two-streams hypothesis）为设计灵感，采用 Mixture-of-Transformer-Experts（MoT） 架构，将专门的几何感知专家（Geometric Perception Expert）与语义感知专家（Semantic Perception Expert）融合在同一 Transformer 主干中，通过共享的多模态自注意力层实现特征交互。G2VLM takes the "two-streams hypothesis" as its design inspiration and adopts a Mixture-of-Transformer-Experts (MoT) architecture, fusing a dedicated Geometric Perception Expert and a Semantic Perception Expert into the same Transformer backbone, with feature interaction realized through shared multimodal self-attention layers.
图 2：G2VLM 架构图。模型包含两个专家流：几何感知专家（"where pathway"，蓝色流）负责低层 3D 视觉信息提取；语义感知专家（"what pathway"，橙色流）负责多模态理解。两流在每个 Transformer 块中通过共享的多模态自注意力进行交互。图片来源：原论文 Figure 2。Figure 2: G2VLM architecture.The model contains two expert streams: the geometric perception expert ("where pathway", blue stream) extracts low-level 3D visual information, while the semantic perception expert ("what pathway", orange stream) handles multimodal understanding. The two streams interact through shared multimodal self-attention inside every Transformer block. Image source: Figure 2 of the original paper.
以 DINOv2 编码器为骨干，提取低层视觉信息，将图像 token 映射至 LLM hidden states，输出 3D 几何预测（相机位姿 + 点图）。使用全局注意力（Global Attention）机制，实验证明优于帧内注意力（Frame-Att.）和混合注意力（Mixed-Att.）方案。Uses a DINOv2 encoder as its backbone to extract low-level visual information, maps image tokens into LLM hidden states, and outputs 3D geometry predictions (camera pose + point map). It adopts a Global Attention mechanism, which experiments show to be superior to the frame-wise attention (Frame-Att.) and mixed attention (Mixed-Att.) alternatives.
语义感知专家（Semantic Expert）Semantic Perception Expert
基于预训练的 Qwen2-VL-2B 模型构建，保留其强大的多模态语言理解和指令跟随能力。以 CLIP 编码器提取语义特征，与几何专家的 DINO 特征互补——消融实验证明双编码器设计（DINO + CLIP）优于单编码器方案。Built on the pretrained Qwen2-VL-2B model, retaining its strong multimodal language understanding and instruction-following ability. A CLIP encoder extracts semantic features that complement the DINO features of the geometric expert — ablations show that the dual-encoder design (DINO + CLIP) outperforms single-encoder alternatives.
图 3：G2VLM 统一模型设计。模型将几何感知专家（3D 重建）和语义感知专家（多模态理解）集成于同一框架。所有 token 在每个 Transformer 块中均可进行共享多模态自注意力运算。图片来源：原论文 Figure 3。Figure 3: G2VLM unified model design.The model integrates the geometric perception expert (3D reconstruction) and the semantic perception expert (multimodal understanding) into a single framework. All tokens can take part in the shared multimodal self-attention computation inside every Transformer block. Image source: Figure 3 of the original paper.
两阶段训练策略Two-stage training strategy
阶段一 — 视觉几何学习（Visual Geometry Learning）：冻结语义专家，从零开始在大规模 3D 标注数据集上训练几何专家。损失函数包含点图重建损失、相机位姿损失和表面法线损失的加权组合。Stage 1 — Visual Geometry Learning:The semantic expert is frozen and the geometric expert is trained from scratch on large-scale 3D annotated datasets. The loss function is a weighted combination of point map reconstruction loss, camera pose loss and surface normal loss.
阶段二 — 空间推理学习（Spatial Reasoning Learning）：解冻语义专家，与几何专家联合训练。探索三种损失策略：Stage 2 — Spatial Reasoning Learning:The semantic expert is unfrozen and trained jointly with the geometric expert. Three loss strategies are explored:
CE Loss Only：仅交叉熵，最佳性价比；CE Loss Only: cross-entropy alone, the best cost-effectiveness;
CE + CE Loss：两路交叉熵；CE + CE Loss: two cross-entropy paths;
VG + CE Loss：视觉几何损失 + 交叉熵，综合最优，但需要 3D 标注数据。VG + CE Loss: visual geometry loss + cross-entropy, best overall, but it requires 3D annotated data.
03实验Experiments
G2VLM 在两大类任务上进行评测：（1）视觉几何任务（单目深度估计、点图估计、相机位姿估计）；（2）空间理解与推理任务（SPAR-Bench、MindCube、OST-Bench、OmniSpatial）。G2VLM is evaluated on two broad families of tasks: (1) visual geometry tasks (monocular depth estimation, point map estimation, camera pose estimation); (2) spatial understanding and reasoning tasks (SPAR-Bench, MindCube, OST-Bench, OmniSpatial).
注：G2VLM 在视觉几何任务上的性能与专业 3D 重建模型（VGGT、π³）竞争，但在部分指标（ETH3D、7-Scenes）上略逊于专用模型——此处数据原文呈现，未作修饰。Note: on visual geometry tasks G2VLM is competitive with specialized 3D reconstruction models (VGGT, π³), yet on some metrics (ETH3D, 7-Scenes) it is slightly behind those dedicated models — the numbers are presented as reported in the original paper, without embellishment.
表 1b：空间理解与推理任务结果Table 1b: Spatial understanding and reasoning task results
模型Model
SPAR-Bench Avg.
MindCube Avg.
OST-Bench Avg.
OmniSpatial Avg.
GPT-4o
38.81
37.58
50.74
59.31
Qwen2-VL-2B（base）Qwen2-VL-2B (base)
24.60
37.83
26.85
41.18
G2VLM-SR（本文）G2VLM-SR (ours)
54.87
48.33
45.54
49.20
注：在 OST-Bench 和 OmniSpatial 上，GPT-4o 的得分（50.74 / 59.31）高于本文模型（45.54 / 49.20）——原文数据，如实呈现。G2VLM-SR 在 SPAR-Bench 和 MindCube 上以 2B 参数量超越 GPT-4o。Note: on OST-Bench and OmniSpatial, GPT-4o scores higher (50.74 / 59.31) than our model (45.54 / 49.20) — the original figures are reported faithfully. On SPAR-Bench and MindCube, G2VLM-SR surpasses GPT-4o with only 2B parameters.
图 4（原文 Figure 5）：G2VLM 定性结果。模型能够对多种开放域图像进行有效的 3D 重建，涵盖物体级、结构级、室内及室外场景，并支持动态和静态内容。图片来源：原论文 Figure 5。Figure 4 (Figure 5 in the original paper): G2VLM qualitative results.The model performs effective 3D reconstruction on a wide range of open-domain images, covering object-level, structure-level, indoor and outdoor scenes, and supports both dynamic and static content. Image source: Figure 5 of the original paper.
表 2：消融研究（SPAR-Bench）Table 2: Ablation study (SPAR-Bench)
模型配置Model configuration
Low
Medium
High
Avg.
Qwen2-VL-2B base
19.43
27.55
28.22
24.60
G2VLM-SR（Frame-Att.）G2VLM-SR (Frame-Att.)
58.23
34.47
53.81
52.34
G2VLM-SR（Mixed-Att.）G2VLM-SR (Mixed-Att.)
59.16
35.33
55.16
53.64
G2VLM-SR（Global-Att.）G2VLM-SR (Global-Att.)
59.99
36.27
56.51
54.87
图 5（原文 Figure 6）：消融实验结果（双编码器 vs 单编码器 + 注意力机制对比）。左图对比 CLIP+DINO 双编码器与单独使用 CLIP 或 DINO 的效果；右图对比不同注意力机制对几何特征和空间推理的影响。Global Attention 在几何精度与空间推理上均表现最优。图片来源：原论文 Figure 6。Figure 5 (Figure 6 in the original paper): Ablation results (dual encoder vs single encoder, plus attention mechanism comparison).Left: the CLIP+DINO dual encoder compared with using CLIP or DINO alone; right: the effect of different attention mechanisms on geometric features and spatial reasoning. Global Attention performs best on both geometric accuracy and spatial reasoning. Image source: Figure 6 of the original paper.
关键消融结论Key ablation findings
双编码器优于单编码器：DINO（几何） + CLIP（语义）的组合显著优于仅使用其中一种编码器。Dual encoders beat a single encoder:The combination of DINO (geometry) + CLIP (semantics) significantly outperforms using either encoder alone.
全局注意力（Global Attention）最优：相比帧内注意力（Frame-Att.）和混合注意力（Mixed-Att.），全局注意力在 SPAR-Bench 上实现最高均值（54.87 vs 52.34 / 53.64）。Global Attention is best:Compared with frame-wise attention (Frame-Att.) and mixed attention (Mixed-Att.), global attention reaches the highest average on SPAR-Bench (54.87 vs 52.34 / 53.64).
几何与语义正向协同：实验确认改善的几何特征能够提升空间推理性能，验证了统一架构设计的合理性。Geometry and semantics reinforce each other:Experiments confirm that improved geometric features raise spatial reasoning performance, validating the rationale behind the unified architecture design.
VG + CE 损失综合最优：联合视觉几何损失和交叉熵损失的训练策略在两类任务上均取得最强结果。VG + CE loss is best overall:The training strategy that jointly uses visual geometry loss and cross-entropy loss achieves the strongest results on both families of tasks.
图 6（原文 Figure 4）：不同损失策略对比。展示 CE Loss Only、CE + CE Loss 和 VG + CE Loss 三种训练策略在几何任务和推理任务上的表现对比。图片来源：原论文 Figure 4。Figure 6 (Figure 4 in the original paper): Comparison of loss strategies.Shows how the three training strategies CE Loss Only, CE + CE Loss and VG + CE Loss compare on geometry tasks and reasoning tasks. Image source: Figure 4 of the original paper.
04局限性Limitations
说明：论文在结论部分包含一处明确陈述的局限性（标注为"stated"）；其余各点为基于模型设计与实验结果推断的局限（标注为"inferred"）。Note:The paper states one explicit limitation in its conclusion (marked "stated"); the remaining points are limitations inferred from the model design and the experimental results (marked "inferred").
大规模模型训练不稳定性（stated）Training instability of large-scale models (stated)
论文原文明确指出："One potential limitation is training instability with large-scale models. This challenge requires advanced optimization techniques, careful data curation, and significant computational resources." 当前实验以 Qwen2-VL-2B 为基础，更大规模模型的训练稳定性和计算成本是重要挑战。The paper states explicitly: "One potential limitation is training instability with large-scale models. This challenge requires advanced optimization techniques, careful data curation, and significant computational resources." The current experiments are based on Qwen2-VL-2B; the training stability and computational cost of larger models remain important challenges.
部分基准上仍逊于专业模型（inferred）Still behind specialized models on some benchmarks (inferred)
在视觉几何任务（ETH3D：0.414 vs VGGT 的 0.28；7-Scenes：0.046 vs π³ 的 0.016）上，G2VLM 与专门的 3D 重建模型相比尚有差距。在 OST-Bench（45.54）和 OmniSpatial（49.20）上也低于 GPT-4o（50.74 / 59.31）。这表明统一架构在任务专精性上存在一定取舍。On visual geometry tasks (ETH3D: 0.414 vs 0.28 for VGGT; 7-Scenes: 0.046 vs 0.016 for π³), G2VLM still lags behind dedicated 3D reconstruction models. On OST-Bench (45.54) and OmniSpatial (49.20) it also scores below GPT-4o (50.74 / 59.31). This indicates that the unified architecture involves a trade-off in task specialization.
依赖多视角图像/视频数据进行训练（inferred）Reliance on multi-view image/video data for training (inferred)
VG + CE Loss 策略（最优方案）需要 3D 标注数据，而此类数据的获取本身存在一定难度。尽管论文指出模型通过多视角图像和视频扩展训练规模以减少对稀缺 3D 数据的依赖，但视觉几何阶段的训练仍依赖大规模 3D 标注数据集。The VG + CE Loss strategy (the best option) requires 3D annotated data, which is itself hard to obtain. Although the paper notes that the model scales up training with multi-view images and videos in order to reduce dependence on scarce 3D data, training in the visual geometry stage still relies on large-scale 3D annotated datasets.
仅基于 2B 参数规模验证（inferred）Validated only at the 2B parameter scale (inferred)
所有实验均基于 Qwen2-VL-2B（2B 参数）进行。大规模模型扩展的有效性、训练稳定性以及性能增益尚未在更大规模参数量（如 7B、72B）上系统验证。All experiments are conducted on Qwen2-VL-2B (2B parameters). The effectiveness, training stability and performance gains of scaling to larger models have not yet been systematically validated at larger parameter counts (e.g. 7B, 72B).
