几何遇见视觉：重新审视蒸馏辐射场中的预训练语义

01 动机Motivation

将大型视觉基础模型的语义特征蒸馏到辐射场（NeRF / Gaussian Splatting）中，已成为语言条件机器人操控与导航的重要基础。既有方法主要使用 CLIP、DINOv2 等纯视觉特征，而近期提出的 VGGT（Visual Geometry Grounded Transformer）则通过 3D 重建任务目标训练，获得了几何感知语义特征。直觉上，视觉-几何特征对位姿估计等空间任务应更有优势——但事实真是如此吗？Distilling the semantic features of large visual foundation models into radiance fields (NeRF / Gaussian Splatting) has become an important foundation for language-conditioned robotic manipulation and navigation. Existing methods mainly use visual-only features such as CLIP and DINOv2, whereas the recently proposed VGGT (Visual Geometry Grounded Transformer) is trained on a 3D reconstruction objective and thereby acquires geometry-grounded semantic features. Intuitively, visual-geometry features should have an edge on spatial tasks such as pose estimation — but is that really the case?

"Do geometry-grounded semantic features offer an edge in distilled fields? … Surprisingly, we find that the pose estimation accuracy decreases with geometry-grounded features."

论文 teaser 图（Figure 1） — **图 1（论文 Figure 1）**：本文针对蒸馏辐射场中视觉-几何特征（VGGT）与纯视觉特征（DINOv2/DINOv3）的三个核心问题展开研究：① 几何保真度（VGGT 更优）、② 语义目标定位（二者相当）、③ 辐射场反演精度（DINOv2 > DINOv3 > VGGT，VGGT 反而更差）。**Figure 1 (paper Figure 1)**: This paper investigates three core questions about visual-geometry features (VGGT) versus visual-only features (DINOv2/DINOv3) in distilled radiance fields: ① geometric fidelity (VGGT is better), ② semantic object localization (the two are comparable), ③ radiance field inversion accuracy (DINOv2 > DINOv3 > VGGT, with VGGT in fact worse).

9个测试场景，跨 3 个基准数据集test scenes across 3 benchmark datasets

3个核心研究问题，涵盖关键机器人应用core research questions covering key robotic applications

SPINE首个无需初始猜测的语义辐射场反演框架the first semantic radiance field inversion framework needing no initial guess

三个核心问题Three Core Questions

Q1：视觉-几何语义特征是否含有更高保真度的空间内容？Q1: Do visual-geometry semantic features contain higher-fidelity spatial content?

是Yes VGGT 特征包含更精细的结构细节，例如更清晰的边缘、更准确的子部件分解，但对象级语义的一致性弱于 DINO 系列。VGGT features contain finer structural detail, such as sharper edges and more accurate sub-part decomposition, but their object-level semantic consistency is weaker than that of the DINO family.

Q2：几何感知是否能提升语义目标定位精度？Q2: Does geometry grounding improve semantic object localization accuracy?

否No 在 GS 与 NeRF 上，视觉-几何特征与纯视觉特征的定位精度无显著差异；VGGT 甚至出现轻微性能退化。On both GS and NeRF, visual-geometry and visual-only features show no significant difference in localization accuracy; VGGT even shows slight degradation.

Q3：视觉-几何特征能否实现更高精度的辐射场反演？Q3: Can visual-geometry features achieve higher-precision radiance field inversion?

否（出人意料）No (surprisingly) DINOv2 在粗位姿估计阶段取得最低旋转误差与平移误差；VGGT 反演精度最差，尽管其几何内容最丰富。DINOv2 attains the lowest rotation and translation errors in the coarse pose estimation stage; VGGT has the worst inversion accuracy, despite having the richest geometric content.

02 方法Method

本文提出了两个核心贡献：① 将 VGGT 视觉-几何特征蒸馏至辐射场的完整流程，② SPINE——一个利用蒸馏语义实现无初始猜测辐射场反演的新框架，由"语义粗反演 + 光度细化"两阶段组成。This paper makes two core contributions: ① a complete pipeline for distilling VGGT visual-geometry features into radiance fields, and ② SPINE — a new framework that leverages distilled semantics for radiance field inversion without an initial guess, composed of two stages: "coarse semantic inversion + photometric refinement".

蒸馏架构图（Figure 2） — **图 2（论文 Figure 2）**：（左）语义蒸馏架构，通过共享 hashgrid 编码（Base Semantics 模块）对 CLIP 与 DINO/VGGT 进行联合监督。（右）VGGT 不同头部（Depth Head、Point Head 及中间层 Layer 0/11/23）输出的语义嵌入 PCA 可视化，Point Head 具有最高几何保真度。**Figure 2 (paper Figure 2)**: (Left) The semantic distillation architecture jointly supervises CLIP and DINO/VGGT through a shared hashgrid encoding (the Base Semantics module). (Right) PCA visualization of the semantic embeddings output by different VGGT heads (Depth Head, Point Head and intermediate Layer 0/11/23); the Point Head has the highest geometric fidelity.

视觉-几何特征提取与蒸馏Visual-Geometry Feature Extraction and Distillation

从 VGGT 的 Depth Head 和 Point Head（分别经深度估计和稠密点云重建训练）中提取语义嵌入，嵌入维度 d_s = 128。同时训练 VGGT 语义场 f_s 与 CLIP 语言场 f_l，二者共享相同的 hashgrid 编码（base semantics），使几何与语言特征关联相同的视觉和几何基底。Semantic embeddings are extracted from VGGT's Depth Head and Point Head (trained for depth estimation and dense point cloud reconstruction respectively), with embedding dimension d_s = 128. The VGGT semantic field f_s and the CLIP language field f_l are trained jointly and share the same hashgrid encoding (base semantics), so that geometric and language features are tied to the same visual and geometric substrate.

训练损失同时包含 Frobenius 范数项和余弦相似度项，以保证数值稳定性：The training loss contains both a Frobenius-norm term and a cosine-similarity term to ensure numerical stability:
L = L_r + Σ‖I_f,c − Î_f,c‖²_F − Σ csim(I_f,c, Î_f,c)

SPINE：语义驱动的辐射场反演SPINE: Semantics-Driven Radiance Field Inversion

粗反演（Coarse Inversion）：训练逆向模型 p_ψ，将语义嵌入（VGGT 使用 camera embedding，DINO 使用 class token）映射至相机位姿的高斯混合模型（GMM）分布，以 Lie 代数 so(3) 参数化旋转，无需初始位姿猜测。Coarse Inversion: an inverse model p_ψ is trained to map semantic embeddings (the camera embedding for VGGT, the class token for DINO) to a Gaussian mixture model (GMM) distribution over camera poses, parameterizing rotation with the Lie algebra so(3), so that no initial pose guess is required.

细反演（Fine Inversion）：以粗估计为起点，通过新视角合成生成 RGB-D 图像，匹配特征点并求解 PnP 问题（使用 RANSAC 提升鲁棒性），最终得到高精度位姿估计。Fine Inversion: starting from the coarse estimate, RGB-D images are generated via novel view synthesis, feature points are matched and a PnP problem is solved (using RANSAC for robustness), finally yielding a high-precision pose estimate.

几何保真因子（Geometric Fidelity Factor, GFF）Geometric Fidelity Factor (GFF)

为定量刻画蒸馏特征的几何内容，作者提出 GFF：先用 Sobel-Feldman 算子对语义图像和 RGB 图像分别提取边缘，再计算语义边缘相对于 RGB 边缘的保留比例：To quantify the geometric content of distilled features, the authors propose GFF: edges are first extracted from the semantic image and the RGB image with a Sobel-Feldman operator, then the proportion of semantic edges retained relative to the RGB edges is computed:
GFF = Σ I_e,sem[i,j,k] / Σ I_e,rgb[i,j,k]
GFF 越高，表示语义特征中保留的场景几何信息越多。The higher the GFF, the more scene geometry is preserved in the semantic features.

03 实验Experiments

在 9 个场景、3 个数据集（LERF、3D-OVS、机器人数据集）上，分别训练 GS 和 NeRF 表示，对比 DINOv2、DINOv3、VGGT 三种语义特征，每场景计算 100 个相机位姿的指标。训练硬件：Nvidia L40 GPU（48GB VRAM）；框架：Nerfstudio，迭代次数 30000。On 9 scenes from 3 datasets (LERF, 3D-OVS and a robot dataset), GS and NeRF representations are trained separately and three semantic features — DINOv2, DINOv3 and VGGT — are compared, computing metrics over 100 camera poses per scene. Training hardware: Nvidia L40 GPU (48GB VRAM); framework: Nerfstudio, 30000 iterations.

Q1：语义特征几何内容（GFF）Q1: Geometric Content of Semantic Features (GFF)

GFF 对比图（Figure 4） — **图 4（论文 Figure 4）**：不同梯度阈值下的 GFF 对比。GS 中 VGGT 特征在所有阈值下边缘数量最多，DINOv2 最少；DINOv2 和 DINOv3 的 GFF 在不同阈值间几乎恒定，说明其几何内容多样性不足。NeRF 中三者 GFF 差异较小。上方为 Teatime 场景的边缘可视化（阈值 0.1 和 0.3）。**Figure 4 (paper Figure 4)**: GFF comparison under different gradient thresholds. In GS, VGGT features give the largest number of edges at every threshold and DINOv2 the fewest; the GFF of DINOv2 and DINOv3 stays almost constant across thresholds, indicating insufficient diversity in their geometric content. In NeRF the three differ less. Above are edge visualizations for the Teatime scene (thresholds 0.1 and 0.3).

Q2 & Q3：语义定位与辐射场反演Q2 & Q3: Semantic Localization and Radiance Field Inversion

语义定位与辐射场反演结果（Figure 5 & 6） — **图 5 & 6（论文 Figure 5 & 6）**：（上）语义目标定位：DINOv2/DINOv3/VGGT 在 GS 和 NeRF 上的 SSIM、PSNR、LPIPS 三项指标无显著差异，可视化显示三者均能有效定位 cookies、sheep、bear 等目标。（下）辐射场反演：现有基线方法（GS 基线 [3]、iNeRF [31]）在无良好初始猜测时表现很差；SPINE（DINOv2 版本）的粗位姿估计已优于基线，细化后进一步提升；VGGT 版 SPINE 的粗/细估计均弱于 DINOv2 版本。**Figures 5 & 6 (paper Figures 5 & 6)**: (Top) Semantic object localization: DINOv2/DINOv3/VGGT show no significant difference in SSIM, PSNR and LPIPS on GS and NeRF, and the visualizations show that all three localize targets such as cookies, sheep and bear effectively. (Bottom) Radiance field inversion: existing baselines (the GS baseline [3] and iNeRF [31]) perform poorly without a good initial guess; the coarse pose estimate of SPINE (DINOv2 version) already outperforms the baselines and improves further after refinement; both the coarse and the fine estimates of the VGGT version of SPINE are weaker than those of the DINOv2 version.

PCA 语义内容可视化PCA Visualization of Semantic Content

PCA 语义内容可视化（Figure 3） — **图 3（论文 Figure 3）**：Teatime 场景的语义特征 PCA 可视化。上行（GS）：DINOv2/DINOv3 特征中 bear 和 sheep 与桌椅区分明显，体现对象级分解；VGGT 则突出场景几何轮廓（边缘清晰）。下行（NeRF）：VGGT 更好地保留了杯子和盘子的轮廓，但 DINO 系列对木纹等细节更敏感。**Figure 3 (paper Figure 3)**: PCA visualization of semantic features on the Teatime scene. Top row (GS): in the DINOv2/DINOv3 features, bear and sheep are clearly separated from the table and chairs, reflecting object-level decomposition, whereas VGGT highlights the geometric contours of the scene (sharp edges). Bottom row (NeRF): VGGT better preserves the contours of the cup and the plate, but the DINO family is more sensitive to details such as wood grain.

实验结论总结Summary of Experimental Findings

任务Task	最优方法Best method	DINOv2	DINOv3	VGGT
几何保真度（GFF，GS）Geometric fidelity (GFF, GS)	VGGT	最低Lowest	中等Medium	最高Highest
语义目标定位（SSIM/PSNR/LPIPS）Semantic object localization (SSIM/PSNR/LPIPS)	三者相当All three comparable	相当Comparable	相当Comparable	轻微退化Slight degradation
粗位姿估计（旋转&平移误差）Coarse pose estimation (rotation & translation error)	DINOv2	最低误差Lowest error	中等Medium	最高误差Highest error
细化后位姿估计Pose estimation after refinement	DINOv2-SPINE	最高成功率Highest success rate	中等Medium	最低成功率Lowest success rate

核心结论（论文原文）：Key conclusion (from the paper): "visual-only features offer greater versatility for a broader range of downstream tasks, although geometry-grounded features contain more geometric detail." 纯视觉特征（DINOv2）在更广泛的下游任务中展现出更强的通用性，而几何感知特征（VGGT）虽几何细节更丰富，却未能转化为更好的任务性能。Visual-only features (DINOv2) exhibit stronger versatility across a broader range of downstream tasks, whereas geometry-grounded features (VGGT), although richer in geometric detail, fail to translate this into better task performance.

04 局限性与未来工作Limitations and Future Work

说明：Note: 以下局限性均来自论文 Section 9"Limitations and Future Work"的明确陈述（stated）。All the limitations below are explicitly stated in Section 9, "Limitations and Future Work", of the paper.

全监督几何感知方式限制了特征通用性Fully supervised geometry grounding limits feature generality

论文指出，现有几何感知方案（如 VGGT）采用全监督方法进行空间感知训练，可能引入归纳偏置，削弱特征的通用性，从而导致辐射场反演等任务性能下降。未来工作将探索自监督几何感知方案，以消除归纳偏置、提升适应性并支持更大规模预训练。The paper notes that existing geometry-grounding schemes (such as VGGT) use fully supervised training for spatial perception, which may introduce inductive biases that weaken the generality of the features and thus degrade performance on tasks such as radiance field inversion. Future work will explore self-supervised geometry-grounding schemes in order to remove inductive biases, improve adaptability and support larger-scale pretraining.

几何内容与视觉语义的协同不足Insufficient synergy between geometric content and visual semantics

实验发现视觉-几何语义虽含有更丰富的结构内容，却未能提升语义目标定位精度，说明几何内容与视觉语义之间缺乏有效协同。未来工作需设计更有效的策略，将几何导向和视觉导向的语义特征融合，以实现更鲁棒的场景理解。The experiments find that although visual-geometry semantics carry richer structural content, they fail to improve semantic object localization accuracy, indicating a lack of effective synergy between geometric content and visual semantics. Future work needs to design more effective strategies for fusing geometry-oriented and vision-oriented semantic features in order to achieve more robust scene understanding.

计算效率低，缺乏轻量化变体Low computational efficiency, no lightweight variants

现有几何感知视觉骨干网络（如 VGGT）相较于无感知骨干网络存在显著的计算开销，且目前尚无轻量级变体，难以用于实时机器人应用（如操控任务）。未来工作将探索高效的空间感知视觉骨干架构。Existing geometry-grounded visual backbones (such as VGGT) carry a significant computational overhead compared with non-grounded backbones, and no lightweight variants exist so far, which makes them hard to use in real-time robotic applications such as manipulation. Future work will explore efficient spatially grounded visual backbone architectures.

细反演性能依赖粗估计质量Fine-inversion performance depends on coarse-estimate quality

SPINE 的细化阶段成功率直接取决于粗位姿估计的误差大小。由于 VGGT 的粗估计精度较低，其细化阶段的成功率也相应更低。这也说明粗反演模型质量对最终位姿精度至关重要。（stated in Section 7.4）The success rate of SPINE's refinement stage depends directly on the magnitude of the error in the coarse pose estimate. Because VGGT's coarse estimates are less accurate, the success rate of its refinement stage is correspondingly lower. This also shows that the quality of the coarse-inversion model is critical to the final pose accuracy. (stated in Section 7.4)