VGGT-Ω

01 研究动机Motivation

近年来的工作表明，前馈式重建模型在许多情况下能够匹敌、甚至超过传统的运动恢复结构（SfM）管线。更进一步，这类模型学到的 token 已被用作有效的「几何感知表征」服务于很多其它任务。这说明重建本身可以作为学习空间理解表征的代理任务，具有基础模型的价值。Recent work shows that feed-forward reconstruction models can in many cases match or even surpass traditional Structure-from-Motion (SfM) pipelines. Beyond that, the tokens such models learn have already been used as effective "geometry-aware representations" for many other tasks. This suggests that reconstruction itself can serve as a proxy task for learning representations of spatial understanding, and thus carries the value of a foundation model.

然而，相比在 2D 视觉与 NLP 中「规模（scale）的作用已被充分理解」，这一点在 3D 计算机视觉中几乎没被探究过。因此本文要回答：However, whereas in 2D vision and NLP "the role of scale is by now well understood", this has barely been explored in 3D computer vision. The paper therefore sets out to answer:

前馈重建模型能否被扩展到更大规模？这种扩展会带来什么收益？Can feed-forward reconstruction models be scaled up, and what does such scaling buy us?

为回答这个问题，本文同时解决三个障碍：To answer this question, the paper addresses three obstacles at once:

02 方法Method

VGGT-Ω 是一个前馈 Transformer f，将 N 张图像映射为相机 旋转四元数 q平移 t视场角 f 与深度图 D。与 VGGT 不同，模型不直接预测点图或 tracking 特征（但仍以损失监督它们）。VGGT-Ω is a feed-forward Transformer f that maps N images to cameras rotation quaternion qtranslation tfield of view f and depth maps D. Unlike VGGT, the model does not directly predict point maps or tracking features (though losses still supervise them).

改进一 · Register Attention（寄存器注意力）Improvement 1 · Register Attention

VGGT 交替使用逐帧自注意力与全局自注意力。「全局注意力是 VGGT 的主要计算瓶颈，但其注意力图非常稀疏」——这暗示少量 token 就足以交换相应信息。VGGT alternates frame-wise self-attention with global self-attention. "Global attention is the main computational bottleneck of VGGT, yet its attention maps are very sparse" — hinting that a small number of tokens suffices to exchange the relevant information.

因此 VGGT-Ω 把 25% 的全局注意力层替换为 register attention，在这些层里「帧间的信息交换被限制在寄存器之间」。更新后的寄存器随后在逐帧注意力块中与各帧图像 token 交互，「从而形成一个聚合并重新分发多帧信息的瓶颈」。本文也把这些寄存器称为「场景（scene）token」。VGGT-Ω therefore replaces 25% of the global attention layers with register attention; in those layers "information exchange between frames is restricted to the registers". The updated registers subsequently interact with each frame's image tokens in the frame-wise attention blocks, "thereby forming a bottleneck that aggregates and redistributes multi-frame information". The paper also calls these registers "scene tokens".

改进二 · 轻量级密集头（去掉昂贵的高分辨率卷积）Improvement 2 · Lightweight dense head (dropping the expensive high-resolution convolutions)

「密集预测头（如 DPT）中的高分辨率卷积层，尽管只占模型很小一部分参数，却为存储前向激活消耗了不成比例的 GPU 显存。」1/4 分辨率以上的卷积块被替换为「单个 MLP + pixel-shuffle 的轻量上采样头，显存极省且性能不降」。
纯卷积-free 解码器在 benchmark 上表现很好，但在室外场景产生方块伪影，因此保留了计算开销极小的早期低分辨率 DPT 卷积。"The high-resolution convolutional layers in dense prediction heads such as DPT account for only a very small share of model parameters, yet consume a disproportionate amount of GPU memory to store forward activations." Convolution blocks above 1/4 resolution are replaced by "a lightweight upsampling head consisting of a single MLP plus pixel-shuffle, which is extremely memory-frugal without any loss in performance".
A fully convolution-free decoder performs very well on benchmarks but produces block artifacts in outdoor scenes, so the early low-resolution DPT convolutions, whose compute cost is negligible, were retained.

改进三 · 单一密集头 + 多任务损失Improvement 3 · A single dense head + multi-task losses

「我们仍使用多任务损失，但只保留一个深度密集头和一个稀疏相机头。」模型虽不直接输出点图和 tracks，但仍以损失监督它们。训练损失包含："We still use a multi-task loss, but keep only one dense depth head and one sparse camera head." Although the model does not directly output point maps and tracks, losses still supervise them. The training loss comprises:

• 相机损失 —— ℓ1（「比 VGGT 使用的 Huber 损失更稳定」）。Camera loss — ℓ1 ("more stable than the Huber loss used by VGGT").
• 深度损失 —— aleatoric 不确定性 + 梯度一致性项 + 相对尺度。Depth loss — aleatoric uncertainty + a gradient-consistency term + relative scale.
• 点图损失 —— 与深度损失同形，残差来自预测深度+相机的反投影。Point-map loss — the same form as the depth loss, with residuals from back-projecting the predicted depth and camera.
• 匹配损失 —— 拉近正样本 token 对（同一 3D 位置）的特征、推开负样本。Matching loss — pulls together the features of positive token pairs (the same 3D location) and pushes negatives apart.

「这三项改造合计节省 70% 训练显存，并略微提升推理速度。」"Together, these three modifications save 70% of training memory and slightly improve inference speed."

动态场景重建Dynamic scene reconstruction

处理动态内容「解锁了数量级更多的互联网式视频用于训练」。模型「只预测深度图与相机参数，避免运动掩码等显式动态输出」。ray map 被否决，因为它「增加昂贵的稠密输出，并会把相机信息与逐像素外观变化纠缠在一起」——例如固定相机拍摄一名舞者，画面运动很大但相机参数不变。期望数据驱动的模型自己学到比手工低秩/局部刚性约束更好的运动先验。Handling dynamic content "unlocks orders of magnitude more internet-style video for training". The model "predicts only depth maps and camera parameters, avoiding explicit dynamic outputs such as motion masks". Ray maps were rejected because they "add an expensive dense output and entangle camera information with per-pixel appearance changes" — for example, a fixed camera filming a dancer produces large image motion while the camera parameters stay unchanged. The expectation is that a data-driven model learns better motion priors on its own than hand-crafted low-rank or local-rigidity constraints.

自监督训练（Teacher–Student）Self-supervised training (teacher–student)

借鉴 DINO 风格的动量教师-学生方法。两个网络都从监督训练好的 VGGT-Ω checkpoint 初始化，对同一组帧施加独立增强（颜色抖动、模糊、随机 90° 旋转、patch 遮挡、帧序打乱）。对齐到统一帧序后，student 通过跨层 ℓ2 特征匹配损失 + 相机/深度回归损失匹配 teacher；teacher 用 EMA 更新，且「相机与深度头在自监督期间冻结」以防坍塌。以此在 1800 万段无标注视频上训练。The recipe borrows the DINO-style momentum teacher-student approach. Both networks are initialized from a supervised VGGT-Ω checkpoint, and independent augmentations (color jitter, blur, random 90° rotation, patch masking, frame shuffling) are applied to the same set of frames. After realignment to a common frame order, the student matches the teacher through a cross-layer ℓ2 feature-matching loss plus camera/depth regression losses; the teacher is updated by EMA, and "the camera and depth heads are frozen during self-supervision" to prevent collapse. Training in this stage uses eighteen million unlabeled video clips.

03 训练数据Training data

「VGGT-Ω 的一个重要方面是扩展训练数据」——将大量公开数据集与一条专为处理现成视频中动态内容而构建的新标注管线结合。"An important aspect of VGGT-Ω is scaling up the training data" — combining a large body of public datasets with a new annotation pipeline built specifically to handle dynamic content in off-the-shelf video.

数据来源Data sources

30+ 公开数据集（Aria、Co3Dv2、DL3DV、Dynamic Replica、Hypersim、ScanNet、Waymo……）+ 内部数据，共「约 3M 序列，每个含 10 到 20,000 张图像」。剔除了 Kubric 与 PointOdyssey，「因为它们的背景几何是假的、深度无效」。30+ public datasets (Aria, Co3Dv2, DL3DV, Dynamic Replica, Hypersim, ScanNet, Waymo, ...) plus internal data, amounting to "about 3M sequences, each containing 10 to 20,000 images". Kubric and PointOdyssey were removed "because their background geometry is fake and their depth invalid".

数据标注管线Data annotation pipeline

从「约 4000 万条互联网式视频」出发、宁缺毋滥，「我们得到约 20 万动态场景和 60 万静态场景，带高质量相机与深度标注」。Starting from "roughly forty million internet-style videos" and favouring quality over quantity, "we obtain about two hundred thousand dynamic scenes and six hundred thousand static scenes with high-quality camera and depth annotations".

阶段Stage	做法Procedure
VLM 预筛VLM pre-filtering	VLM 判定 50% 的片段太难、40% 可重建但精度低，仅剩 10% 进入下一阶段；同时抽取「是否动态」等元数据。The VLM judges 50% of clips too hard and 40% reconstructable but of low accuracy, leaving only 10% to enter the next stage; it also extracts metadata such as whether the clip is dynamic.
动态掩码Dynamic masking	用 Grounding DINO 检测人、车等可动类别；这些区域从匹配、跟踪、验证中排除。Grounding DINO detects movable categories such as people and vehicles; those regions are excluded from matching, tracking and verification.
特征匹配与跟踪Feature matching and tracking	SIFT、SuperPoint & SuperGlue、ALIKED & LightGlue、VGGSfM Tracker 集成。An ensemble of SIFT, SuperPoint & SuperGlue, ALIKED & LightGlue, and the VGGSfM Tracker.
重建与过滤Reconstruction and filtering	RANSAC 内点过少时用 VGGT 初始化相机，再用 COLMAP 光束法平差；按注册率、视场角、畸变率启发式过滤；patch-based MVS 估计稠密深度。When RANSAC yields too few inliers, cameras are initialized with VGGT and then bundle-adjusted with COLMAP; heuristic filtering on registration rate, field of view and distortion rate; patch-based MVS estimates dense depth.
多视一致性Multi-view consistency	深度反投影、再投影到其它视图比较；有效深度像素 <5% 的序列丢弃。Depth is back-projected and re-projected into other views for comparison; sequences with <5% valid depth pixels are discarded.
监督几何过滤Supervised geometric filtering	用 camera-up 一致性、视差角、轨迹平滑度等手工特征，喂给在 500 静态 + 500 动态人工标注序列上训练的 XGBoost + 随机森林 + CatBoost 集成。Hand-crafted features such as camera-up consistency, parallax angle and trajectory smoothness feed an XGBoost + random forest + CatBoost ensemble trained on 500 static + 500 dynamic manually annotated sequences.

数据质量 —— 噪声会被「记住」Data quality — noise gets "memorized"

「拥有大量数据是不够的；数据的质量同样有强烈影响。」本文观察到「模型虽然学到了 3D 重建的一般原理，但也会死记硬背特异的噪声」。这些失败模式「不影响标准 benchmark 中的大多数图像，因此在定量结果里可能检测不到」。"Having a large amount of data is not enough; the quality of the data has just as strong an effect." The paper observes that "although the model learns the general principles of 3D reconstruction, it also memorizes idiosyncratic noise by rote". These failure modes "do not affect most images in standard benchmarks, and may therefore be undetectable in quantitative results".

04 实验Experiments

四个模型规模——200M / 500M / 1B / 10B——在 128 块 96GB H100 上（bfloat16、梯度检查点、FSDP）训练 240K 步（160K 监督 + 50K 自监督 + 30K 监督）。在三个静态 + 三个动态 benchmark 上评测。Four model scales — 200M / 500M / 1B / 10B — trained for 240K steps (160K supervised + 50K self-supervised + 30K supervised) on 128 96GB H100 GPUs (bfloat16, gradient checkpointing, FSDP). Evaluated on three static + three dynamic benchmarks.

基准测试Benchmarks

• 相机位姿（AUC）。在静态（7-Scenes、NRGBD、ETH3D）与动态（DyCheck、Sintel、TUM-Dynamic）上，「我们的模型在静态与动态数据集、严格与宽松阈值下都一致超越所有基线」。优化式 MegaSaM 在动态 Sintel 有竞争力，「但在宽基线或弱纹理场景退化」。Camera pose (AUC). On static (7-Scenes, NRGBD, ETH3D) and dynamic (DyCheck, Sintel, TUM-Dynamic) benchmarks, "our model consistently outperforms all baselines on both static and dynamic datasets and under both strict and loose thresholds". The optimization-based MegaSaM is competitive on dynamic Sintel, "but degrades on wide-baseline or weakly textured scenes".
• 深度（AbsRel↓ / δ₁.₂₅↑）。在静态上超越基线，「在动态场景中更甚」（Sintel：δ₁.₂₅ 86.1→93.5，AbsRel 0.118→0.081）。Depth (AbsRel↓ / δ₁.₂₅↑). It beats the baselines on static data, "and even more so on dynamic scenes" (Sintel: δ₁.₂₅ 86.1→93.5, AbsRel 0.118→0.081).
• 规模效应。「更大的 10B 变体一致地优于 1B 模型，说明扩大重建模型直接提升相机与深度精度。」Scaling effects. "The larger 10B variant consistently outperforms the 1B model, showing that scaling up reconstruction models directly improves camera and depth accuracy."

Sintel（动态）Sintel (dynamic)	之前最佳Previous best	VGGT-Ω	提升Gain
相机 AUC@3°Camera AUC@3°	22.5	40.0	+77%
相机 AUC@30°Camera AUC@30°	58.3	79.1	+35%
深度 δ₁.₂₅Depth δ₁.₂₅	74.1	93.5	+26%

消融（1B 模型，2M 序列）Ablations (1B model, 2M sequences)

设置Setting	Point error ↓	结论Takeaway
数据每步 10× 递增Data increased 10× per step	0.275 → 0.073	单调下降，类幂律Monotone decrease, roughly power-law
仅全局注意力Global attention only	0.071	基准Baseline
+ 25% register attention	0.073	「与原版几乎相同」"Almost identical to the original"
去掉点 + 匹配损失Remove point + matching losses	0.078（变差）0.078 (worse)	多任务损失有用The multi-task loss helps
VGGT 多头多任务设置VGGT's multi-head multi-task setup	0.070	略好但难规模化Slightly better but hard to scale
+ 10% 自监督步+ 10% self-supervised steps	0.073 → 0.070	略升；OOD 泛化更好A slight gain; better OOD generalization

标注质量。在 Sintel 上，本管线伪标签达到 96.4% AUC@30°（MegaSaM 62.1%）、深度 δ₁.₂₅ 99.3%（MegaSaM 77.2%）。「我们生成伪标签的目标不是最大化产量，而是只保留极有可能正确的序列与像素」——管线刻意保守。Annotation quality. On Sintel, the pseudo-labels from this pipeline reach 96.4% AUC@30° (MegaSaM 62.1%) and depth δ₁.₂₅ 99.3% (MegaSaM 77.2%). "Our goal in generating pseudo-labels is not to maximize yield, but to keep only those sequences and pixels that are highly likely to be correct" — the pipeline is deliberately conservative.

05 Register 的应用Applications of the registers

一个关键发现：「学到的寄存器可以在重建之外被复用」——它们「携带高层、很可能相当语义化的信息，可与语言空间对齐」。A key finding: "the learned registers can be reused beyond reconstruction" — they "carry high-level, quite possibly rather semantic information that can be aligned with language space".

06 局限性与坦诚观察Limitations and candid observations

在「Further Insights / Discussion」中，作者分享了「尚未严格证实、但我们认为对分享仍有用」的经验性观察。其中不少正是当前方法的局限与权衡。In the "Further Insights / Discussion" section, the authors share empirical observations that are "not yet rigorously verified, but that we believe are still useful to share". Quite a few of them are precisely the limitations and trade-offs of the current method.

纯 MLP 解码头产生方块伪影Pure-MLP decoding heads produce block artifacts

「纯 MLP 头会在预测深度图上产生可见的 patch/block 伪影，尽管它们在定量指标上常优于卷积方案、更快、更省显存。」该问题「在室外场景比室内更普遍，尤其当场景含远景物体时」（无界深度）。mipmap 风格监督、概率建模等补救「都没能可靠去除伪影」，因此折中保留少量浅层卷积。作者仍认为「纯 MLP 密集解码头是一个有前景、重要的方向」。"Pure-MLP heads produce visible patch/block artifacts in the predicted depth maps, even though they often beat convolutional variants on quantitative metrics, and are faster and more memory-frugal." The problem "is more prevalent in outdoor than indoor scenes, especially when the scene contains distant objects" (unbounded depth). Remedies such as mipmap-style supervision and probabilistic modelling "all failed to remove the artifacts reliably", so a few shallow convolutions were kept as a compromise. The authors still regard "a purely MLP-based dense decoding head as a promising and important direction".

自监督训练收益有限、且很难做好Self-supervised training brings limited gains and is hard to get right

「迄今我们发现它有助于改善模型泛化，尤其是 OOD 数据，但对大多数 benchmark 影响很小。要做得更好并非易事。」作者尝试了新视图合成、RayZer/E-RayZer 变体、token 生成、NeRF/高斯泼溅目标、token 掩码、时序等——「在我们的实现里只有 teacher-student 奏效」，且仍需预训练模型。「自监督重建对社区而言仍是开放问题。」"So far we have found that it helps model generalization, especially on OOD data, but has very little effect on most benchmarks. Doing better is far from easy." The authors tried novel-view synthesis, RayZer/E-RayZer variants, token generation, NeRF/Gaussian-splatting objectives, token masking, temporal cues and more — "in our implementation only teacher-student worked", and it still requires a pretrained model. "Self-supervised reconstruction remains an open problem for the community."

无界几何（如天空深度）难以处理Unbounded geometry (such as sky depth) is hard to handle

曾尝试一个预测无效区域的额外分支，希望模型忽略天空。「尽管模型准确预测了无效掩码，天空像素仍出现在深度估计的前景中，很可能是这些区域缺乏监督信号所致。」该预测头最终从模型中移除。An extra branch predicting invalid regions was tried, in the hope that the model would ignore the sky. "Although the model predicted the invalid mask accurately, sky pixels still appeared in the foreground of the depth estimate, most likely because these regions lack a supervision signal." That prediction head was eventually removed from the model.

预测归一化以稳定性换外观Prediction normalization trades stability for appearance

「一旦模型收敛，我们观察到训练时加不加预测归一化在定量性能上没有差别。归一化的主要好处是定性的……缺点是优化变得不稳定，学习曲线更陡，需要更细致的调参以避免梯度爆炸。」"Once the model has converged, we observe no difference in quantitative performance between training with and without prediction normalization. The main benefit of normalization is qualitative ... the downside is that optimization becomes unstable, the learning curves steeper, and more careful tuning is needed to avoid exploding gradients."

为简单性主动牺牲了部分精度Some accuracy was deliberately sacrificed for simplicity

使用 VGGT 的相机迭代细化、向密集头注入原始 RGB 等技巧，「在多个数据集上 AUC@3° 进一步提升 4%–6%、δ₁.₂₅ 约 2%」，但「我们刻意优先保持模型整体的简单性」，以便为社区提供更干净的基座。Tricks such as VGGT's iterative camera refinement and injecting raw RGB into the dense head give "a further 4%–6% on AUC@3° and about 2% on δ₁.₂₅ across several datasets", but "we deliberately prioritized keeping the model as a whole simple", so as to offer the community a cleaner base.

辅助输入在预训练阶段反而有害Auxiliary inputs are harmful during the pretraining stage

「在预训练阶段引入这些先验，即使随机或跨迭代掩码施加，往往是有害的。」只在微调阶段提供条件辅助输入（时序、相机、深度、尺度）「非常有效」。"Introducing these priors during pretraining, even when applied randomly or masked across iterations, tends to be harmful." Providing conditional auxiliary inputs (temporal cues, camera, depth, scale) only during fine-tuning is "very effective".

对训练数据噪声敏感，会记住特异噪声Sensitive to training-data noise; idiosyncratic noise is memorized

如「数据质量」一节，模型会「死记硬背特异噪声」（如把人当作地面的一部分），失败在标准 benchmark 上不可见——只能靠激进的数据过滤缓解，即强烈依赖数据清洗管线。As in the "data quality" section, the model "memorizes idiosyncratic noise by rote" (for instance treating a person as part of the ground), and these failures are invisible on standard benchmarks — they can only be mitigated by aggressive data filtering, which means a strong dependence on the data-cleaning pipeline.