Flow3r 提出以2D稠密对应关系（"光流"）作为监督信号，使视觉几何网络能够从无标注视频中学习，无需昂贵的3D/相机标注。核心洞察是：光流预测模块应当被"因子化"——用源视图的几何特征与目标视图的相机特征共同预测光流，从而同时指导场景几何与相机运动的学习，并自然推广至动态场景。Flow3r proposes using 2D dense correspondence ("optical flow") as the supervisory signal, enabling visual geometry networks to learn from unlabeled videos without expensive 3D/camera annotations. The key insight is that the flow prediction module should be "factored": flow is predicted from the geometry latents of the source view together with the camera latents of the target view, so that it guides the learning of both scene geometry and camera motion at once, and naturally generalizes to dynamic scenes.
关键词Keywordsfactored optical flowvisual geometry learning无标注视频监督unlabeled video supervision前馈式三维重建feed-forward 3D reconstruction相机位姿估计camera pose estimation动态场景重建dynamic scene reconstruction稠密对应关系dense correspondence伪标注流监督pseudo-label flow supervision可扩展几何学习scalable geometry learning多视角Transformermulti-view Transformer
01动机Motivation
当前最先进的前馈式3D/4D重建系统依赖稠密几何与相机位姿标注进行训练——这类标注在大规模场景下获取代价高昂，对于动态真实世界场景尤其稀缺。这一瓶颈使视觉几何学习无法像语言模型或视觉Transformer那样从海量无标注数据中扩展。State-of-the-art feed-forward 3D/4D reconstruction systems are trained with dense geometry and camera pose annotations, which are expensive to obtain at scale and particularly scarce for dynamic real-world scenes. This bottleneck keeps visual geometry learning from scaling on massive unlabeled data the way language models or vision Transformers do.
"Current feed-forward 3D/4D reconstruction systems rely on dense geometry and pose supervision — expensive to obtain at scale and particularly scarce for dynamic real-world scenes."
图1. Flow3r 总览。Flow3r 利用无标注视频（通过光流监督）配合有标注3D数据，实现可扩展的视觉几何学习。这使模型能够在野外场景下进行精准多视角3D重建，尤其对有标注数据极少的场景（如交互视频、动态场景）效果显著。Figure 1. Flow3r overview.Flow3r leverages unlabeled videos (supervised through optical flow) together with labeled 3D data to enable scalable visual geometry learning. This lets the model perform accurate multi-view 3D reconstruction in the wild, and is especially effective for settings where labeled data is scarce, such as interaction videos and dynamic scenes.
核心问题The Core Question
光流（两帧之间的像素级稠密对应关系）是经典多视角方法中的核心信号，也是近年稠密对应估计器的输出目标。
Flow3r 的关键研究问题是：如何利用光流有效地监督视觉几何学习？
现有方法（如 VGGT）虽尝试将光流预测作为辅助目标，但仅通过局部特征匹配预测光流，主要鼓励视觉判别特征，并不直接促进几何与相机位姿的学习。Optical flow, the pixel-level dense correspondence between two frames, is the central signal of classical multi-view methods and the output target of recent dense correspondence estimators. The key research question of Flow3r is: how can optical flow effectively supervise visual geometry learning? Existing methods such as VGGT do use flow prediction as an auxiliary objective, but they predict flow purely from local feature matching, which mainly encourages visually discriminative features and does not directly promote the learning of geometry and camera pose.
8静态+动态基准 全部达到 SOTASOTA on all static + dynamic benchmarks
Flow3r 在标准有监督视觉几何网络（如 π³、VGGT）基础上，附加一个因子化光流预测头，将源视图的几何特征（geometry latents）与目标视图的相机特征（camera latents）融合后，通过 DPT 头解码出稠密光流预测。训练时对有标注数据使用真实监督，对无标注视频则由教师模型（UFM）提供伪标注光流进行监督。On top of a standard supervised visual geometry network (such as π³ or VGGT), Flow3r attaches a factored flow prediction head that fuses the geometry latents of the source view with the camera latents of the target view and decodes dense flow through a DPT head. During training, labeled data is supervised with ground truth, while unlabeled videos are supervised by pseudo-ground-truth flow from a teacher model (UFM).
图2. 光流预测机制对比。(a) 视觉几何骨干网络：编码器输出相机 token 与 patch token；(b) 基于匹配的对应关系预测：直接从局部特征匹配预测光流（VGGT等方案）；(c) 基于投影几何的因子化：将预测的3D点显式投影至目标视图（仅适用于静态场景，对几何误差敏感）；(d) 本文的因子化光流预测：将源视图几何特征与目标视图相机特征在隐空间中融合后解码，自然扩展至动态场景。Figure 2. Comparison of flow prediction mechanisms.(a) Visual geometry backbone: the encoder outputs camera tokens and patch tokens; (b) matching-based correspondence prediction: flow is predicted directly from local feature matching (as in VGGT); (c) factorization through projective geometry: the predicted 3D points are explicitly projected into the target view (valid only for static scenes and sensitive to geometry errors); (d) our factored flow prediction: source-view geometry latents and target-view camera latents are fused in latent space and then decoded, which extends naturally to dynamic scenes.
对于静态场景，源视图 i 到目标视图 j 的光流仅由源视图几何（全局点图）和目标相机位姿决定。
受此启发，Flow3r 不进行解析投影，而是训练一个学习型非对称光流预测模块：For static scenes, the flow from source view i to target view j is determined solely by the geometry of the source view (the global point map) and the pose of the target camera. Inspired by this, Flow3r does not perform analytic projection but trains a learned asymmetric flow prediction module:
F̂i→j = Φflow(gi, cj)
其中 gi 为源视图几何特征，cj 为目标视图相机特征（来自多视角 Transformer 的输出）。
这一设计绕过了对显式几何元素的解码，提升了鲁棒性，并使模型能够端对端训练。
对于动态场景，光流自然被理解为相机运动与场景运动的叠加，无需额外设计。Here gi denotes the geometry latents of the source view and cj the camera latents of the target view (both produced by the multi-view Transformer). This design bypasses decoding explicit geometric entities, improves robustness and lets the model be trained end to end. For dynamic scenes, flow is naturally understood as the superposition of camera motion and scene motion, with no extra design needed.
总体架构与训练目标Overall Architecture and Training Objective
图3. Flow3r 总体架构。多视角 Transformer 同时输出相机 token（ci）和 patch token（xi）。对有标注数据，直接用真实相机位姿与几何监督相应输出。对无标注数据，将融合后的 patch token 与相机 token 一同送入光流预测头，输出预测光流并与教师模型（Flow Teacher，即 UFM）产生的伪标注对比计算损失。Figure 3. Overall architecture of Flow3r.The multi-view Transformer outputs camera tokens (ci) and patch tokens (xi) at the same time. For labeled data, the corresponding outputs are supervised directly with ground-truth camera poses and geometry. For unlabeled data, the fused patch tokens and camera tokens are fed to the flow prediction head, and the predicted flow is compared against the pseudo-labels produced by the teacher model (the Flow Teacher, i.e. UFM) to compute the loss.
Flow3r 在两阶段训练：首先冻结骨干网络，仅在有标注数据上训练光流头；然后解冻全部参数，结合有标注与无标注数据端对端微调。
流监督损失采用鲁棒回归损失（广义 Charbonnier loss），并使用可视性掩码 C 屏蔽遮挡区域：Flow3r is trained in two stages: the backbone is first frozen and only the flow head is trained on labeled data; then all parameters are unfrozen and fine-tuned end to end on labeled and unlabeled data together. The flow supervision loss uses a robust regression loss (a generalized Charbonnier loss) and a visibility mask C to mask out occluded regions:
flow-tracking（VGGT 方案）flow-tracking (the VGGT approach)
基于双视图 patch 特征的匹配头A matching head built on two-view patch features
主要学习视觉判别特征Mainly learns visually discriminative features
不直接促进几何或相机位姿学习Does not directly promote the learning of geometry or camera pose
实验表明几乎无法提升几何质量Experiments show it barely improves geometry quality
flow-projective（解析投影方案）flow-projective (the analytic projection approach)
将预测3D点显式投影至目标相机Explicitly projects the predicted 3D points into the target camera
仅适用于静态场景Applicable only to static scenes
对解码几何误差敏感，甚至降低性能Sensitive to errors in the decoded geometry, and can even degrade performance
Flow3r（flow-factored）在所有场景上均优于此方案Flow3r (flow-factored) outperforms this approach on all scenes
03实验Experiments
Flow3r 在两类设置下评估：(1) 受控对照实验，验证因子化光流的有效性；(2) 将其集成进 π³ 与 VGGT 骨干，在8个基准（4个动态 + 4个静态）上对比前沿前馈方法。
有标注数据集涵盖 CO3Dv2、Habitat、ARKitScenes、ScanNet++ 等约11个多视角重建数据集（约34K序列），无标注视频来自 Kinetics-700、SpatialVID、EPIC-KITCHENS（~800K序列）。Flow3r is evaluated in two settings: (1) controlled experiments that verify the effectiveness of factored flow; (2) integration into the π³ and VGGT backbones, compared against state-of-the-art feed-forward methods on 8 benchmarks (4 dynamic + 4 static). The labeled data covers about 11 multi-view reconstruction datasets including CO3Dv2, Habitat, ARKitScenes and ScanNet++ (about 34K sequences), and the unlabeled videos come from Kinetics-700, SpatialVID and EPIC-KITCHENS (~800K sequences).
Table 1：因子化光流对视觉几何学习的提升（受控实验）Table 1: Gains of factored flow for visual geometry learning (controlled experiment)
模型变体Model variant
静态 RRA@30↑Static RRA@30↑
静态 RTA@30↑Static RTA@30↑
静态 CD↓Static CD↓
静态 MSE↓Static MSE↓
动态 RRA@30↑Dynamic RRA@30↑
动态 RTA@30↑Dynamic RTA@30↑
动态 MSE↓Dynamic MSE↓
3d-sup（无无标注数据基线）3d-sup (baseline without unlabeled data)
0.7500
0.6929
0.030
0.088
66.01
62.37
0.637
flow-projective
0.6700
0.4572
0.033
0.088
61.23
56.12
0.710
flow-tracking
0.7438
0.7021
0.030
0.089
68.56
62.95
0.628
flow-factored（Flow3r）flow-factored (Flow3r)
0.7700
0.7366
0.026
0.078
76.26
68.84
0.598
Table 2：动态场景基准测试对比Table 2: Comparison on dynamic-scene benchmarks
方法Method
Kinetics700 MSE↓
Kinetics700 f-score@5↑
EPIC-KITCHENS MSE↓
EPIC-KITCHENS f-score@5↑
Sintel f-score@5↑
Bonn f-score@5↑
DUSt3R
0.312
0.528
0.338
0.493
0.271
0.800
CUT3R
0.303
0.575
0.338
0.493
0.676
0.899
VGGT
0.220
0.617
0.220
0.620
0.595
0.884
π³
0.200
0.620
0.200
0.620
0.523
0.905
Flow3r（本文）Flow3r (ours)
0.256
0.599
0.199
0.622
0.426
0.954
Flow3r 在所有动态数据集上一致优于 DUSt3R、CUT3R、VGGT 和 π³。Flow3r consistently outperforms DUSt3R, CUT3R, VGGT and π³ on every dynamic dataset.
Table 3：静态场景基准测试对比Table 3: Comparison on static-scene benchmarks
方法Method
CO3Dv2 RTA@30↑
CO3Dv2 f-score@5↑
ScanNet RTA@30↑
ScanNet f-score@5↑
NRGBD f-score@5↑
7-Scenes f-score@5↑
DUSt3R
90.68
0.783
57.14
0.831
93.25
0.714
CUT3R
90.68
0.737
71.39
0.740
95.63
0.695
VGGT
87.62
0.884
89.17
0.931
99.21
0.665
π³
97.49
0.905
91.44
0.964
99.20
0.737
Flow3r（本文）Flow3r (ours)
97.62
0.876
92.89
0.943
99.60
0.807
图4. 因子化光流促进视觉几何学习。与基线（3d-sup）和其他光流监督变体（flow-projective、flow-tracking）相比，Flow3r（flow-factored）能产生更精准的场景重建，突显无标注数据与因子化光流设计的优势。Figure 4. Factored flow promotes visual geometry learning.Compared with the baseline (3d-sup) and the other flow-supervision variants (flow-projective, flow-tracking), Flow3r (flow-factored) yields more accurate scene reconstructions, highlighting the benefit of unlabeled data and of the factored flow design.
无标注数据扩展性（Table 4）Scaling with Unlabeled Data (Table 4)
固定1K有标注 OmniWorld 序列，逐步增加无标注 SpatialVID 数据量：Fixing 1K labeled OmniWorld sequences, unlabeled SpatialVID data is added progressively:
有标注序列数Labeled sequences
无标注序列数Unlabeled sequences
RRA@30↑
MSE↓
OmniWorld (1K)
—
66.01
0.637
OmniWorld (1K)
SpatialVID (3K)
76.26
0.598
OmniWorld (1K)
SpatialVID (10K)
78.45
0.560
OmniWorld (1K)
SpatialVID (20K)
81.12
0.532
OmniWorld (4K，参考基线）
—
78.68
0.565
OmniWorld (4K, reference baseline)
使用20K条无标注视频（1K有标注 + 20K无标注）可超越4K有标注数据的模型，证明因子化光流监督的扩展性。With 20K unlabeled videos (1K labeled + 20K unlabeled) the model surpasses one trained on 4K labeled sequences, showing the scalability of factored flow supervision.
图5. 无标注视频扩展效果。随着 SpatialVID 无标注数据量不断增加，动态场景重建质量持续提升，甚至超越使用更多有标注数据训练的模型，验证了因子化光流监督的可扩展性。Figure 5. Effect of scaling unlabeled video.As the amount of unlabeled SpatialVID data keeps growing, dynamic-scene reconstruction quality keeps improving and even surpasses models trained with more labeled data, confirming the scalability of factored flow supervision.
定性结果对比Qualitative Comparison
图6. 野外数据定性结果。在静态场景、动态场景和交互视频上，与 VGGT、π³、CUT3R 等前馈方法对比重建效果。Flow3r 能推断出更整洁、更准确的场景结构，而基线方法常出现错位输出或错误的运动估计。Figure 6. Qualitative results in the wild.Reconstructions on static scenes, dynamic scenes and interaction videos are compared with feed-forward methods such as VGGT, π³ and CUT3R. Flow3r infers cleaner and more accurate scene structure, whereas the baselines often produce misaligned outputs or wrong motion estimates.
消融实验（Table 5）Ablation Study (Table 5)
以 VGGT 和 π³ 为骨干，逐步加入有标注流监督与无标注流监督：With VGGT and π³ as backbones, labeled and unlabeled flow supervision are added step by step:
+ 有标注数据上的流监督（多任务）：提升有限+ flow supervision on labeled data (multi-task): limited gain
+ 有标注+无标注数据的完整流监督（Flow3r）：EPIC-KITCHENS f-score = 0.622，7-Scenes f-score = 0.807+ full flow supervision on labeled and unlabeled data (Flow3r): EPIC-KITCHENS f-score = 0.622, 7-Scenes f-score = 0.807
无标注数据带来的增益主要来自无标注视频，而非多任务学习本身。The gain comes mainly from the unlabeled videos, not from multi-task learning itself.
04局限性Limitations
说明：论文第5节（Discussion）明确陈述了以下三点局限性，均为作者明确指出（stated），非推断。Note: Section 5 (Discussion) of the paper explicitly states the three limitations below; all are stated by the authors rather than inferred.
依赖教师模型提供伪标注（Teacher Model Dependency）Dependence on a teacher model for pseudo-labels (Teacher Model Dependency)
"Flow3r relies on off-the-shelf models to provide pseudo-ground-truth flow supervision, and there can be domains where such 2D prediction fails." — 在某些分布外领域（如医学图像、航拍等），预训练2D光流模型可能给出低质量伪标注，从而影响几何学习。in some out-of-distribution domains (medical images, aerial imagery and the like), a pretrained 2D flow model may give low-quality pseudo-labels and thereby harm geometry learning.
"Although our factored flow formulation elegantly handles dynamic scenes…, Flow3r may struggle under complex scenes with multiple independently moving components." — 当场景中存在多个独立运动物体时，单一相机特征 token 可能无法完整编码所有运动信息。when a scene contains several independently moving objects, a single camera latent token may not encode all of the motion information.
当前实验规模距真正大规模仍有差距（Scale Gap to Web-Scale）Current experiments are still short of truly large scale (Scale Gap to Web-Scale)
"Our current experiments operate at a moderate scale (~800K video sequences for flow supervision), and scaling to truly large-scale settings (~10-100M videos) presents an exciting but unexplored direction." — 尽管~800K序列已展现出良好的扩展性，更大规模训练的潜力尚未被探索，且对算力与存储提出了新挑战。although ~800K sequences already show good scaling behaviour, the potential of larger-scale training remains unexplored and places new demands on compute and storage.
推断（inferred）：因子化光流对独立流估计并非最优Inferred: factored flow is suboptimal for standalone flow estimation
作者本人指出：The authors themselves note: "our factored flow prediction is suboptimal for standalone flow estimation — since it enforces an information bottleneck by conditioning on the target-view camera token rather than patch features that contain richer visual cues." 即该设计优化的是几何监督信号质量，而非光流预测本身的精度。That is, the design optimizes the quality of the geometric supervision signal rather than the accuracy of flow prediction itself.
