以几何感知特征统一跨视角（第一/第三人称）实例分割Unifying cross-view (first-person / third-person) instance segmentation with geometry-aware features
Yulu Gao*, Bohao Zhang*, Zongheng Tang, Jitong Liao, Wenjun Wu, Si Liu — Beihang University / Hangzhou International Innovation Institute of Beihang University
VGGT-Segmentor（VGGT-S）将 VGGT 的几何感知跨视角特征表示与全新的 Union Segmentation Head 相结合，
解决 ego–exo 场景下直接像素投影产生的系统性漂移问题，实现精确的实例级跨视角分割。
在 Ego–Exo4D 基准上，VGGT-S 分别达到 67.7%（Ego→Exo）
和 68.0%（Exo→Ego）的 average IoU，大幅超越此前最优方法。VGGT-Segmentor (VGGT-S) combines the geometry-aware cross-view feature representation of VGGT with a new Union Segmentation Head,
resolving the systematic drift produced by direct pixel projection in ego–exo scenes and achieving accurate instance-level cross-view segmentation.
On the Ego–Exo4D benchmark, VGGT-S reaches an average IoU of 67.7% (Ego→Exo)
and 68.0% (Exo→Ego) respectively, substantially surpassing the previous best methods.
在 ego（第一人称）与 exo（第三人称）视角之间定位并分割同一实例是 embodied AI 和远程协作的核心挑战。
由于视角、尺度与遮挡的剧烈变化，直接在像素级别进行匹配极其困难。Locating and segmenting the same instance across ego (first-person) and exo (third-person) views is a core challenge for embodied AI and remote collaboration.
Because viewpoint, scale and occlusion change drastically, matching directly at the pixel level is extremely difficult.
"While recent geometry-aware models like VGGT provide a strong foundation for feature alignment,
we find they often fail at dense prediction tasks due to significant pixel-level projection drift,
even when their internal object-level attention remains consistent."
作者发现 VGGT 在 ego–exo 场景下呈现出一种矛盾现象：其内部 attention 能够稳定地聚焦于目标对象区域，
但像素级点投影却存在系统性漂移（如图 1 中心列所示）。
这表明 VGGT 的特征对齐能力本质上是可靠的，问题在于如何将这种"对象级一致性"转化为精确的像素级分割结果。The authors observe a contradictory phenomenon for VGGT in ego–exo scenes: its internal attention focuses stably on the target object region,
yet pixel-level point projection suffers from systematic drift (see the center column of Figure 1).
This indicates that the feature alignment capability of VGGT is intrinsically reliable; the problem is how to turn this "object-level consistency" into accurate pixel-level segmentation results.
图 1：VGGT 跨视角对应关系可视化。
左：源图像（带有源采样点）。
中：目标图像中直接应用 VGGT 点投影的结果 — 可见明显的系统性漂移与错位。
右：源图像中的星形标记与目标图像中 VGGT 内部 attention map 的对应关系 — 说明 VGGT 在对象级别仍保持跨视角的一致对齐。Figure 1: Visualization of VGGT cross-view correspondence.
Left: the source image (with source sampling points).
Middle: the result of directly applying VGGT point projection in the target image — clear systematic drift and misalignment are visible.
Right: the correspondence between the star markers in the source image and the internal VGGT attention map in the target image — showing that VGGT still maintains consistent cross-view alignment at the object level.
67.7%Ego→Exo average IoU（supervised）Ego→Exo average IoU (supervised)
68.0%Exo→Ego average IoU（supervised）Exo→Ego average IoU (supervised)
54.1%Ego→Exo average IoU（zero-shot）Ego→Exo average IoU (zero-shot)
80.7%MvMHAT AP（跨数据集泛化）MvMHAT AP (cross-dataset generalization)
02方法 · MethodMethod
VGGT-S 以 VGGT encoder 为骨干，引入 Union Segmentation Head，
通过三个协同阶段将高层特征对齐转化为精确的分割 mask：
Mask Prompt Fusion → Point-Guided Prediction → Iterative Mask Refinement。VGGT-S takes the VGGT encoder as its backbone and introduces a Union Segmentation Head
that turns high-level feature alignment into accurate segmentation masks through three cooperating stages:
Mask Prompt Fusion → Point-Guided Prediction → Iterative Mask Refinement.
图 2：VGGT-S 整体架构与三阶段 Union Segmentation Head。
(A) 总体流程：输入图像对经 VGGT encoder 提取几何感知特征，再经 DPT decoder 得到特征图 Fs、Ft，
随后交由 Union Segmentation Head 处理。
(B) Mask Prompt Fusion：将源 mask Ms 编码后注入源特征，经 Bottleneck Fusion 模块实现双视角交互。
(C) Point-Guided Prediction：K-Means 采样代表点并由 VGGT tracking head 投影至目标帧，
结合点嵌入与图像特征进行迭代解码，输出初始 mask。
(D) Mask Refinement：两轮迭代逐步精化边界与遮挡区域。Figure 2: The overall VGGT-S architecture and the three-stage Union Segmentation Head.
(A) Overall pipeline: the input image pair goes through the VGGT encoder to extract geometry-aware features, then through a DPT decoder to obtain the feature maps Fs and Ft,
which are subsequently handed to the Union Segmentation Head.
(B) Mask Prompt Fusion: the source mask Ms is encoded and injected into the source features, and a Bottleneck Fusion module realizes dual-view interaction.
(C) Point-Guided Prediction: K-Means samples representative points that the VGGT tracking head projects onto the target frame;
point embeddings and image features are decoded iteratively to output the initial mask.
(D) Mask Refinement: two rounds of iteration progressively refine boundaries and occluded regions.
Stage 1 · Mask Prompt Fusion
源 mask Ms 通过卷积编码为高维嵌入 Em，直接加到源特征图 Fs 上，
形成 Fs′。随后，Bottleneck Fusion 模块先将 Fs′ 和 Ft
下采样至较低分辨率（实验中为 37×37），经 Self-Attention 捕获双视角交互后再上采样回原分辨率，
产生携带目标语义的融合特征 Fs★ 和 Ft★。
该步骤以紧凑的表示统一了 mask 语义与跨视角几何信息。The source mask Ms is encoded by convolution into a high-dimensional embedding Em and added directly to the source feature map Fs,
forming Fs′. The Bottleneck Fusion module then first downsamples Fs′ and Ft
to a lower resolution (37×37 in the experiments), captures the dual-view interaction with Self-Attention and upsamples back to the original resolution,
producing the fused features Fs★ and Ft★ that carry the target semantics.
This step unifies mask semantics and cross-view geometric information in a compact representation.
Stage 2 · Point-Guided Prediction
从源 mask 中用 K-Means 聚类采样 5 个代表点（默认），
利用 VGGT 的 tracking head 将其投影到目标帧，得到目标域的初始点位置。
点嵌入、图像特征与可学习 token 共同构成初始 query，
经多个带有 self-attention 和双向 point-to-image cross-attention 的轻量 decoder block 迭代细化，
输出对目标区域的初步预测 mask。这一设计利用了 VGGT 对象级别对齐的稳定性，
绕开了像素级投影的不可靠性。Five representative points are sampled from the source mask by K-Means clustering (the default),
and the tracking head of VGGT projects them onto the target frame to obtain the initial point locations in the target domain.
Point embeddings, image features and learnable tokens jointly form the initial query,
which is refined iteratively by several lightweight decoder blocks with self-attention and bidirectional point-to-image cross-attention,
outputting a preliminary predicted mask for the target region. This design exploits the stability of the object-level alignment of VGGT
and bypasses the unreliability of pixel-level projection.
Stage 3 · Iterative Mask Refinement
对初步 mask 进行两轮（默认）迭代精化：每轮通过 dot-product 操作在目标特征图上逐步锐化边界、
填补遮挡区域，同时保持较低的计算开销。完整推理延迟为 161.4 ms（标准配置 518×518 输入）。The preliminary mask undergoes two rounds of iterative refinement (the default): each round uses dot-product operations on the target feature map to progressively sharpen boundaries
and fill occluded regions while keeping the computational cost low. The full inference latency is 161.4 ms (standard configuration with 518×518 input).
Single-Image Self-Supervised Training
为消除对成对标注的依赖，论文提出单图自监督训练策略：对同一图像施加两类 augmentation 族——
VGGT-adaptive（缩放、轻微旋转，几何对应关系仍有效）与 VGGT-non-adaptive（大旋转、翻转，
需对点位置加入合成扰动）。在 SA-1B 数据集的 1/20 子集上预训练即可获得强大的迁移能力，
实现无配对标注的 correspondence-free 预训练变体。To remove the dependence on paired annotations, the paper proposes a single-image self-supervised training strategy: two families of augmentation are applied to the same image —
VGGT-adaptive (scaling and slight rotation, where the geometric correspondence remains valid) and VGGT-non-adaptive (large rotation and flipping,
which require synthetic perturbation of the point locations). Pre-training on a 1/20 subset of the SA-1B dataset already yields strong transfer ability,
giving a correspondence-free pre-training variant that needs no paired annotations.
图 4：Union Segmentation Head 的效果可视化。
尽管 VGGT 将点投影到错误位置，Union Segmentation Head 仍能将预测 mask 调整到几何一致的正确位置。
放大后效果更为明显。Figure 4: Visualization of the effect of the Union Segmentation Head.
Even though VGGT projects the points to wrong locations, the Union Segmentation Head can still adjust the predicted mask to the geometrically consistent, correct position.
The effect is more obvious when zoomed in.
03实验 · ExperimentsExperiments
主要在 Ego–Exo4D 基准（ego–exo 实例分割）上评估，同时在 MvMHAT（多视角多人关联）上验证跨数据集泛化能力。
评价指标使用 average IoU（Ego–Exo4D）和 AP（MvMHAT）。Evaluation is carried out mainly on the Ego–Exo4D benchmark (ego–exo instance segmentation), while cross-dataset generalization is verified on MvMHAT (multi-view multi-human association).
The evaluation metrics are average IoU (Ego–Exo4D) and AP (MvMHAT).
在 MvMHAT 数据集上仅微调 1 epoch，AP 即达 80.7%，超越 DOMR 9.6%，
验证了 correspondence-free 预训练的强迁移性。Fine-tuning for only 1 epoch on the MvMHAT dataset already brings AP to 80.7%, exceeding DOMR by 9.6%,
which confirms the strong transferability of correspondence-free pre-training.
消融实验（Table 3）Ablation study (Table 3)
配置Configuration
Ego→Exo 提升Ego→Exo gain
Exo→Ego 提升Exo→Ego gain
+ Bottleneck Fusion (BF)
+14.7%
+15.2%
+ Point-Guided Prediction (PGP)
+12.0%
+11.2%
+ Mask Refinement (MR)
+5.5%
+4.5%
完整模型full model
+32.2%
+30.9%
三个组件各自贡献显著，BF 带来最大增益（引入 mask 语义与双视角交互），
PGP 次之（利用点投影弥补像素漂移），MR 进一步精化边界。
超参分析表明：Bottleneck Fusion 分辨率 37×37、采样点数 5、精化迭代次数 2 为最优配置。Each of the three components contributes significantly: BF brings the largest gain (introducing mask semantics and dual-view interaction),
PGP comes next (using point projection to compensate for pixel drift), and MR further refines boundaries.
Hyper-parameter analysis shows that a Bottleneck Fusion resolution of 37×37, 5 sampled points and 2 refinement iterations form the optimal configuration.
图 3：VGGT-S 与 DOMR 的定性对比。
第一行（Ego→Exo）：DOMR 误将砧板识别为目标，VGGT-S 正确定位锅。
第二行（Exo→Ego）：场景中存在两个外观相似的瓶子，DOMR 因缺乏几何信息出现混淆，
VGGT-S 凭借几何线索准确区分目标瓶子。Figure 3: Qualitative comparison between VGGT-S and DOMR.
First row (Ego→Exo): DOMR mistakes the cutting board for the target, while VGGT-S correctly locates the pot.
Second row (Exo→Ego): the scene contains two bottles of similar appearance, and DOMR is confused for lack of geometric information,
whereas VGGT-S tells the target bottle apart accurately thanks to geometric cues.
04局限性 · LimitationsLimitations
Note:原论文无独立的 Limitations 章节。以下各点结合论文结论与设计分析得出，
标注来源（stated = 论文明确提及；inferred = 根据设计推断）。The original paper has no standalone Limitations section. The points below are derived from the paper's conclusion together with an analysis of the design,
with the source annotated (stated = explicitly mentioned in the paper; inferred = inferred from the design).
依赖 VGGT 的特征质量（stated + inferred）Reliance on the feature quality of VGGT (stated + inferred)
VGGT-S 的核心假设是 VGGT 内部 attention 在对象级别保持一致性。
在极端遮挡或 VGGT 本身特征失效的场景下，Union Segmentation Head 的纠正能力有限。
（inferred from design）The core assumption of VGGT-S is that the internal attention of VGGT stays consistent at the object level.
In scenes with extreme occlusion, or where the VGGT features themselves break down, the corrective ability of the Union Segmentation Head is limited.
(inferred from design)
自监督训练数据规模限制（inferred）Limited scale of the self-supervised training data (inferred)
当前 correspondence-free 预训练仅使用 SA-1B 的 1/20 子集；
扩大至全量数据或引入更多多样化场景是否能进一步提升性能尚未验证。
（inferred from design）The current correspondence-free pre-training uses only a 1/20 subset of SA-1B;
whether scaling up to the full data or introducing more diverse scenes would further improve performance has not been verified.
(inferred from design)
推理延迟与实时部署（inferred）Inference latency and real-time deployment (inferred)
完整 pipeline 在标准 518×518 输入下延迟为 161.4 ms，
对于需要实时反馈的 embodied AI 应用场景可能仍有压力，论文未对模型压缩或加速进行讨论。
（inferred from design）The full pipeline has a latency of 161.4 ms at the standard 518×518 input,
which may still be demanding for embodied AI applications that require real-time feedback; the paper does not discuss model compression or acceleration.
(inferred from design)
任务范围局限于双视角实例分割（stated）Task scope limited to two-view instance segmentation (stated)
VGGT-S 当前设计针对单对（ego + exo）双视角的实例级分割；
论文结论提到该框架是 "a simple yet scalable solution"，
但推广至多视角或视频序列的连续帧尚未在本文探讨。
（stated in conclusion）The current VGGT-S design targets instance-level segmentation for a single pair (ego + exo) of views;
the paper's conclusion calls the framework "a simple yet scalable solution",
but extending it to multiple views or to consecutive frames of a video sequence is not explored in this work.
(stated in conclusion)
