INSID3: Training-Free In-Context Segmentation with DINOv3
无需训练、无需辅助模型，仅凭单个冻结 DINOv3 backbone 实现多粒度上下文分割No training and no auxiliary models: multi-granularity in-context segmentation from a single frozen DINOv3 backbone
Claudia Cuttano*, Gabriele Trivigno*, Christoph Reich, Daniel Cremers, Carlo Masone, Stefan Roth · Politecnico di Torino / TU Darmstadt / TU Munich / hessian.AI / ELIZA / MCML
INSID3 提出了一种极简主义的 in-context segmentation（ICS）框架：给定一个标注好的参考示例，模型仅依靠冻结的 DINOv3 特征，在无需任何 fine-tuning、分割解码器或辅助模型的情况下，对目标图像中的语义类别、object part 或个性化实例完成分割，并在多个 benchmark 上全面超越此前方法。INSID3 introduces a minimalist in-context segmentation (ICS) framework: given a single annotated reference example, the model relies only on frozen DINOv3 features to segment semantic categories, object parts or personalized instances in the target image, requiring no fine-tuning, no segmentation decoder and no auxiliary model, while consistently outperforming prior methods across multiple benchmarks.
in-context segmentationtraining-freeDINOv3positional debiasingagglomerative clusteringone-shot 分割one-shot segmentation跨域泛化cross-domain generalization自监督表征self-supervised representation视觉基础模型vision foundation model
01动机（Motivation）Motivation
In-context segmentation（ICS）旨在给定一个或几个标注示例，分割任意概念——无论是 object、part 还是个性化实例。现有方案存在两条路线的固有局限：(1) fine-tuning 路线提升了域内精度，但损害了泛化能力；(2) 多模型组合路线保留了泛化性，但引入了架构复杂度，且 SAM 的分割粒度固定，无法灵活覆盖 semantic / part / personalized 三个层次。In-context segmentation (ICS) aims to segment arbitrary concepts — objects, parts or personalized instances — given one or a few annotated examples. Existing solutions suffer from the inherent limitations of two lines of work: (1) fine-tuning improves in-domain accuracy but hurts generalization; (2) multi-model pipelines preserve generalization but introduce architectural complexity, and the segmentation granularity of SAM is fixed, so it cannot flexibly cover the semantic / part / personalized levels.
"Can a single self-supervised backbone support both semantic matching and segmentation, without any supervision or auxiliary models?"
作者发现，大规模自监督预训练的 DINOv3 密集特征同时具备强语义对应能力和空间连贯结构，但其中混杂了显著的 positional bias——特征相似性会被图像内的相对空间位置所污染，必须加以去除才能获得可靠的跨图像匹配。The authors observe that the dense features of large-scale self-supervised DINOv3 provide both strong semantic correspondence and spatially coherent structure, but are entangled with a pronounced positional bias — feature similarity is contaminated by relative spatial position inside the image, and this bias must be removed to obtain reliable cross-image matching.
Figure 1.INSID3 综合结果与概览：直接从冻结 DINOv3 特征完成 in-context segmentation，无需任何解码器、fine-tuning 或辅助模型。图中雷达图展示了与训练型方法（橙色）和 training-free 方法（蓝色）的对比，INSID3（绿色）在九个 benchmark 上全面领先。Overall results and overview of INSID3: in-context segmentation performed directly from frozen DINOv3 features, without any decoder, fine-tuning or auxiliary model. The radar chart compares INSID3 (green) with training-based methods (orange) and training-free methods (blue); INSID3 leads across all nine benchmarks.
+7.5%mIoU 平均提升，相较最强 training-free 基线 GF-SAMaverage mIoU gain over GF-SAM, the strongest training-free baseline
3×参数量更少（304 M vs 945 M），无需 SAMfewer parameters (304 M vs 945 M), no SAM required
9个 benchmark 覆盖语义、part、个性化分割benchmarks covering semantic, part and personalized segmentation
0额外监督：无 mask 标注、无类别标签、无 fine-tuningextra supervision: no mask annotations, no class labels, no fine-tuning
02方法（Method）Method
INSID3 流程共四步：① Positional Debiasing（去除位置偏差），② Fine-grained Clustering（聚类分解目标图像为连贯区域），③ Seed-cluster Selection（通过 backward correspondence 筛选候选簇，再用 cross-image similarity 定位种子簇），④ Cluster Aggregation（结合 cross-similarity 与 self-similarity 生成最终 mask）。The INSID3 pipeline has four steps: ① Positional Debiasing (removing the positional bias), ② Fine-grained Clustering (decomposing the target image into coherent regions by clustering), ③ Seed-cluster Selection (filtering candidate clusters via backward correspondence, then locating the seed cluster via cross-image similarity), ④ Cluster Aggregation (combining cross-similarity and self-similarity to produce the final mask).
Figure 3.INSID3 方法概览。参考图与目标图的 DINOv3 特征首先经过 positional debiasing，再对目标图做 agglomerative clustering 得到结构化分解；通过 backward correspondence 保留候选簇，再由参考区域原型定位 seed cluster；最终融合 cross-image similarity 与 self-similarity 扩展 seed cluster 得到预测 mask。Overview of the INSID3 method. The DINOv3 features of the reference and target images first undergo positional debiasing; agglomerative clustering on the target image then yields a structured decomposition; backward correspondence keeps the candidate clusters, and the reference-region prototype locates the seed cluster; finally cross-image similarity and self-similarity are fused to expand the seed cluster into the predicted mask.
① Positional Debiasing（位置偏差去除）① Positional Debiasing
DINOv3 特征存在显著的 positional bias：即便使用均匀低复杂度图像（"noise images"），提取的特征仍保留明显的空间结构，使得跨图像的相似度图被位置信息污染。INSID3 在噪声图像上提取特征矩阵 Fnoise ∈ ℝP×D，对其做 PCA 取前 s 个主成分构成位置子空间 B，随后将参考图和目标图特征投影到该子空间的正交补空间：
F̃ = F(1D − BB⊤)，从而在不需要任何训练的情况下有效解耦语义信息与位置信息。DINOv3 features carry a pronounced positional bias: even on uniform, low-complexity images ("noise images") the extracted features retain a clear spatial structure, so that cross-image similarity maps are contaminated by positional information. INSID3 extracts a feature matrix Fnoise ∈ ℝP×D on noise images, applies PCA and keeps the top s principal components as the positional subspace B, then projects the reference and target features onto the orthogonal complement of that subspace: F̃ = F(1D − BB⊤), effectively disentangling semantic from positional information without any training.
Figure 2.DINOv3 密集特征的区域级分组。每对图像展示原图（左）与对 DINOv3 特征做 agglomerative clustering 后的聚类图（右）——自然形成语义连贯的区域分解，为后续 in-context segmentation 提供结构化表示。Region-level grouping of DINOv3 dense features. Each pair shows the original image (left) and the clustering map obtained by agglomerative clustering of the DINOv3 features (right) — semantically coherent region decompositions emerge naturally and provide a structured representation for the subsequent in-context segmentation.
② Fine-grained Clustering（细粒度聚类）② Fine-grained Clustering
对去偏后的目标图特征 F̃t 做 agglomerative clustering，聚类粒度由阈值 τ 控制（论文设置 τ = 0.6）。聚类结果将目标图分解为一组空间连贯的候选区域 {Gk}，其空间结构对 part 和个性化分割尤为关键。Agglomerative clustering is applied to the debiased target features F̃t, with the clustering granularity controlled by a threshold τ (the paper sets τ = 0.6). The result decomposes the target image into a set of spatially coherent candidate regions {Gk}, whose spatial structure is especially critical for part and personalized segmentation.
③ Seed-cluster Selection（种子簇选择）③ Seed-cluster Selection
先以 backward correspondence 在去偏空间中筛选候选簇 Ccand（即参考区域中有足够多 patch 将其最近邻落在该簇内的候选集合）；再计算各候选簇原型 p̃kt 与参考区域原型 p̃r 的 cross-image similarity score：skcross = ⟨p̃kt, p̃r⟩，取得分最高的簇作为 seed cluster G*。Candidate clusters Ccand are first filtered with backward correspondence in the debiased space (the set of clusters in which enough patches of the reference region place their nearest neighbours); the cross-image similarity score between each candidate prototype p̃kt and the reference prototype p̃r is then computed: skcross = ⟨p̃kt, p̃r⟩, and the highest-scoring cluster is taken as the seed cluster G*.
④ Cluster Aggregation（簇聚合）④ Cluster Aggregation
以 seed cluster 为锚点，将其余候选簇中语义相近的区域融入最终预测。融合 cross-image similarity（语义对齐）与 self-similarity（与 seed cluster 的内部亲和度）两项得分，阈值 α = 0.2 控制合并力度；最终 mask 经 CRF refinement 提升边界精度。Using the seed cluster as an anchor, semantically close regions among the remaining candidate clusters are merged into the final prediction. Two scores are fused — cross-image similarity (semantic alignment) and self-similarity (internal affinity with the seed cluster) — with a threshold α = 0.2 controlling the merging strength; the final mask is refined by CRF to improve boundary accuracy.
03实验（Experiments）Experiments
在三类 one-shot 分割任务上评估：语义分割（COCO-20i、LVIS-92i、ISIC、SUIM、iSAID、Chest X-Ray）、part 分割（PASCAL-Part、PACO-Part）、个性化分割（PerMIS）。指标均为 mIoU（%，↑）。Evaluated on three families of one-shot segmentation tasks: semantic segmentation (COCO-20i, LVIS-92i, ISIC, SUIM, iSAID, Chest X-Ray), part segmentation (PASCAL-Part, PACO-Part) and personalized segmentation (PerMIS). The metric throughout is mIoU (%, ↑).
INSID3 在九个 benchmark 的平均 mIoU 为 55.1%，相较最强 training-free 基线 GF-SAM 提升 +7.5%，同时参数量仅为其 1/3（304 M vs 945 M），且完全不依赖 mask 级别的预训练监督。INSID3 reaches an average mIoU of 55.1% over the nine benchmarks, +7.5% above GF-SAM, the strongest training-free baseline, while using only 1/3 of its parameters (304 M vs 945 M) and relying on no mask-level pre-training supervision.
Figure 5.INSID3 与 GF-SAM、SegIC 在 one-shot 语义（左）、part（右上）、个性化（右下）分割上的定性对比。SegIC 在域内表现较好，但跨域及 part 粒度泛化能力弱；GF-SAM 依赖 SAM 的强分割先验，但在 part 细粒度任务上受限；INSID3 跨任务、跨域均表现稳健。Qualitative comparison of INSID3 with GF-SAM and SegIC on one-shot semantic (left), part (top right) and personalized (bottom right) segmentation. SegIC performs well in domain but generalizes poorly across domains and at part granularity; GF-SAM relies on the strong segmentation prior of SAM but is limited on fine-grained part tasks; INSID3 remains robust across tasks and domains.
在语义对应任务 SPair-71k 上，INSID3 的 debiasing 策略对 DINOv3 特征的 PCK@Tτ 有稳定提升。实验表明，DINOv3 比 DINOv2 的 positional bias 更明显（特征更强，位置结构也更突出），但经过 debiasing 后 DINOv3 的语义对应质量显著超越 DINOv2。On the semantic correspondence task SPair-71k, the debiasing strategy of INSID3 yields a consistent improvement of PCK@Tτ for DINOv3 features. The experiments show that DINOv3 exhibits a more pronounced positional bias than DINOv2 (stronger features come with a more salient positional structure), yet after debiasing the semantic correspondence quality of DINOv3 clearly surpasses DINOv2.
补充图.Supplementary figure.雷达图形式展示 INSID3 与各基线在九个 benchmark 上的 mIoU 对比，覆盖 PASCAL-Part、LVIS-92i、PerMIS、COCO-20i、PACO-Part、Chest X-Ray、iSAID、ISIC、SUIM。INSID3（绿线）在所有轴上均表现突出。Radar-chart view of the mIoU comparison between INSID3 and the baselines over the nine benchmarks, covering PASCAL-Part, LVIS-92i, PerMIS, COCO-20i, PACO-Part, Chest X-Ray, iSAID, ISIC and SUIM. INSID3 (green line) stands out on every axis.
消融实验（Ablation）Ablation
论文在 COCO-20i 和 PASCAL-Part 上验证了各模块的贡献：去除 positional debiasing 会导致语义对应精度显著下降；去除 clustering 或 aggregation 均会降低 mIoU；debiasing rank s 和聚类粒度 τ 均有最优区间（分别约 s=16、τ=0.6），对超参数变化具有一定鲁棒性。此外，论文对比了多种 debiasing 替代策略（DINOv2 vs DINOv3 中的位置偏差程度、不同投影方式），验证了基于噪声图像 PCA 的方案最为简洁有效。The paper validates the contribution of each module on COCO-20i and PASCAL-Part: removing positional debiasing leads to a marked drop in semantic correspondence accuracy; removing clustering or aggregation both lower mIoU; the debiasing rank s and the clustering granularity τ each have an optimal range (about s=16 and τ=0.6 respectively), with a certain robustness to hyperparameter changes. The paper further compares several alternative debiasing strategies (the degree of positional bias in DINOv2 vs DINOv3, different projection schemes), showing that PCA on noise images is the simplest and most effective option.
04局限性（Limitations）Limitations
注：Note:本文未设置单独的 Limitations 章节。以下条目部分来自 Conclusion 的隐含表述，部分为从方法设计中推断（已标注）。This paper has no dedicated Limitations section. The items below are partly drawn from implicit statements in the Conclusion and partly inferred from the method design (marked accordingly).
依赖 DINOv3 的特征质量（stated: 隐含）Reliance on the feature quality of DINOv3 (stated: implicit)
INSID3 完全依赖单一 DINOv3 backbone 的特征，其性能上限受限于该 backbone 的表征能力。若 DINOv3 特征在某类图像上匹配能力不足（如高度抽象的艺术图像），方法可能失效。INSID3 depends entirely on the features of a single DINOv3 backbone, so its performance ceiling is bounded by the representational power of that backbone. If DINOv3 features match poorly on some class of images (for instance highly abstract artistic images), the method may fail.
推理速度（inferred: 从设计推断）Inference speed (inferred: from the design)
方法在目标图像上做 agglomerative clustering 并通过 CRF refinement 后处理，这两步在大分辨率图像上具有较高计算开销，实时推理可能受限。论文中未提供具体推理速度数据。The method runs agglomerative clustering on the target image and post-processes it with CRF refinement; both steps are computationally expensive on high-resolution images, so real-time inference may be limited. The paper reports no concrete inference speed figures.
固定超参数的泛化性（inferred: 从实验设计推断）Generalization of the fixed hyperparameters (inferred: from the experimental setup)
聚类粒度 τ 和 aggregation 阈值 α 在所有 benchmark 上均使用相同设置（τ=0.6、α=0.2），尽管消融实验表明对超参数有一定鲁棒性，但极端场景（如医学图像 vs 遥感图像）可能仍需调整。The clustering granularity τ and the aggregation threshold α use the same setting on all benchmarks (τ=0.6, α=0.2); although the ablation shows a certain robustness to hyperparameters, extreme scenarios (such as medical images vs remote sensing images) may still require tuning.
仅限单 backbone、无组合灵活性（inferred: 从方法简洁性推断）Single backbone only, no compositional flexibility (inferred: from the simplicity of the method)
极简主义设计的代价是在依赖 SAM mask 质量的场景（如细密纹理分割）中，相较 GF-SAM 等利用 SAM 强分割先验的方法仍有差距（见 Table 1 部分 benchmark）。The price of the minimalist design is that in scenarios hinging on SAM mask quality (such as fine-texture segmentation) a gap remains with respect to methods exploiting the strong segmentation prior of SAM, such as GF-SAM (see part of the benchmarks in Table 1).
