用概念提示检测、分割并追踪图像与视频中的一切目标Detecting, segmenting and tracking everything in images and videos with concept prompts
Anonymous Authors（双盲评审） · Meta FAIR · ICLR 2026 Under ReviewAnonymous Authors (double-blind review) · Meta FAIR · ICLR 2026 Under Review
SAM 3 是对 SAM 2 的重大扩展，引入了 Promptable Concept Segmentation (PCS) 任务：给定一个短语名词短语（如 "yellow school bus"）、图像范例（image exemplars），或二者的组合，模型自动检测、分割并追踪图像/视频中所有匹配该概念的实例，同时保留跨帧的目标身份。SAM 3 在 PCS 任务上比现有系统提升约 2×，并在交互式视觉分割（PVS）基准上持续优于 SAM 2。SAM 3 is a major extension of SAM 2 that introduces the Promptable Concept Segmentation (PCS) task: given a short noun phrase (e.g. "yellow school bus"), image exemplars, or a combination of both, the model automatically detects, segments and tracks every instance matching that concept in an image or video, while preserving object identity across frames. SAM 3 improves over existing systems by about 2× on PCS, and consistently outperforms SAM 2 on the interactive visual segmentation (PVS) benchmark.
任务：开放词汇实例分割 + 视频追踪Task: open-vocabulary instance segmentation + video tracking数据：SA-Co/HQ 含 4M 独特概念标签Data: SA-Co/HQ with 4M unique concept labels推理延迟：单张图像 30 ms（100+ 检测目标，H200）Inference latency: 30 ms per image (100+ detected objects, H200)📄 arXiv:2511.16719OpenReview 页面page
SAM 1 和 SAM 2 开创了交互式分割先河，但它们依赖点、框、掩码等几何提示，每次只能分割单个目标实例，无法回答"视频中所有的猫在哪里"这类概念级查询。现实应用（机器人、内容创作、AR、数据标注、科学研究）迫切需要一个能够理解视觉概念、并一次性找出所有匹配实例的模型。SAM 1 and SAM 2 pioneered interactive segmentation, but they rely on geometric prompts such as points, boxes and masks and can segment only a single object instance at a time, so they cannot answer concept-level queries such as "where are all the cats in this video". Real applications (robotics, content creation, AR, data annotation, scientific research) urgently need a model that understands visual concepts and finds all matching instances at once.
"SAM 1 and SAM 2 focus on visual prompts and segment a single object instance per prompt … they did not address the broader task of finding and segmenting all instances of a concept appearing anywhere in the input."
图 1：SAM 3 在可提示视觉分割（左，鼠标点击）和可提示概念分割（右，短语 / 图像范例 / 组合）两项任务上均超越 SAM 2。右侧示例展示模型同时找出视频中所有匹配给定视觉概念的目标实例。Figure 1:SAM 3 surpasses SAM 2 on both promptable visual segmentation (left, mouse clicks) and promptable concept segmentation (right, phrase / image exemplar / combination). The example on the right shows the model simultaneously finding every object instance in the video that matches the given visual concept.
2×图像 & 视频 PCS 增益（相比现有最优系统）Image & video PCS gain (over the previous best systems)
47.0LVIS zero-shot mask AP（前最优 38.5）LVIS zero-shot mask AP (previous best 38.5)
4MSA-Co/HQ 独特概念标签数量Number of unique concept labels in SA-Co/HQ
SAM 3 由共享视觉骨干网络 Perception Encoder (PE) 的 检测器（图像级）与 追踪器（内存式视频）构成。检测器基于 DETR 范式，输入文本/图像范例提示后预测所有匹配实例的分割掩码。追踪器继承 SAM 2 的 Transformer 编解码器架构，在视频中传播 masklet（时空掩码）。两者通过四阶段渐进式训练策略联合优化。SAM 3 consists of a detector (image level) and a tracker (memory-based video) that share a Perception Encoder (PE) visual backbone. The detector follows the DETR paradigm and, given text or image-exemplar prompts, predicts segmentation masks for all matching instances. The tracker inherits the Transformer encoder-decoder architecture of SAM 2 and propagates masklets (spatio-temporal masks) through the video. The two are optimised jointly with a four-stage progressive training strategy.
图 3：SAM 3 架构总览。PE 骨干网络（青色）同时服务检测器与追踪器。检测器（新增组件，黄色高亮）包含 Fusion Encoder、DETR 解码器及 Presence Token；追踪器（蓝色，继承自 SAM 2）维护内存库以在视频帧间传播 masklet。Figure 3:Overview of the SAM 3 architecture. The PE backbone (cyan) serves the detector and the tracker at the same time. The detector (new components, highlighted in yellow) contains a Fusion Encoder, a DETR decoder and the Presence Token; the tracker (blue, inherited from SAM 2) maintains a memory bank in order to propagate masklets across video frames.
Presence Token — 解耦识别与定位Presence Token — decoupling recognition from localisation
传统 DETR 中每个 proposal query 既要识别（what）又要定位（where），两者目标冲突：识别需要全局上下文，而定位本质上是局部的。SAM 3 引入一个可学习的全局 presence token，专门预测目标概念是否出现在图像/帧中（p(NP is present in input)）。每个 proposal query 只需在"概念已出现"的条件下解决定位问题，最终分数 = 自身分数 × presence 分数。消融实验显示该设计将 SA-Co/Gold CGF1 提升 +5.7（从 57.6 到 63.3），image-level MCC 从 0.77 提升至 0.82。In a conventional DETR, every proposal query has to both recognise (what) and localise (where), and the two objectives conflict: recognition needs global context, whereas localisation is intrinsically local. SAM 3 introduces a learnable global presence token dedicated to predicting whether the target concept appears in the image/frame (p(NP is present in input)). Each proposal query then only has to solve localisation conditioned on the concept being present, and the final score = its own score × the presence score. Ablations show that this design raises SA-Co/Gold CGF1 by +5.7 (from 57.6 to 63.3) and image-level MCC from 0.77 to 0.82.
Image Exemplars 与交互式精化Image exemplars and interactive refinement
SAM 3 支持图像范例提示（正框 / 负框），可单独使用或与文本提示组合。每个范例由位置嵌入、标签嵌入及 ROI 池化视觉特征拼接后经小型 Transformer 编码，与文本 token 拼接共同构成 prompt tokens。交互式地加入范例后，模型能泛化地检测/抑制相似目标（而不仅修正单个实例），3 次点击后比文字提示提升 +18.6 CGF1，比理想 PVS 修正提升 +9.7。SAM 3 supports image-exemplar prompts (positive / negative boxes), used on their own or combined with a text prompt. Each exemplar is encoded by a small Transformer over the concatenation of its position embedding, label embedding and ROI-pooled visual features, and is concatenated with the text tokens to form the prompt tokens. Once exemplars are added interactively, the model detects/suppresses similar objects in a generalising way (rather than merely correcting a single instance): after three clicks it improves over the text prompt by +18.6 CGF1, and over ideal PVS correction by +9.7.
视频追踪架构Video tracking architecture
给定视频与提示 P，检测器在每帧上发现新目标 Ot，追踪器将前一时刻的 masklet Mt-1 传播到当前帧得到 M̂t，随后通过基于 IoU 的匹配函数将二者关联并更新。对于遮挡/干扰物等追踪失败场景，SAM 3 引入两项时序消歧策略：masklet detection score（时序窗口内持续匹配得分）和定期用高置信度检测结果重新初始化追踪器内存库。Given a video and a prompt P, the detector discovers new objects Ot in every frame, while the tracker propagates the masklet Mt-1 of the previous time step into the current frame to obtain M̂t; the two are then associated and updated through an IoU-based matching function. For tracking failures caused by occlusion or distractors, SAM 3 introduces two temporal disambiguation strategies: a masklet detection score (a sustained matching score within a temporal window) and periodic re-initialisation of the tracker memory bank with high-confidence detections.
图 4：SA-Co 数据引擎四阶段总览。从媒体输入、NP 生成、候选掩码提议，到掩码验证（MV）与穷举验证（EV），人类与 AI 标注员协同工作。AI 验证器（基于 Llama 3.2 微调）的引入使吞吐量翻倍，最终积累 5.2M 图像、4M 独特 NP 的高质量训练集。Figure 4:Overview of the four phases of the SA-Co data engine. From media input, NP generation and candidate mask proposal through to mask verification (MV) and exhaustive verification (EV), human and AI annotators work together. Introducing the AI verifier (fine-tuned from Llama 3.2) doubles throughput, ultimately accumulating a high-quality training set of 5.2M images and 4M unique NPs.
数据引擎（SA-Co Data Engine）Data engine (SA-Co Data Engine)
数据引擎分四阶段迭代：The data engine iterates over four phases:
Phase 1（纯人工验证）：随机采样图像，SAM 2 + 开放词汇检测器生成候选掩码，全人工掩码验证（MV）与穷举验证（EV），收集 4.3M 图像-NP 对构成初始 SA-Co/HQ。Phase 1 (human-only verification):images are sampled at random, SAM 2 plus an open-vocabulary detector generates candidate masks, mask verification (MV) and exhaustive verification (EV) are carried out entirely by humans, collecting 4.3M image-NP pairs that constitute the initial SA-Co/HQ.
Phase 2（人工 + AI 验证）：用 Phase 1 标注微调 Llama 3.2，创建 AI 验证器，吞吐量翻倍；引入 Llama 生成对抗性硬负例 NP；累计新增 122M 图像-NP 对。Phase 2 (human + AI verification):Llama 3.2 is fine-tuned on the Phase 1 annotations to create an AI verifier, doubling throughput; Llama-generated adversarial hard-negative NPs are introduced; 122M further image-NP pairs are accumulated.
Phase 3（规模扩展与领域扩张）：扩展至 15 个数据集领域；利用 22.4M 节点的 SA-Co 本体（基于 Wikidata）挖掘长尾细粒度概念；新增 19.5M 图像-NP 对。Phase 3 (scaling up and domain expansion):extended to 15 dataset domains; the SA-Co ontology of 22.4M nodes (built on Wikidata) is used to mine long-tail fine-grained concepts; 19.5M new image-NP pairs are added.
Phase 4（视频标注）：扩展至视频，结合场景/运动过滤、目标密度采样，收集 52.5K 视频、467K masklet（SA-Co/VIDEO）。Phase 4 (video annotation):extended to video, combining scene/motion filtering with object-density sampling to collect 52.5K videos and 467K masklets (SA-Co/VIDEO).
最终训练集包括：SA-Co/HQ（5.2M 图像，4M NPs）、SA-Co/SYN（合成数据，38M NPs，1.4B 掩码）、SA-Co/EXT（15 个外部数据集，增强硬负例）和 SA-Co/VIDEO。The final training set comprises SA-Co/HQ (5.2M images, 4M NPs), SA-Co/SYN (synthetic data, 38M NPs, 1.4B masks), SA-Co/EXT (15 external datasets, with enhanced hard negatives) and SA-Co/VIDEO.
03实验Experiments
SAM 3 在图像/视频 PCS（开放词汇实例分割与追踪）、少样本适应（目标检测/计数）以及与 MLLM 结合的复杂语言查询分割等任务上进行全面评估。评测基准涵盖 LVIS、COCO、SA-Co/Gold/Silver/Bronze/Bio、SA-Co/VEval、ODinW13、RF-100VL、ReasonSeg、OmniLabel 等。SAM 3 is evaluated comprehensively on image/video PCS (open-vocabulary instance segmentation and tracking), few-shot adaptation (object detection/counting) and segmentation from complex language queries in combination with an MLLM. The benchmarks cover LVIS, COCO, SA-Co/Gold/Silver/Bronze/Bio, SA-Co/VEval, ODinW13, RF-100VL, ReasonSeg, OmniLabel and others.
注：SA-Co/Gold CGF1 = 65.0 达到人类下限估计（74.2）的 88%。Gemini 2.5（强通用 LLM 基线）CGF1 为 19.8，SAM 3 超出其约 3.3×。Note: SA-Co/Gold CGF1 = 65.0 reaches 88% of the estimated human lower bound (74.2). Gemini 2.5, a strong general-purpose LLM baseline, reaches a CGF1 of 19.8, which SAM 3 exceeds by about 3.3×.
SA-Co/VEval 上 SAM 3 达到人类 pHOTA 下限的 >80%。GLEE 基线（未在 SA-Co 上训练）CGF1 接近 0，突出了大规模多概念视频分割的难度。On SA-Co/VEval, SAM 3 reaches >80% of the human pHOTA lower bound. The CGF1 of the GLEE baseline (not trained on SA-Co) is close to 0, which highlights how difficult large-scale multi-concept video segmentation is.
图（Table 1 摘录页）：图像概念分割主要评测结果汇总，含 LVIS 实例分割、SA-Co 各分割、COCO/COCO-O 框检测、ADE-847/PC-59/Cityscapes 语义分割多项指标，与 OWLv2、GroundingDINO、LLMDet、APE、DINO-X、Gemini 2.5 等基线对比。Figure (excerpt page of Table 1):Summary of the main evaluation results for image concept segmentation, covering LVIS instance segmentation, the SA-Co splits, COCO/COCO-O box detection and ADE-847/PC-59/Cityscapes semantic segmentation, compared with baselines such as OWLv2, GroundingDINO, LLMDet, APE, DINO-X and Gemini 2.5.
模拟人机协作：从文本提示出发，每轮迭代添加一个正/负范例框。实验显示：Simulating human-model collaboration: starting from a text prompt, one positive/negative exemplar box is added per iteration. The experiments show that:
3 次点击后，SAM 3 PCS 相比仅用文本提示提升 +18.6 CGF1；after 3 clicks, SAM 3 PCS improves by +18.6 CGF1 over the text prompt alone;
与"理想 PVS 修正"（逐实例手动修正）相比提升 +9.7；compared with "ideal PVS correction" (manual instance-by-instance correction) the gain is +9.7;
4 次点击后性能趋于平稳（范例无法修复掩码质量问题），此时切换至 PVS 模式可获得进一步提升，两种方式互补。after 4 clicks performance plateaus (exemplars cannot repair mask-quality problems); switching to PVS mode at that point yields further gains, so the two modes are complementary.
目标计数Object counting
模型Model
CountBench Acc↑
PixMo-Count Acc↑
DINO-X
82.9
85.0
Gemini 2.5 Pro
92.4
78.2
Molmo-72B
92.4
88.8
SAM 3
95.6
87.3
SAM 3 Agent（与 MLLM 结合）SAM 3 Agent (combined with an MLLM)
SAM 3 可与多种 MLLM（Qwen2.5-VL 7B/72B、Llama4 Maverick、Gemini 2.5 Pro）结合，MLLM 将复杂语言查询分解为 NP 调用 SAM 3，迭代优化输出。零样本情况下在 ReasonSeg 和 OmniLabel 上分别达到 73.8 gIoU 和 46.7 AP，超越所有专用方法（包括 GPT-4o + SegZero 等）。SAM 3 can be combined with a variety of MLLMs (Qwen2.5-VL 7B/72B, Llama4 Maverick, Gemini 2.5 Pro): the MLLM decomposes a complex language query into NPs, calls SAM 3 and iteratively refines the output. Zero-shot, it reaches 73.8 gIoU on ReasonSeg and 46.7 AP on OmniLabel, surpassing all specialised methods (including GPT-4o + SegZero).
关键消融Key ablations
消融项Ablation
设置Setting
CGF1
IL MCC
pmF1
Presence Token
无without
57.6
0.77
74.7
有with
63.3
0.82
77.1
训练数据Training data
仅 EXTEXT only
30.9
0.46
66.3
EXT + SYN
39.7
0.57
70.6
EXT + SYN + HQ
54.3
0.74
73.5
硬负例（每图 #）Hard negatives (# per image)
0 个0
31.8
0.44
70.2
5 个5
44.8
0.62
71.9
30 个30
49.2
0.68
72.3
消融验证：Presence Token（+5.7 CGF1）、硬负例训练（IL MCC 从 0.44 大幅提升至 0.68）、高质量人工数据 HQ（比纯合成数据 SYN 再提升 +14.6 CGF1）均是关键贡献。The ablations confirm the key contributions: the Presence Token (+5.7 CGF1), hard-negative training (IL MCC rising sharply from 0.44 to 0.68) and high-quality human data HQ (a further +14.6 CGF1 over purely synthetic SYN data).
04局限性Limitations
说明：以下局限性均来自论文附录 C"Limitations"章节，为作者明确陈述（stated）。Note:all limitations below are taken from Appendix C "Limitations" of the paper and are explicitly stated by the authors.
细粒度领域外概念泛化能力不足Insufficient generalisation to fine-grained out-of-domain concepts
"SAM 3 struggles to generalize to fine-grained out-of-domain concepts (e.g., aircraft types, medical terms) in a zero-shot manner, especially in niche visual domains (e.g., thermal imagery)." 虽然少量微调可快速适应新概念，但零样本表现有限。"SAM 3 struggles to generalize to fine-grained out-of-domain concepts (e.g., aircraft types, medical terms) in a zero-shot manner, especially in niche visual domains (e.g., thermal imagery)." A small amount of fine-tuning can adapt the model to new concepts quickly, but zero-shot performance remains limited.
文本提示限于简单名词短语（noun phrases）Text prompts are restricted to simple noun phrases
"SAM 3 is constrained to simple noun phrase prompts and does not support multi-attribute queries beyond one or two attributes or longer phrases including referring expressions." 即不直接支持复杂指代表达。与 MLLM 结合可缓解此问题，但需要外部大模型参与。"SAM 3 is constrained to simple noun phrase prompts and does not support multi-attribute queries beyond one or two attributes or longer phrases including referring expressions." In other words, complex referring expressions are not supported directly. Combining SAM 3 with an MLLM alleviates the problem, but requires an external large model.
视频推理成本随追踪目标数线性增长Video inference cost grows linearly with the number of tracked objects
"The cost of SAM 3 inference scales linearly with the number of objects being tracked." 实时 30 FPS 需要多卡并行：2×H200 支持约 10 个目标，4×H200 支持约 28 个，8×H200 支持约 64 个。"There is no shared object-level contextual information to aid in resolving ambiguities in multi-object tracking scenarios.""The cost of SAM 3 inference scales linearly with the number of objects being tracked." Real-time 30 FPS requires multi-GPU parallelism: 2×H200 supports about 10 objects, 4×H200 about 28, and 8×H200 about 64. "There is no shared object-level contextual information to aid in resolving ambiguities in multi-object tracking scenarios."
概念级与实例级交互模式切换不连续Switching between concept-level and instance-level interaction modes is discontinuous
"Supporting concept-level interactivity for PCS, alongside instance-level interactivity for PVS, poses several challenges. To support instance-level modifications without affecting all other instances of the concept, we enforce a hard 'mode-switch' within the model from concept to instance mode." 作者指出，未来工作可以更无缝地交织概念提示与实例提示。"Supporting concept-level interactivity for PCS, alongside instance-level interactivity for PVS, poses several challenges. To support instance-level modifications without affecting all other instances of the concept, we enforce a hard 'mode-switch' within the model from concept to instance mode." The authors point out that future work could interleave concept prompts and instance prompts more seamlessly.
