cs.CV · Allen Institute for AI / University of Washington · 2026
MolmoPoint: Better Pointing for VLMs with Grounding Tokens
用可训练的 grounding token 直接“选中”视觉 token 来指向目标，而不是让模型生成文本坐标Pointing at targets by having trainable grounding tokens directly "select" visual tokens, instead of making the model generate textual coordinates
Christopher Clark, Yue Yang, Jae Sung Park, Zixian Ma, Jieyu Zhang, Rohun Tripathi, Mohammadreza Salehi, Sangho Lee, Taira Anderson, Winson Han, Ranjay Krishna — Allen Institute for AI, University of Washington
大多数 VLM 的 pointing 能力靠在文本输出里生成坐标数字（或离散坐标 bin token）实现，这需要模型学一套复杂的坐标系统，还会拉高输出 token 数。MolmoPoint 让模型生成三个特殊的 grounding token——<PATCH>、<SUBPATCH>、<LOCATION>——逐层从粗到细地在视觉 token 里“选出”目标位置，在图像、GUI、视频指向与目标追踪上都拿到新 SOTA，且训练样本效率明显更高。Most VLMs implement pointing by generating coordinate digits in their text output (or special tokens for discrete coordinate bins), which forces the model to learn a complicated coordinate system and inflates the number of output tokens. MolmoPoint instead has the model generate three special grounding tokens—<PATCH>, <SUBPATCH>, <LOCATION>—which "select" the target location among the visual tokens in a coarse-to-fine, stage-by-stage manner, reaching new SOTA on image, GUI and video pointing as well as object tracking, with markedly better training sample efficiency.
Pointing（指向）是 VLM 的基础能力之一，在机器人抓取/导航规划、GUI agent 操作界面、chain-of-thought 计数等任务里都要用到。现有 VLM 通常用两种方式指向：直接生成文本坐标数字，或生成对应离散坐标 bin 的特殊 token。这两种方式都要求模型额外学一套坐标系统，且每个点要占用多个输出 token，推理延迟高。Pointing is one of the fundamental capabilities of a VLM, needed for robot grasping and navigation planning, GUI agents operating interfaces, chain-of-thought counting and similar tasks. Existing VLMs usually point in one of two ways: directly generating textual coordinate digits, or generating a special token for the corresponding discrete coordinate bin. Both require the model to learn an extra coordinate system, and each point occupies several output tokens, which raises inference latency.
“Instead, we propose a more intuitive pointing mechanism that directly selects the visual tokens that contain the target concept.”
MolmoPoint 的核心想法：模型如果已经在某个视觉 token 的隐藏状态里编码了某个物体/动作/部件，那么直接生成一个能匹配该 token embedding 的 query 向量去“选中”它，比重新学习坐标数字要自然得多——这也更紧密地耦合了视觉识别与指向能力。The core idea of MolmoPoint: if the model has already encoded an object, action or part in the hidden state of some visual token, then directly generating a query vector that matches that token embedding in order to "select" it is far more natural than relearning coordinate digits—and it also couples visual recognition and pointing much more tightly.
Figure 1：MolmoPoint 总览。模型先用 LLM 的隐藏状态给粗粒度图像 patch 打分，再用 ViT 图像特征给最高分 patch 内的细粒度 subpatch 打分，最后在最高分 subpatch 内选出一个点。Figure 1: Overview of MolmoPoint. The model first scores coarse-grained image patches with the LLM hidden states, then scores the fine-grained subpatches inside the top-scoring patch with ViT image features, and finally selects a point inside the top-scoring subpatch.
70.7%PointBench 图像指向新 SOTANew SOTA on PointBench image pointing
61.1%ScreenSpot-Pro，fully open 组 GUI 指向新 SOTAScreenSpot-Pro, new SOTA for GUI pointing in the fully open group
59.1%视频指向人类偏好胜率（vs 文本坐标基线）Human-preference win rate on video pointing (vs. the text-coordinate baseline)
+6.3Molmo2Track 整体 J&F 提升（追踪）Overall J&F gain on Molmo2Track (tracking)
02 Method
要预测一个点，模型依次生成三个特殊 grounding token：<PATCH> 通过对 LLM 视觉 token 隐藏状态做 cross-attention 选出一个粗粒度 patch；<SUBPATCH> 再对该 patch 内更细粒度的 ViT patch 特征打分，选出一个 subpatch；<LOCATION> 最后在该 subpatch 内定位一个具体点。三个 token 被当作输入时都会带上所选位置的 embedding，让模型“记住”自己已经指向了哪里。To predict a point, the model generates three special grounding tokens in turn: <PATCH> selects a coarse-grained patch by cross-attending over the hidden states of the LLM visual tokens; <SUBPATCH> then scores the finer-grained ViT patch features inside that patch and selects a subpatch; <LOCATION> finally localizes a concrete point inside that subpatch. When fed back as inputs, all three tokens carry the embedding of the selected location, letting the model "remember" where it has already pointed.
Figure 2：Grounding token 打分机制。Key 由图像 token 与 ViT patch 特征构建，Query 由 <PATCH> 与 <SUBPATCH> token 的隐藏状态构建，用于对 patch/subpatch 打分；<LOCATION> token 在最高分 subpatch 内预测最终输出点。Figure 2: The grounding-token scoring mechanism. Keys are built from image tokens and ViT patch features, queries from the hidden states of the <PATCH> and <SUBPATCH> tokens, and are used to score patches/subpatches; the <LOCATION> token predicts the final output point inside the top-scoring subpatch.
Patch Selection：把指向变成一次检索Patch Selection: turning pointing into a retrieval step
模型为 <PATCH> token 的隐藏状态构建 query 向量，为每个 <IMAGE> token 构建 key 向量，做点积打分并 softmax，训练时用交叉熵对齐真实目标 token，推理时直接 argmax 选分最高的图像 token。选中的 <IMAGE> token 的输入 embedding 会被加到后续 <PATCH> 输入表示上，让模型“知道”自己刚指向了哪个 token。训练时会把 ground-truth 点按其在输入序列中出现的顺序排序，并 mask 掉排在已选 token 之前的 <IMAGE> token，强制模型按一致的顺序依次生成点，避免重复指向同一处。The model builds a query vector from the hidden state of the <PATCH> token and a key vector for every <IMAGE> token, scores them by dot product and applies a softmax; at training time a cross-entropy loss aligns the scores with the true target token, and at inference the highest-scoring image token is taken directly by argmax. The input embedding of the selected <IMAGE> token is added to the subsequent <PATCH> input representation, so the model "knows" which token it has just pointed at. During training the ground-truth points are sorted by the order in which they appear in the input sequence, and the <IMAGE> tokens preceding the already-selected token are masked out, forcing the model to emit points in a consistent order and avoiding repeated pointing at the same place.
Location Refinement：两级细化到亚像素Location Refinement: two levels of refinement down to sub-pixel
Molmo2 里每个 <IMAGE> token 由 4 个 ViT patch 池化而来，对应 28×28 像素区域，粒度太粗。<SUBPATCH> token 用同样的打分方式，在未池化的 ViT patch 特征上选出更细的 14×14 分辨率 subpatch；最后 <LOCATION> token 用一个线性层，在该 subpatch 内的 3×3 网格中预测 9 个位置之一，最终精度约为 4.7 像素。In Molmo2 each <IMAGE> token is pooled from 4 ViT patches and corresponds to a 28×28 pixel region, which is far too coarse. The <SUBPATCH> token uses the same scoring scheme to select a finer subpatch at 14×14 resolution from the unpooled ViT patch features; finally the <LOCATION> token uses a linear layer to predict one of 9 positions in a 3×3 grid inside that subpatch, for a final precision of about 4.7 pixels.
Rotary Embedding + No-More-Points：让序列生成更稳Rotary Embedding + No-More-Points: making sequential generation more stable
选 patch 时对候选 patch 相对于上一个 <PATCH> 所选位置的距离施加 RoPE 编码，帮助模型感知已经指过哪里，从而生成一致、有序、不重复的点序列。此外允许 <PATCH> 输出一个专门的 “no-more-points” 类别，用于告诉模型该停止指向了——论文指出这能防止模型在视频场景里生成过多冗余点的退化行为。When selecting a patch, RoPE encoding is applied to the distance of each candidate patch relative to the location chosen by the previous <PATCH>, helping the model sense where it has already pointed and thus produce a consistent, ordered, non-repeating sequence of points. In addition, <PATCH> is allowed to output a dedicated "no-more-points" class that tells the model to stop pointing—the paper notes that this prevents the degenerate behaviour of generating too many redundant points in video settings.
03 Experiments
评测覆盖四条主线：自然图像指向（PointBench、PixMo-Points）、GUI grounding（ScreenSpot-V2/Pro、OSWorldG）、视频指向与计数（BURST-VideoCount、Molmo2-VC/VP，外加人类偏好评测）、以及目标追踪（Molmo2Track、MeViS、Ref-YT-VOS、Ref-DAVIS、ReasonVOS）。关键对照组是 Molmo2——它和 MolmoPoint 用完全相同的数据和训练流程，唯一区别是指向机制，这样能干净地隔离出 grounding token 的贡献。The evaluation covers four main threads: natural-image pointing (PointBench, PixMo-Points), GUI grounding (ScreenSpot-V2/Pro, OSWorldG), video pointing and counting (BURST-VideoCount, Molmo2-VC/VP, plus a human-preference evaluation), and object tracking (Molmo2Track, MeViS, Ref-YT-VOS, Ref-DAVIS, ReasonVOS). The key control is Molmo2—it uses exactly the same data and training pipeline as MolmoPoint and differs only in the pointing mechanism, which cleanly isolates the contribution of the grounding tokens.
论文强调 MolmoPoint-8B 在 PointBench 上比 Molmo2 高近 2 分，其中 reasoning 与 spatial reasoning 子项各涨 5 分；在 PixMo-Points 上甚至超过 GPT-5.2、Gemini-3 等专有模型（PixMo-Points F1：GPT-5.2 31.6、Gemini3-Pro 77.8，MolmoPoint-8B 89.2）。视频指向方面，自动指标有涨有跌（Molmo2-VP 上 MolmoPoint 略有下降），因此作者额外做了 470 条视频指向查询的人类偏好评测：排除 152 个平局后，MolmoPoint-8B 拿到 188 票、Molmo2 拿到 130 票，即 59.1% 的胜率。追踪方面，去掉 grounding token 会让整体 F1 掉 4.6、HOTA 掉 4.0（Table 7），进一步去掉新引入的 MolmoPoint-Track 数据在 Misc 域还会再掉 7.6 F1。The paper stresses that MolmoPoint-8B scores nearly 2 points higher than Molmo2 on PointBench, with the reasoning and spatial-reasoning subsets each gaining 5 points; on PixMo-Points it even surpasses proprietary models such as GPT-5.2 and Gemini-3 (PixMo-Points F1: GPT-5.2 31.6, Gemini3-Pro 77.8, MolmoPoint-8B 89.2). For video pointing the automatic metrics go both ways (MolmoPoint drops slightly on Molmo2-VP), so the authors additionally ran a human-preference evaluation over 470 video pointing queries: after excluding 152 ties, MolmoPoint-8B received 188 votes and Molmo2 received 130, i.e. a 59.1% win rate. For tracking, removing the grounding tokens costs 4.6 overall F1 and 4.0 HOTA (Table 7), and further removing the newly introduced MolmoPoint-Track data costs another 7.6 F1 in the Misc domain.
Figure 5：样本效率。左：仅用极少量指向训练样本微调时的表现；右：全量预训练过程中的指向表现曲线。Figure 5: Sample efficiency. Left: performance when fine-tuning with only a very small number of pointing training examples. Right: pointing performance curves over the course of full pre-training.
“MolmoPoint-8B initially performs worse, likely due to needing to learn new parameters from scratch, but quickly improves to a 20 point gain when using 8192 examples”（完整指向数据集约有近 50 万样本）；全量预训练时 MolmoPoint 也更快达到峰值表现。两者共同说明：grounding token 比文本坐标更容易学、样本效率更高。"MolmoPoint-8B initially performs worse, likely due to needing to learn new parameters from scratch, but quickly improves to a 20 point gain when using 8192 examples" (the full pointing dataset contains close to half a million examples); in full pre-training MolmoPoint also reaches peak performance sooner. Together these show that grounding tokens are easier to learn and more sample-efficient than text coordinates.
Figure 6：定性对比。上：Molmo2 在多个视频帧里生成了一整排错误的点。中：Molmo2 无法精确定位“x”标记（放大图为同一点的局部特写）。左下：MolmoPoint-8B 找到了第二个、被部分遮挡的马鞍。右下：MolmoPoint-8B 漏掉了其中一个盘子。Figure 6: Qualitative comparison. Top: Molmo2 generates a whole row of wrong points across several video frames. Middle: Molmo2 cannot precisely localize the "x" marker (the zoomed inset is a close-up of the same point). Bottom left: MolmoPoint-8B finds the second, partially occluded saddle. Bottom right: MolmoPoint-8B misses one of the plates.
Ablations（Table 9，轻量训练管线上的消融）Ablations (Table 9, ablations on a lightweight training pipeline)
去掉 rotary embedding：图像上小幅下降，视频上影响更明显；去掉 no-more-points 类别：表现下降，且视频“过量计数”（overcount，即预测点数 >10 且 ≥ 真值 2 倍）的比例翻倍以上；把点的生成顺序随机打乱：视频上明显掉分，但在 PointBench 上反而略有提升——作者猜测这可能是让模型按“最容易的点先来”的顺序生成带来的收益，但要不破坏视频表现还需要更多工作。Removing the rotary embedding: a small drop on images, a more pronounced effect on video. Removing the no-more-points class: performance drops, and the rate of video "overcounting" (more than 10 predicted points and at least two times the ground truth) more than doubles. Randomly shuffling the generation order of the points: a clear drop on video, but a slight improvement on PointBench instead—the authors speculate this may come from letting the model generate in an "easiest points first" order, though more work is needed to obtain it without hurting video performance.
04 Limitations
Note:论文正文没有单独设立 “Limitations” 小节；第 1 条是作者在 Qualitative Results / Conclusion 中明确提到的观察或未来方向，其余为根据方法设计推断（已标注）。The body of the paper has no standalone "Limitations" section; item 1 is an observation or future direction explicitly mentioned by the authors in the Qualitative Results / Conclusion, while the rest are inferred from the method design (marked as such).
高频物体计数仍有 off-by-one 误差（stated）Off-by-one errors persist when counting high-frequency objects (stated)
“we observe that it occasionally produces off-by-one errors when counting high-frequency objects.” 说明即便定位机制更精确，模型在密集/重复物体场景下的计数仍不完全可靠。"we observe that it occasionally produces off-by-one errors when counting high-frequency objects." This shows that even with a more precise localization mechanism, the model's counting in dense or repetitive object scenes is still not fully reliable.
目前只作用于视觉 token，尚未扩展到其它模态（stated as future work）Currently applies only to visual tokens, not yet extended to other modalities (stated as future work)
Conclusion 中写道：“Future work could extend this approach to include other modalities, such as pointing to text tokens to highlight important parts of the text or pointing to audio tokens to reference a sound.” 说明当前 grounding token 机制被限定在图像/视频视觉 token 上，指向文本或音频 token 尚未实现。The Conclusion states: "Future work could extend this approach to include other modalities, such as pointing to text tokens to highlight important parts of the text or pointing to audio tokens to reference a sound." So the current grounding-token mechanism is confined to image/video visual tokens; pointing to text or audio tokens has not been realized.
定位精度受 ViT patch/subpatch 离散粒度上限约束（inferred）Localization precision is bounded by the discrete granularity of ViT patches/subpatches (inferred)
<LOCATION> token 只在 3×3 网格里选 9 个位置之一，对应约 4.7 像素的精度上限——这是一种比文本坐标更粗的离散化方式，理论上无法达到任意亚像素精度，只是论文没有把这一点明确列为局限。The <LOCATION> token only picks one of 9 positions in a 3×3 grid, corresponding to a precision ceiling of about 4.7 pixels—a coarser discretization than text coordinates, which in theory cannot reach arbitrary sub-pixel precision; the paper simply does not list this explicitly as a limitation.
严格的顺序生成约束在某些场景下可能不是最优（inferred）The strict sequential-generation constraint may not be optimal in some settings (inferred)
消融显示打乱点的生成顺序会明显拉低视频表现，但反而小幅提升 PointBench，说明当前“按输入序列顺序强制排序 + mask”这一设计并非对所有任务都是最佳选择，是一种为换取视频稳定性而做的权衡。The ablations show that shuffling the generation order of the points clearly lowers video performance yet slightly improves PointBench, indicating that the current design of "forced ordering by input-sequence position + masking" is not the best choice for every task, but a trade-off made to buy video stability.
