OpenFrontier: General Navigation with Visual-Language Grounded Frontiers
用视觉 frontier 作为语义锚点，把 VLM 的语言先验直接接到图像空间的可行走目标上Using visual frontiers as semantic anchors, wiring the language priors of VLMs directly onto traversable targets in image space
Esteban Padilla-Cerdio*, Boyang Sun* (等贡献equal contribution), Marc Pollefeys, Hermann Blum · ETH Zürich, Microsoft Spatial AI Lab, University of Bonn
OpenFrontier 把导航建模为“稀疏 subgoal 识别 + 到达”问题：直接在 2D 图像上检测 visual frontier，用 set-of-marks 方式一次性喂给 VLM 打相关性分数，再把结果 grounding 回 3D 空间驱动导航——全程 zero-shot，不需要密集 3D 语义建图、任务专属策略训练，也不需要对 VLM 做微调。OpenFrontier formulates navigation as a "sparse subgoal identification + reaching" problem: it detects visual frontiers directly on 2D images, feeds them to a VLM in one shot via set-of-marks prompting to obtain relevance scores, and then grounds the results back into 3D space to drive navigation — entirely zero-shot, requiring neither dense 3D semantic mapping, nor task-specific policy training, nor fine-tuning of the VLM.
ETH Zürich · University of BonnHabitat ObjectNav: HM3D / MP3D / OVONReal-world: Boston Dynamics Spot📄 arXiv:2603.05377Project Page
传统 object-goal navigation 依赖密集 3D 重建 + 手工设计的 goal metric，泛化性差；近期基于 LLM/VLM 的 vision-language navigation (VLN) 和 vision-language-action (VLA) 方法虽然能用自然语言条件化端到端策略，但"typically require interactive training, large-scale data collection, or task-specific fine-tuning with a mobile agent"。作者认为，比起更强的语义先验、更大的数据集，真正稀缺的是一个能把语义推理接到物理导航决策上的"有效接口"。Classical object-goal navigation relies on dense 3D reconstruction plus hand-designed goal metrics and generalizes poorly; recent LLM/VLM-based vision-language navigation (VLN) and vision-language-action (VLA) methods can condition end-to-end policies on natural language, but they "typically require interactive training, large-scale data collection, or task-specific fine-tuning with a mobile agent". The authors argue that what is truly scarce is not stronger semantic priors or larger datasets, but an effective interface that connects semantic reasoning to physical navigation decisions.
“We formulate navigation as a sparse subgoal identification and reaching problem and observe that providing visual anchoring targets for high-level semantic priors enables highly efficient goal-conditioned navigation.”
基于这一观察，作者选择经典探索机器人学中的 navigation frontier 作为语义锚点——frontier 天然对应可行走的、有物理意义的候选区域，稀疏、可解释，且不需要密集建图即可获得。OpenFrontier 因此做到：不需要 task-specific training、不需要 fine-tuning，且对使用哪个 VLM 做推理保持无关（agnostic）。Building on this observation, the authors adopt the navigation frontier from classical exploration robotics as the semantic anchor — frontiers naturally correspond to traversable, physically meaningful candidate regions, they are sparse and interpretable, and they can be obtained without dense mapping. OpenFrontier therefore needs no task-specific training and no fine-tuning, and stays agnostic to which VLM is used for inference.
Top：OpenFrontier 直接在图像空间检测并语义评估 visual frontier，无需密集 3D 重建即可实现语言条件的目标推理；箭头颜色编码目标相关度，红色（高）到黄色（低）。Bottom：单帧观测中的 visual frontier 被逐步 grounding 到 3D metric space，驱动跨多样目标的长时程、自然语言条件导航，全程 zero-shot，无需任务专属微调。（Fig. 1，原文 caption 节选翻译）Top: OpenFrontier detects and semantically evaluates visual frontiers directly in image space, enabling language-conditioned goal reasoning without dense 3D reconstruction; arrow color encodes target relevance, from red (high) to yellow (low). Bottom: visual frontiers in single-frame observations are progressively grounded into 3D metric space, driving long-horizon, natural-language-conditioned navigation across diverse targets, entirely zero-shot and without task-specific fine-tuning. (Fig. 1, translated excerpt of the original caption)
77.3%HM3D ObjectNav Val · SR，Table I 中最高HM3D ObjectNav Val · SR, highest in Table I
35.6%HM3D ObjectNav Val · SPL
20.1%OVON Val Unseen · SPL，Table I 中最高OVON Val Unseen · SPL, highest in Table I
OpenFrontier 的核心原则是：“detect and evaluate navigation targets directly in the 2D image domain whenever possible”。给定带位姿的 RGB 观测 I_t、相机外参 T_t 和内参 K，以及自然语言目标 L，系统先在图像上识别一组 visual frontier F = {X, g}（X 为 3D 位姿，g 为 information gain），再把选中的 frontier grounding 到 3D 空间生成 goal pose，交给底层 motion planner 执行。The core principle of OpenFrontier is to "detect and evaluate navigation targets directly in the 2D image domain whenever possible". Given a posed RGB observation I_t, camera extrinsics T_t and intrinsics K, together with a natural-language goal L, the system first identifies a set of visual frontiers F = {X, g} on the image (X is the 3D pose, g the information gain), then grounds the selected frontier into 3D space to produce a goal pose and hands it to a low-level motion planner for execution.
System Overview（Fig. 2）。给定带位姿的 RGB 观测和自然语言导航目标，OpenFrontier 在图像上检测 visual frontier，直接查询 VLM，利用图内上下文评估每个 frontier 对目标的相关概率；随后这些 frontier 连同更新后的 information gain 被提升到 3D metric space，作为目标条件候选，由全局模块统一管理以更新导航目标、进行路径规划并判断终止。System Overview (Fig. 2). Given a posed RGB observation and a natural-language navigation goal, OpenFrontier detects visual frontiers on the image and queries the VLM directly, using in-image context to assess the probability that each frontier is relevant to the goal; these frontiers, together with their updated information gain, are then lifted into 3D metric space as goal-conditioned candidates and managed by a global module that updates the navigation target, plans paths and decides termination.
Visual-Frontier Identification
受 FrontierNet 启发，OpenFrontier 直接从单张 2D 图像检测并评估 frontier（称为 visual frontier），不依赖密集 3D 建图，也不把检测和评估拆成两步。frontier 检测流程沿用 FrontierNet，对关键帧 RGB 观测做后处理，输出一组 frontier cluster F_i，每个 cluster 先在图像域表示，再反投影到 3D 空间得到位姿，同时给出探索先验 information gain 估计 ĝ_i（预测该 frontier 关联的未知空间体积）。为引入任务相关的语义先验，OpenFrontier 用 set-of-marks 策略把每个 frontier 的 2D 质心标注（marker）叠加到 RGB 图像上，连同语言目标 L 一起喂给 VLM，让模型对每个 marker 输出一个概率分数 p_i ∈ [0,1]，表示朝该 frontier 前进是否有助于达成目标。最终 utility 计算为：Inspired by FrontierNet, OpenFrontier detects and evaluates frontiers (termed visual frontiers) directly from a single 2D image, without relying on dense 3D mapping and without splitting detection and evaluation into two stages. The frontier detection pipeline follows FrontierNet: keyframe RGB observations are post-processed into a set of frontier clusters F_i, each first represented in the image domain and then back-projected into 3D space to obtain a pose, together with an exploration-prior information gain estimate ĝ_i (the predicted volume of unknown space associated with that frontier). To inject task-relevant semantic priors, OpenFrontier uses a set-of-marks strategy that overlays a marker at the 2D centroid of each frontier on the RGB image and feeds it, along with the language goal L, to the VLM, which outputs for every marker a probability score p_i ∈ [0,1] indicating whether moving toward that frontier helps achieve the goal. The final utility is computed as:
g_i = p_i · ĝ_i ， p_i = VLM(I_t, F_i, L)
这种做法让 VLM 只需在 2D 图像空间内做推理，而不必对多视角输入或抽象场景级表示做显式 3D 空间推理——作者指出这更贴近当前 VLM 的训练分布，"where current models are known to be less reliable" 在 3D 空间推理上。This design lets the VLM reason only inside the 2D image space instead of performing explicit 3D spatial reasoning over multi-view inputs or abstract scene-level representations — the authors note that this is closer to the training distribution of current VLMs, whereas 3D spatial reasoning is "where current models are known to be less reliable".
Set-of-Marks Query（Fig. 3）。检测到的 visual frontier 与对应 RGB 图像一起，通过 set-of-marks prompting 联合查询 VLM：每个 frontier 在图像上被标记，VLM 结合局部视觉上下文评估其与给定导航指令的相关性；所得相关概率用于对来自 FrontierNet 的探索驱动 information gain 重新加权，从而把任务相关的语义先验与探索融合在一起。Set-of-Marks Query (Fig. 3). The detected visual frontiers are queried jointly with the corresponding RGB image through set-of-marks prompting: each frontier is marked on the image, and the VLM assesses its relevance to the given navigation instruction from the local visual context; the resulting relevance probabilities reweight the exploration-driven information gain from FrontierNet, fusing task-relevant semantic priors with exploration.
长时程导航需要在全局层面维护和更新目标。OpenFrontier 维护一组活跃 frontier 目标，周期性根据机器人当前状态重新计算每个 frontier 的全局 utility：Long-horizon navigation requires maintaining and updating goals at the global level. OpenFrontier keeps a set of active frontier goals and periodically recomputes the global utility of every frontier from the robot’s current state:
u_i = g_i / ‖p_r − p_i‖
utility 最高的 frontier 被选为下一个导航目标，交给底层规划器（可以是 map-free 的 PointGoal 策略，也可以是基于 3D 体素图的无碰撞规划器，二者切换不改变 frontier management 逻辑）。导航过程中，RGB 观测同时被一个开放词表分割模型（SAM3）处理以生成目标物体的分割掩膜；一旦检测到有效掩膜，就用深度与相机位姿估计目标 3D 质心，并在其附近插入一个具有（近似无穷）高 utility 的 viewpoint frontier，引导机器人靠近以获得直接视觉访问。到达 viewpoint frontier 后，系统再次用同一个 VLM 查询对应 RGB 观测以核验目标是否存在：确认则以该质心为最终目标并尽量靠近；未确认则丢弃该假设，移除 viewpoint frontier。The frontier with the highest utility is selected as the next navigation goal and handed to the low-level planner (either a map-free PointGoal policy or a collision-free planner built on a 3D voxel map; switching between them does not change the frontier management logic). During navigation, RGB observations are simultaneously processed by an open-vocabulary segmentation model (SAM3) to produce a segmentation mask of the target object; once a valid mask is detected, depth and camera pose are used to estimate the 3D centroid of the target, and a viewpoint frontier with (near-infinite) high utility is inserted nearby to guide the robot closer for direct visual access. After reaching the viewpoint frontier, the system queries the same VLM again on the corresponding RGB observation to verify whether the target is present: if confirmed, that centroid becomes the final goal and the robot approaches it as closely as possible; if not, the hypothesis is discarded and the viewpoint frontier is removed.
03 Experiments
在 Habitat 框架内评测三个 object-goal navigation 数据集：HM3D ObjectNav、MP3D ObjectNav（闭集词表）与 OVON（开放词表，语言描述更多样），指标为 Success Rate (SR, %) 与 Success weighted by Path Length (SPL, %)。frontier reweighting 默认使用 Gemini-2.5-flash；低层导航用带 VLFM 预训练权重的 DD-PPO point-goal policy；frontier 检测用 FrontierNet 预训练权重；所有 benchmark 用同一套系统参数，每 6 步做一次 frontier 检测与推理；实验在单张 RTX 4090 (24GB) 上完成。Evaluation is carried out within the Habitat framework on three object-goal navigation datasets: HM3D ObjectNav, MP3D ObjectNav (closed vocabulary) and OVON (open vocabulary, with more diverse language descriptions), with Success Rate (SR, %) and Success weighted by Path Length (SPL, %) as metrics. Frontier reweighting uses Gemini-2.5-flash by default; low-level navigation uses a DD-PPO point-goal policy with VLFM pretrained weights; frontier detection uses FrontierNet pretrained weights; all benchmarks share the same set of system parameters, with frontier detection and inference run every 6 steps; the experiments were completed on a single RTX 4090 (24GB).
OpenFrontier 在“单一、统一配置”下于三个数据集上 outperform 大多数 baseline。原文明确指出：与最强的竞争方法相比，“OpenFrontier is 1.5% lower in SPL on HM3D validation and 0.5% lower in SR on OVON than Uni-NaVid, and 0.3% lower in SR than UniGoal.” 但 Uni-NaVid 是在这些 benchmark 及其动作空间上专门微调过的 VLN 模型，UniGoal 依赖密集建图、场景图构建以及 LLM+VLM 联合使用；相比之下，“UniGoal performs more than 20% lower in SR than OpenFrontier on HM3D.” 也就是说 OpenFrontier 用最轻量的设计取得了稳定且有竞争力的表现。Under a "single, unified configuration", OpenFrontier outperforms most baselines on all three datasets. The paper states explicitly that, compared with the strongest competing methods, "OpenFrontier is 1.5% lower in SPL on HM3D validation and 0.5% lower in SR on OVON than Uni-NaVid, and 0.3% lower in SR than UniGoal." Yet Uni-NaVid is a VLN model specifically fine-tuned on these benchmarks and their action spaces, and UniGoal depends on dense mapping, scene-graph construction and the joint use of an LLM and a VLM; by contrast, "UniGoal performs more than 20% lower in SR than OpenFrontier on HM3D." In other words, OpenFrontier attains stable and competitive performance with the lightest design.
Navigation Results（Fig. 4，HM3D 场景 5cdEh9F2hJL，目标 bed）。红色方块和阴影区域表示 ground-truth 目标位置及其成功区域。相比 InstructNav 和 VLFM 两个代表性 baseline，OpenFrontier 在多岔路口等复杂场景中展现更强的决策能力，能直接朝目标区域前进，避免对无关区域的冗余探索。Navigation Results (Fig. 4, HM3D scene 5cdEh9F2hJL, target bed). The red square and the shaded region mark the ground-truth target location and its success area. Compared with the two representative baselines InstructNav and VLFM, OpenFrontier shows stronger decision-making in complex scenes such as multi-way junctions, heading straight toward the target region and avoiding redundant exploration of irrelevant areas.
把 frontier-reweighting VLM 从 Gemini-2.5-flash 换成其他公开或开源模型（包括更老的 LLaVA 1.6），“within OpenFrontier, VLM-induced variance is small (≤ 3% SR)”，说明框架为不同 VLM 提供了稳定通用的接口。而在同一 VLM backbone 下与 baseline 对比时，“At a matched VLM, OpenFrontier substantially outperforms baselines, indicating that the gain is driven by the visual-frontier design rather than by VLM strength.” 此外，作者还用不同目标语境测试上下文推理能力：对比 “Plant” 与 “Plant in Bathroom” 两个目标，OpenFrontier 对相同 frontier 位置给出不同相关概率，前者的轨迹偏向浴室相关区域，后者则偏向常见的客厅区域。Replacing the frontier-reweighting VLM Gemini-2.5-flash with other public or open-source models (including the older LLaVA 1.6), "within OpenFrontier, VLM-induced variance is small (≤ 3% SR)", which shows that the framework offers a stable, general interface for different VLMs. When compared with baselines under a matched VLM backbone, "At a matched VLM, OpenFrontier substantially outperforms baselines, indicating that the gain is driven by the visual-frontier design rather than by VLM strength." In addition, the authors probe contextual reasoning with different goal contexts: for the two goals "Plant" and "Plant in Bathroom", OpenFrontier assigns different relevance probabilities to the same frontier locations, the trajectory of the former leaning toward bathroom-related areas and that of the latter toward the usual living-room areas.
在真实世界部署中，作者把 OpenFrontier 集成进 ROS，部署到 Boston Dynamics Spot 上，摄像头装在机械臂末端，位姿由机载 vision-inertial odometry 提供，展示了在大型室内环境中自主导航寻找灭火器等目标的例子。小规模量化评测中，“covering 5 different target objects with 10 navigation trials in total, and obtain a 70% success rate.”For real-world deployment, the authors integrate OpenFrontier into ROS and run it on a Boston Dynamics Spot, with the camera mounted at the end of the arm and the pose supplied by onboard vision-inertial odometry, demonstrating autonomous navigation toward targets such as a fire extinguisher in large indoor environments. In a small-scale quantitative evaluation, "covering 5 different target objects with 10 navigation trials in total, and obtain a 70% success rate."
04 Limitations
Note：本节前三点为论文 “Insights → When and Why It Fails” 小节中作者明确陈述（stated）的失败模式与局限；最后一点为编者根据实验设置推断（inferred），论文未将其单列为 limitation。Note: The first three points of this section are failure modes and limitations explicitly stated by the authors in the paper’s "Insights → When and Why It Fails" subsection; the last point is inferred by the editor from the experimental setup and is not singled out as a limitation in the paper.
失败模式因 benchmark 而异，但模式一致（stated）Failure modes differ across benchmarks but follow a consistent pattern (stated)
“As Fig. 8 shows, failure types differ across benchmarks but follow a consistent pattern.” 在 HM3D / MP3D / OVON 三个 benchmark 上，作者的 Sankey 失败案例分析显示：“the two most common failure modes are false-positive target detections and termination due to exhausting the step budget.”"As Fig. 8 shows, failure types differ across benchmarks but follow a consistent pattern." Across the three benchmarks HM3D / MP3D / OVON, the authors’ Sankey failure-case analysis shows that "the two most common failure modes are false-positive target detections and termination due to exhausting the step budget."
终止判据在开放世界导航中并不平凡（stated）Termination criteria are non-trivial in open-world navigation (stated)
“One major challenge lies in termination criteria, which is non-trivial in open-world navigation settings.” 部分失败也受 Habitat benchmark 自身限制影响。"One major challenge lies in termination criteria, which is non-trivial in open-world navigation settings." Some failures are also affected by limitations of the Habitat benchmark itself.
进度停滞后缺乏失败恢复机制（stated）No failure-recovery mechanism once progress stalls (stated)
“our analysis indicates that these cases also expose a broader limitation of OpenFrontier, namely its limited ability to recover from failure once progress stalls. Even after a target is detected and navigation reduces to point-goal execution, the robot may move in incorrect directions or become trapped in local minima. In such cases, the system currently lacks a mechanism to quickly recognize failure and trigger re-reasoning or replanning at the semantic level.” 作者认为可靠的失败检测与恢复是未来重要的改进方向。"our analysis indicates that these cases also expose a broader limitation of OpenFrontier, namely its limited ability to recover from failure once progress stalls. Even after a target is detected and navigation reduces to point-goal execution, the robot may move in incorrect directions or become trapped in local minima. In such cases, the system currently lacks a mechanism to quickly recognize failure and trigger re-reasoning or replanning at the semantic level." The authors regard reliable failure detection and recovery as an important direction for future work.
真实世界量化评测规模较小（inferred）The real-world quantitative evaluation is small in scale (inferred)
论文报告的真实机器人量化评测仅覆盖 “5 different target objects with 10 navigation trials in total”（70% 成功率），样本量较小，尚不能像仿真 benchmark 那样充分验证真实世界场景下的统计显著性；这一点由编者从实验设置推断，论文正文未将其列为 limitation。The quantitative real-robot evaluation reported in the paper covers only "5 different target objects with 10 navigation trials in total" (70% success rate); the sample is small and cannot validate statistical significance in real-world settings as thoroughly as the simulation benchmarks. This point is inferred by the editor from the experimental setup and is not listed as a limitation in the body of the paper.
Failure Case Analysis（Fig. 8 左图，HM3D）。跨三个 benchmark，最常见的两类失败是目标误检（false-positive target detections）与步数耗尽终止（termination due to exhausting the step budget）。Failure Case Analysis (Fig. 8, left panel, HM3D). Across the three benchmarks, the two most common failure types are false-positive target detections and termination due to exhausting the step budget.
