LongNav-R1: Horizon-Adaptive Multi-Turn RL for Long-Horizon VLA Navigation

01 Motivation

现有 SOTA 导航方法大多采用"single-turn imitation learning paradigm"，把导航简化为基于即时局部上下文的一系列孤立动作预测。论文指出这种范式有两个致命缺陷：一是缺乏因果推理，把每一步当作独立事件，忽视了早期探索对后期目标导向效率的先决作用；二是行为僵化，严格模仿专家轨迹而不是针对目标成功率去优化，导致智能体"myopic and brittle"，难以从错误中恢复或适应分布偏移。Most existing SOTA navigation methods adopt a "single-turn imitation learning paradigm", reducing navigation to a series of isolated action predictions conditioned on the immediate local context. The paper identifies two fatal defects in this paradigm: first, the absence of causal reasoning — every step is treated as an independent event, ignoring the prerequisite role that early exploration plays in later goal-directed efficiency; second, behavioral rigidity — expert trajectories are imitated strictly instead of optimizing for goal success rate, leaving the agent "myopic and brittle" and hard to recover from mistakes or adapt to distribution shift.

"LongNav-R1 treats the navigation task as a continuous conversation between the VLA policy and the physical environment." —— 用多轮对话取代单轮孤立预测，让模型能学习当前动作与远期奖励之间的因果关系，并通过在线交互鼓励多样化轨迹探索，克服人类示范带来的行为刚性。"LongNav-R1 treats the navigation task as a continuous conversation between the VLA policy and the physical environment." — Replacing single-turn isolated prediction with multi-turn conversation lets the model learn the causal relation between the current action and distant rewards, and encourages diverse trajectory exploration through online interaction, overcoming the behavioral rigidity induced by human demonstrations.

02 Method

整体流程分三步迭代：i) 通过多轮交互式 rollout 收集长程轨迹；ii) 用提出的 horizon-adaptive estimator 计算动作级优势 (advantage)；iii) 通过聚合后的优化目标更新 VLA 模型。训练策略上先做 "Balanced IL Warm-up"（在人类示范数据上做 teacher-forcing 的监督预训练），再进入 "Online multi-turn RL training" 阶段用 HAPO 目标继续优化。The overall pipeline iterates over three steps: i) collecting long-horizon trajectories via multi-turn interactive rollouts; ii) computing action-level advantages with the proposed horizon-adaptive estimator; iii) updating the VLA model with the aggregated optimization objective. The training strategy first performs a "Balanced IL Warm-up" (teacher-forcing supervised pre-training on human demonstration data), then enters an "Online multi-turn RL training" stage that keeps optimizing with the HAPO objective.

多轮 VLA 建模 (Multi-turn RL formulation)Multi-turn VLA modeling (Multi-turn RL formulation)

动作按自回归方式生成，条件是全局指令 ℐ、历史 {v₁:t₋₁, a₁:t₋₁} 与当前观测 vₜ。这一形式带来两点好处：一是可复用 KV cache，历史信息缓存后只需计算新观测，这对涉及数百步、生成数十万视觉 token 的长程导航至关重要；二是序列化建模能显式捕获先前交互的因果依赖，把最终轨迹结果归因到具体动作，从而提升学习效率与长程表现。此外论文还引入"online token pruning"——只保留与历史上下文特征相似度低于阈值 δ 的视觉 token，进一步压缩计算开销。Actions are generated autoregressively, conditioned on the global instruction ℐ, the history {v₁:t₋₁, a₁:t₋₁} and the current observation vₜ. This formulation brings two benefits: first, the KV cache can be reused — once the history is cached only the new observation needs to be computed, which is crucial for long-horizon navigation that spans hundreds of steps and generates hundreds of thousands of visual tokens; second, sequential modeling explicitly captures the causal dependencies of prior interactions and attributes the final trajectory outcome to specific actions, thereby improving learning efficiency and long-horizon performance. The paper further introduces "online token pruning" — keeping only the visual tokens whose feature similarity with the historical context falls below a threshold δ, further compressing the computational cost.

Horizon-Adaptive Policy Optimization (HAPO)

HAPO 是一个 critic-free 的优势估计框架：把 baseline 建模为对 rollout buffer 做核回归 (kernel regression) 的非参数值函数估计，而不是训练一个独立的价值网络。具体地，advantage 定义为 A = G − V_ℬ^K(s)，其中 baseline V_ℬ^K(s) 通过 leave-one-out 核回归在 buffer 内跨轨迹计算。论文证明该框架具有一般性：当核函数取常数、特征只在终止状态定义时，就退化为 REINFORCE++（"the entire batch shares a uniform baseline"）。HAPO 采用以离散时间步为特征、Gaussian kernel 做时序平滑的"time-aware kernel"，让 baseline 能区分"高方差的早期探索阶段"和"高精度的后期目标锁定阶段"，从而实现更精确的时间信度分配 (temporal credit assignment)。训练目标在结构上类似 REINFORCE++ 损失（含 clip 和 KL 正则），但优势项换成了 HAPO 的时序自适应估计。HAPO is a critic-free advantage estimation framework: the baseline is modeled as a non-parametric value function estimate obtained by kernel regression over the rollout buffer, instead of training a separate value network. Concretely, the advantage is defined as A = G − V_ℬ^K(s), where the baseline V_ℬ^K(s) is computed across trajectories inside the buffer by leave-one-out kernel regression. The paper proves the framework is general: when the kernel is constant and the features are defined only at terminal states, it degenerates into REINFORCE++ ("the entire batch shares a uniform baseline"). HAPO adopts a "time-aware kernel" that takes the discrete time step as its feature and applies a Gaussian kernel for temporal smoothing, letting the baseline distinguish the "high-variance early exploration phase" from the "high-precision late goal-locking phase" and thus achieve more accurate temporal credit assignment. The training objective is structurally similar to the REINFORCE++ loss (with clip and KL regularization), but its advantage term is replaced by HAPO’s horizon-adaptive estimate.

03 Experiments

在 object-goal navigation 与 open-vocabulary navigation 两类任务、四个基准（HM3D V1、V2、MP3D、HM3D-OVON）上评估，指标为 success rate (SR) 与 success rate weighted by path length (SPL)。VLA 策略以 Qwen3-VL-2B 为 base，仿真中相机高度 0.88m、输出 640×480 RGB，成功距离阈值 1m，动作集合为 MOVE FORWARD (0.25m) / TURN LEFT/RIGHT (30°) / STOP。此外还在真实环境（Orbbec Femto Bolt 传感器 + TidyBot++ 移动底盘）做零样本部署验证。Evaluation covers two task families, object-goal navigation and open-vocabulary navigation, over four benchmarks (HM3D V1, V2, MP3D, HM3D-OVON), with success rate (SR) and success rate weighted by path length (SPL) as metrics. The VLA policy takes Qwen3-VL-2B as its base; in simulation the camera height is 0.88m and the output is 640×480 RGB, the success distance threshold is 1m, and the action set is MOVE FORWARD (0.25m) / TURN LEFT/RIGHT (30°) / STOP. Zero-shot deployment is additionally validated in the real world (Orbbec Femto Bolt sensor + TidyBot++ mobile base).

与 SOTA 对比Comparison with SOTA

"LongNav-R1 consistently outperforms existing baselines across all datasets, improving success rates to 76.0%, 83.7%, and 63.0% on HM3D V1, V2, and MP3D, respectively." 值得注意的是，LongNav-R1 仅用 RGB 观测就超过了依赖完整 odometry + depth 的方法；相比此前端到端 VLA 模型 UniNavid，路径效率 (path efficiency) 提升 7.2%。在 open-vocabulary 的 HM3D-OVON 上，模型只在 HM3D 上训练、未做任何微调即达到 SOTA，比传统 transformer-based RL 方法（RL、BCRL、DAgRL）高约 30%，并超越单轮 RL baseline Nav-R1。"LongNav-R1 consistently outperforms existing baselines across all datasets, improving success rates to 76.0%, 83.7%, and 63.0% on HM3D V1, V2, and MP3D, respectively." Notably, LongNav-R1 uses RGB observations alone yet surpasses methods that rely on full odometry + depth; compared with the earlier end-to-end VLA model UniNavid, path efficiency improves by 7.2%. On the open-vocabulary HM3D-OVON benchmark the model is trained only on HM3D and reaches SOTA without any fine-tuning, roughly 30% above traditional transformer-based RL methods (RL, BCRL, DAgRL), and it also beats the single-turn RL baseline Nav-R1.

TABLE III（节选）: Effectiveness of multi-turn reinforcement learning across horizon length. 注意 SFT / REINFORCE++ / HAPO(σ=∞) 在 >200 步的长程场景成功率均为 0.0 —— 只有带受限时序窗口的 HAPO(σ=30) 能在长程场景解锁非零成功率。TABLE III (excerpt): Effectiveness of multi-turn reinforcement learning across horizon length. Note that SFT / REINFORCE++ / HAPO(σ=∞) all score 0.0 on the >200-step long-horizon setting — only HAPO(σ=30), with its bounded temporal window, unlocks a non-zero success rate in the long-horizon regime.

Ablations

核带宽 (kernel bandwidth) σ 的选择直接影响 baseline 的估计误差与最终性能："HAPO with σ=30 can achieve smaller estimation error compared to σ=∞ ... this reduced estimation error directly correlates with improved performance." 即受限时序窗口的核回归优于无限窗口（等价于全局均匀）的 baseline，说明简单的均匀密集估计无法刻画机器人导航中复杂的时序动态。计算开销方面，"LongNav-R1 remains highly performant, requiring only 0.23s to process a 400-step sequence"，约 5 FPS，可用于真实环境部署——这得益于多轮结构支持的 KV cache 复用与在线 token 剪枝。The choice of kernel bandwidth σ directly affects the estimation error of the baseline and the final performance: "HAPO with σ=30 can achieve smaller estimation error compared to σ=∞ ... this reduced estimation error directly correlates with improved performance." That is, kernel regression with a bounded temporal window beats an infinite window (equivalent to a globally uniform) baseline, showing that a simple uniform dense estimate cannot characterize the complex temporal dynamics of robot navigation. As for computational cost, "LongNav-R1 remains highly performant, requiring only 0.23s to process a 400-step sequence", about 5 FPS, which is usable for real-world deployment — thanks to the KV cache reuse and online token pruning that the multi-turn structure enables.

04 Limitations