TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies

01 动机Motivation

大型视觉-语言-动作（VLA）模型在机器人操控领域展现出强大的泛化能力，但它们往往只"看当下"——忽视了机器人自身的历史运动轨迹，导致决策更多是对当前帧的被动反应，而非对空间历史的主动推理。Large vision-language-action (VLA) models show strong generalization in robotic manipulation, yet they tend to "look only at the present" — ignoring the robot’s own past motion trace, so that decisions become passive reactions to the current frame rather than active reasoning over spatial history.

"VLA-powered robots often struggle to maintain awareness of their past movements, leading to decisions that are more reactive to current inputs rather than informed by spatial history."

已有方法尝试将历史帧拼接输入或用文本坐标描述轨迹，但效果有限：拼接六帧历史图像反而使性能下降 6%；文本轨迹描述仅带来约 2.4% 的提升。根本原因在于 VLA 的 backbone（视觉编码器）并未针对时序感知专门训练，无法从原始图像差异中有效提取运动信息。Prior work has tried feeding concatenated past frames or describing traces with textual coordinates, but the gains are limited: concatenating six past frames actually degrades performance by 6%, and textual trace descriptions bring only about a 2.4% improvement. The root cause is that the VLA backbone (the visual encoder) is not specifically trained for temporal awareness and cannot effectively extract motion information from raw image differences.

TraceVLA teaser：视觉轨迹叠加示意 — TraceVLA 的核心思路：将 Co-Tracker 提取的历史点轨迹（彩色曲线）直接叠加在当前机器人观测图像上，形成"视觉轨迹提示"，随后连同原始图像一起输入给 VLA 模型（中间插入 separator token）。此方法无需修改模型结构，仅在输入层面增强了空间时序信息。The core idea of TraceVLA: the historical point traces extracted by Co-Tracker (colored curves) are overlaid directly on the current robot observation image, forming a "visual trace prompt", which is then fed to the VLA model together with the original image (with a separator token in between). The method requires no change to the model architecture and enriches spatial-temporal information purely at the input level.

~10%SimplerEnv 137 项平均提升
（vs. OpenVLA 7B）Average gain over the 137 SimplerEnv configurations
(vs. OpenVLA 7B)

3.5×真实机器人任务成功率提升
（WidowX-250）Success-rate gain on real robot tasks
(WidowX-250)

+4.2%LIBERO 四套基准平均提升
（74.8% vs. 70.6%）Average gain over the four LIBERO suites
(74.8% vs. 70.6%)

8/10非训练任务 pick-place corn
（OpenVLA 仅 1/10）Unseen task pick-place corn
(OpenVLA only 1/10)

02 方法Method

TraceVLA 在不改变 VLA 模型结构的前提下，通过"输入层视觉增强"注入历史运动信息：用 Co-Tracker 从历史帧中提取稠密点轨迹，过滤出活跃轨迹后叠加到当前观测帧，再将原始帧与叠加帧组成双图像序列输入模型。Without altering the VLA model architecture, TraceVLA injects past motion information through "visual augmentation at the input layer": Co-Tracker extracts dense point traces from past frames, the active traces are filtered out and overlaid on the current observation frame, and the original frame together with the overlaid frame forms a dual-image sequence fed to the model.

视觉轨迹生成流程（Figure 2）：① Co-Tracker 对历史图像序列进行稠密点跟踪，生成每个点在各帧的位置轨迹；② 以像素位移超过阈值 κ 为准则，过滤出"活跃"轨迹（排除静态背景点）；③ 将选中的轨迹以彩色曲线渲染叠加在当前观测帧上。模型输入由 [原始帧 + separator token + 轨迹叠加帧] 构成。Visual trace generation pipeline (Figure 2): ① Co-Tracker performs dense point tracking over the historical image sequence, producing the position trace of every point across the frames; ② traces whose pixel displacement exceeds the threshold κ are kept as "active" traces (static background points are excluded); ③ the selected traces are rendered as colored curves and overlaid on the current observation frame. The model input consists of [original frame + separator token + trace-overlaid frame].

三步生成 Visual TraceGenerating a Visual Trace in Three Steps

Step 1：点轨迹提取Step 1: Point Trace Extraction

使用 Co-Tracker 对过去 N 帧（默认 N=6）进行稠密点跟踪，输出每个采样点在各历史帧的 2D 位置序列。Co-Tracker 是专门针对视频跟踪优化的模型，能可靠地跨帧定位物体与末端执行器。A Co-Tracker is used to run dense point tracking over the past N frames (N=6 by default), outputting the 2D position sequence of every sampled point across the historical frames. Co-Tracker is a model optimized specifically for video tracking and can reliably localize objects and the end-effector across frames.

Step 2：活跃轨迹过滤Step 2: Active Trace Filtering

计算每个点在历史帧中的总像素位移；仅保留位移超过阈值 κ 的点轨迹。此步骤排除静态背景，聚焦于末端执行器及与任务相关的运动对象。The total pixel displacement of each point across the historical frames is computed; only point traces whose displacement exceeds the threshold κ are kept. This step excludes the static background and focuses on the end-effector and on task-relevant moving objects.

Step 3：视觉叠加与双图输入Step 3: Visual Overlay and Dual-Image Input

将过滤后的轨迹以彩色线段渲染在当前观测帧上，生成"轨迹叠加帧"。模型接收 [原始帧 | separator token | 轨迹叠加帧] 的双图像序列，其中 separator token 帮助模型区分两种信息来源。The filtered traces are rendered as colored line segments on the current observation frame, producing a "trace-overlaid frame". The model receives the dual-image sequence [original frame | separator token | trace-overlaid frame], in which the separator token helps the model distinguish the two sources of information.

训练 Dropout 机制Training-Time Dropout

以概率 α 随机丢弃训练样本中的视觉轨迹（仅输入原始帧），使模型在推理时 Co-Tracker 跟踪失败或轨迹缺失的情况下仍能正常工作，增强鲁棒性。With probability α the visual trace is randomly dropped from a training sample (only the original frame is fed in), so that the model still works at inference time when Co-Tracker tracking fails or traces are missing, which improves robustness.

模型基座与训练Base Model and Training

TraceVLA 在 OpenVLA（基于 Prismatic 7B VLM）基础上 fine-tune，训练数据为来自 Open X-Embodiment 数据集的 150K 机器人操作轨迹。同时提供轻量版 TraceVLA-Phi3（4B），基于 OpenVLA-OFT 框架，具有更低推理延迟。视觉编码器与语言模型权重均参与微调。TraceVLA is fine-tuned on top of OpenVLA (built on the Prismatic 7B VLM), with 150K robot manipulation trajectories from the Open X-Embodiment dataset as training data. A lightweight variant TraceVLA-Phi3 (4B) is also provided, built on the OpenVLA-OFT framework and offering lower inference latency. Both the visual encoder and the language model weights take part in fine-tuning.

03 实验Experiments

在三个评测维度上全面验证：仿真 SimplerEnv（137 配置）、LIBERO 四套基准（细粒度 fine-tune 场景），以及真实 WidowX-250 机械臂（8 项操作任务，含训练内与训练外任务）。Validated comprehensively along three evaluation axes: the SimplerEnv simulator (137 configurations), the four LIBERO suites (fine-grained fine-tuning setting), and a real WidowX-250 arm (8 manipulation tasks, covering both seen and unseen tasks).

SimplerEnv 仿真基准SimplerEnv Simulation Benchmark

任务Task	OpenVLA (7B)	TraceVLA (7B)	Δ
Move Near	47.1%	53.7%	+6.6%
Pick Coke Can	15.3%	28.0%	+12.7%
Open/Close Drawer	49.5%	57.0%	+7.5%
Overall	40.2%	47.7%	+7.5%

任务Task	OpenVLA-Phi3 (4B)	TraceVLA-Phi3 (4B)	Δ
Move Near	46.1%	50.4%	+4.3%
Pick Coke Can	46.7%	52.2%	+5.5%
Open/Close Drawer	22.5%	31.0%	+8.5%
Overall	39.9%	44.0%	+4.1%

LIBERO 基准（fine-tune 场景）LIBERO Benchmark (fine-tuning setting)

Suite	OpenVLA	TraceVLA	Δ
Spatial	82.6%	84.6%	+2.0%
Object	83.8%	85.2%	+1.4%
Goal	70.4%	75.1%	+4.7%
Long	45.7%	54.1%	+8.4%
Average	70.6%	74.8%	+4.2%

真实机器人实验（WidowX-250）Real-Robot Experiments (WidowX-250)

真实机器人实验结果 — TraceVLA 在 WidowX-250 真实机器人上的实验结果（Figure 6）：左侧为训练内 seen 任务，右侧为训练外 unseen 任务。Pick-place corn（unseen）：TraceVLA 8/10 成功，OpenVLA 仅 1/10。TraceVLA 能根据语言指令正确定位空间目标，而 OpenVLA 在空间语义理解上频繁失败。Experimental results of TraceVLA on the real WidowX-250 robot (Figure 6): the left side shows seen tasks from training, the right side unseen tasks. Pick-place corn (unseen): TraceVLA succeeds 8/10 times, OpenVLA only 1/10. TraceVLA correctly localizes the spatial target from the language instruction, whereas OpenVLA frequently fails at spatial semantic understanding.

消融实验Ablation Studies

SimplerEnv 各任务性能对比 — SimplerEnv 137 配置下各任务类别的性能对比图（Figure 3）：TraceVLA (7B) 与 TraceVLA-Phi3 (4B) 在所有任务上均优于对应基线。Per-task-category performance comparison under the 137 SimplerEnv configurations (Figure 3): TraceVLA (7B) and TraceVLA-Phi3 (4B) both outperform their corresponding baselines on all tasks.

Further fine-tuning alone（仅进一步微调，无轨迹）：仅带来 1.1% 提升，说明性能增益主要来自视觉轨迹，而非微调本身。Further fine-tuning alone (further fine-tuning only, no traces): brings a gain of just 1.1%, showing that the performance gain comes mainly from the visual traces rather than from fine-tuning itself.
历史帧拼接（6 帧原始图像输入）：性能反而下降 6%，冗余视觉信息对 VLA 产生干扰。Concatenating past frames (6 raw images as input): performance actually drops by 6%; redundant visual information interferes with the VLA.
Text trace vs. visual trace：文字坐标描述 +2.4%；视觉叠加方式 +6.4%（在文字基础上再增），视觉编码方式更有效。Text trace vs. visual trace: textual coordinate descriptions give +2.4%, whereas the visual overlay gives +6.4% (a further gain on top of the textual form); visual encoding is the more effective form.
轨迹长度 N：N=6 最优；N=3 仅得 3.2% 提升；N=12 因视觉杂乱略有下降。Trace length N: N=6 is optimal; N=3 yields only a 3.2% gain; N=12 drops slightly because of visual clutter.
可视化参数（线宽、透明度、颜色）：在合理范围内对性能影响可忽略不计。Visualization parameters (line width, opacity, color): within a reasonable range their effect on performance is negligible.

04 局限性Limitations

Note：以下局限性均由作者在论文中明确陈述（stated）。Note: All the limitations below are explicitly stated by the authors in the paper.

训练显存开销增加（~10 GB）Increased training memory footprint (~10 GB)

双图像输入使每个样本的视觉 token 数量翻倍，在 batch size 32、8 张 H100 的配置下，显存消耗额外增加约 10 GB。对于显存受限的训练环境，需适当减小 batch size 或使用梯度检查点。The dual-image input doubles the number of visual tokens per sample; under a configuration with batch size 32 on 8 H100 GPUs, memory consumption grows by roughly 10 GB. In memory-constrained training environments the batch size must be reduced accordingly or gradient checkpointing used.

Co-Tracker 推理延迟（每步约 0.034 秒）Co-Tracker inference latency (about 0.034 s per step)

每个时间步需额外运行 Co-Tracker 进行点跟踪，引入约 0.034 秒的推理开销。对于要求毫秒级响应的高频控制任务，该延迟可能成为瓶颈。Every time step requires an additional Co-Tracker run for point tracking, introducing about 0.034 s of inference overhead. For high-frequency control tasks that demand millisecond-level response, this latency may become a bottleneck.

Co-Tracker 长轨迹跟踪失效Co-Tracker fails on long traces

Co-Tracker 在连续 30–40 步后可能丢失部分追踪点，需要定期重新初始化（recalibration）。在需要极长操作序列的任务中，跟踪失效率会上升，影响轨迹质量。After 30–40 consecutive steps Co-Tracker may lose some tracked points and needs periodic re-initialization (recalibration). In tasks requiring extremely long manipulation sequences the tracking failure rate rises, degrading trace quality.

轨迹遮挡关键视觉信息Traces occlude critical visual information

叠加的彩色轨迹线有时会遮挡末端执行器或任务关键物体（如待抓取的小型物体），在视觉密集的场景中可能造成干扰。尽管可视化参数对整体性能影响较小，但极端遮挡情况尚未充分评估。The overlaid colored trace lines sometimes occlude the end-effector or task-critical objects (such as a small object waiting to be grasped), which can be disruptive in visually dense scenes. Although the visualization parameters have little effect on overall performance, extreme occlusion cases have not yet been adequately evaluated.