From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models
系统研究四种 latent action 监督策略，揭示图像型 vs. 动作型 latent action 的任务适配规律A systematic study of four latent action supervision strategies, revealing how image-based vs. action-based latent actions match different task types
本文在统一的 VLA baseline 下，系统对比四种 latent action 监督策略：图像型（image-based）策略在长视野推理与场景泛化上表现更优，动作型（action-based）策略在复杂运动协调任务上更具优势；整体而言，直接让 VLM 预测离散 latent action token（LA-Direct / LA-Tok）的显式监督效果最强。Under a unified VLA baseline, this paper systematically compares four latent action supervision strategies: image-based strategies perform better on long-horizon reasoning and scene generalization, while action-based strategies are more advantageous on tasks requiring complex motion coordination; overall, explicit supervision that has the VLM directly predict discrete latent action tokens (LA-Direct / LA-Tok) is the most effective.
视觉-语言-动作模型（VLA）近年兴起，但异构动作空间（不同机器人、不同任务的动作格式各异）给策略学习带来挑战。Latent action 作为中间表示，可将视觉/动作信息压缩为统一的离散 token，从而桥接 VLM 和机器人控制。然而，latent action 的设计空间复杂——应从图像侧提取还是动作侧提取？如何与 VLM 集成？哪种监督形式最有效？这些问题缺乏系统研究。Vision-language-action models (VLA) have risen in recent years, but heterogeneous action spaces (different robots and different tasks use different action formats) pose a challenge for policy learning. As an intermediate representation, latent actions can compress visual/action information into unified discrete tokens, thereby bridging the VLM and robot control. However, the design space of latent actions is complex—should they be extracted on the image side or the action side? How should they be integrated with the VLM? Which form of supervision is most effective? These questions lack systematic study.
"We investigate how different latent action supervision choices affect VLA policy learning under a unified baseline."
图 1：Latent action 在 VLA 中的整体框架。左：统一的 VLA pipeline；右：两种视角（图像型 vs. 动作型）及四种集成策略（LA-Align、LA-Direct、LA-Cond、LA-Tok）。图像型 latent action 通过 VQ-VAE 从视觉转换中提取离散表示；动作型 latent action 则从连续动作序列中提取行为模式。Figure 1: Overall framework of latent actions in VLA. Left: the unified VLA pipeline; right: two perspectives (image-based vs. action-based) and four integration strategies (LA-Align, LA-Direct, LA-Cond, LA-Tok). Image-based latent actions extract discrete representations from visual transitions via VQ-VAE; action-based latent actions instead extract behavioral patterns from continuous action sequences.
+10.8%LA-Direct 在 LIBERO-Long 上的提升（85.8% → 96.6%）Gain of LA-Direct on LIBERO-Long (85.8% → 96.6%)
+17.5%LA-Tok 在 RoboTwin 2.0 平均成功率提升Average success-rate gain of LA-Tok on RoboTwin 2.0
+20.9%LA-Cond 多任务联合训练消除负迁移的提升Gain from LA-Cond eliminating negative transfer in multi-task joint training
4×四种系统策略在统一 baseline 下全面对比Four systematic strategies compared comprehensively under a unified baseline
02方法Method
作者在 Qwen3-VL-2B 为骨干的统一 VLA baseline 上，从两个互补视角设计四种 latent action 集成策略，并以统一目标函数 ℒ = ℒ_action + λℒ_latent 进行训练，以隔离各策略对最终性能的影响。On a unified VLA baseline with Qwen3-VL-2B as the backbone, the authors design four latent action integration strategies from two complementary perspectives, and train them with the unified objective ℒ = ℒ_action + λℒ_latent so as to isolate each strategy's effect on final performance.
图 3：五种架构实例对比。(a) Baseline：标准 VLA，无 latent action；(b) LA-Align：通过余弦相似度正则化做隐式表示对齐；(c) LA-Direct：VLM 直接预测离散 latent action token；(d) LA-Cond：VLM 联合预测 latent action token，再条件解码最终动作；(e) LA-Tok：将连续动作离散化为 token 作为 VLM 监督目标。Figure 3: Comparison of five architecture instances. (a) Baseline: standard VLA without latent actions; (b) LA-Align: implicit representation alignment via cosine-similarity regularization; (c) LA-Direct: the VLM directly predicts discrete latent action tokens; (d) LA-Cond: the VLM jointly predicts latent action tokens, then conditionally decodes the final action; (e) LA-Tok: continuous actions are discretized into tokens serving as the VLM's supervision target.
图像型 latent action 使用 VQ-VAE 将相邻帧的视觉变化编码为离散 token z_t^img，捕捉场景级动态。三种集成策略：Image-based latent actions use VQ-VAE to encode the visual change between adjacent frames into discrete tokens z_t^img, capturing scene-level dynamics. Three integration strategies:
LA-Align（Strategy 1）：隐式对齐，用余弦相似度约束 VLM 隐状态与 latent action 表示对齐，不改变输出形式。LA-Align (Strategy 1): implicit alignment, using cosine similarity to constrain the VLM hidden states to align with the latent action representation, without changing the output form.
LA-Direct（Strategy 2）：显式直接解码，VLM 直接预测离散 latent action token，作为附加监督目标。LA-Direct (Strategy 2): explicit direct decoding, where the VLM directly predicts discrete latent action tokens as an additional supervision target.
LA-Cond（Strategy 3）：显式条件解码，VLM 联合预测 latent token，动作头再以此为条件生成最终动作。LA-Cond (Strategy 3): explicit conditional decoding, where the VLM jointly predicts latent tokens and the action head then generates the final action conditioned on them.
动作型 latent action 将连续动作 chunk 压缩为离散 token z_t^act，采用频域与时域结合："FFT modeling low-frequency trends, 1D temporal convolutions capturing fast variations." Strategy 4（LA-Tok）将这些离散 token 直接作为 VLM 的监督目标，实现异构动作空间的统一表示。Action-based latent actions compress continuous action chunks into discrete tokens z_t^act, combining the frequency and time domains: "FFT modeling low-frequency trends, 1D temporal convolutions capturing fast variations." Strategy 4 (LA-Tok) uses these discrete tokens directly as the VLM's supervision target, achieving a unified representation of heterogeneous action spaces.
所有策略共享同一骨干网络（Qwen3-VL-2B）、placeholder 设计和动作头，确保对比的公平性。All strategies share the same backbone network (Qwen3-VL-2B), placeholder design and action head, ensuring fairness of the comparison.
03实验Experiments
实验在三个 benchmark 上展开：仿真环境 LIBERO（长视野任务 LIBERO-Long 等 4 个子集）、RoboTwin 2.0（双臂高难度操作，20 项任务）以及真实环境 JAKA 机械臂任务（碗叠放等多步操作）。每组实验各报告成功率（%）。Experiments are conducted on three benchmarks: the simulated LIBERO (4 subsets including the long-horizon LIBERO-Long), RoboTwin 2.0 (high-difficulty bimanual manipulation, 20 tasks), and real-world JAKA robot-arm tasks (multi-step operations such as bowl stacking). Each group of experiments reports success rate (%).
主要结果：形态-任务对应规律Main results: the modality-task correspondence
Benchmark
Baseline
LA-Align
LA-Direct
LA-Cond
LA-Tok
LIBERO 平均LIBERO average
93.1%
97.0%
97.1%
96.6%
95.5%
LIBERO-Long
85.8%
94.8%
96.6%
94.2%
92.6%
RoboTwin 2.0 平均RoboTwin 2.0 average
60.5%
70.5%
71.8%
73.8%
78.0%
图像型策略（LA-Direct）在需要长视野推理的 LIBERO-Long 上取得 +10.8% 提升（85.8% → 96.6%）；动作型策略（LA-Tok）在运动协调要求更高的 RoboTwin 2.0 上取得 +17.5% 平均提升。The image-based strategy (LA-Direct) achieves a +10.8% gain on LIBERO-Long, which requires long-horizon reasoning (85.8% → 96.6%); the action-based strategy (LA-Tok) achieves a +17.5% average gain on RoboTwin 2.0, which demands more motion coordination.
表示形式对比：离散 token vs. 连续回归Comparison of representation forms: discrete token vs. continuous regression
消融实验表明，离散 token 监督显著优于等价连续回归变体：Ablation experiments show that discrete token supervision is significantly better than the equivalent continuous regression variants:
LA-Direct：97.1% vs. 连续变体 94.4%LA-Direct: 97.1% vs. continuous variant 94.4%
LA-Tok：95.5% vs. 连续变体 93.3%LA-Tok: 95.5% vs. continuous variant 93.3%
"Directly supervising the VLM to predict latent actions provides a simple and consistent supervision target."
真实环境结果Real-world results
图 4：真实世界 JAKA 机械臂操作任务结果（完成度百分比，0–100 分，10 次 rollout 均值）。LA-Direct 在 Stack 4 Bowls 任务上得分 79 vs. baseline 48；LA-Tok 在 Stack 2–3 Bowls 任务上达到最高 89。Figure 4: Results on real-world JAKA robot-arm manipulation tasks (completion percentage, 0–100 points, mean over 10 rollouts). LA-Direct scores 79 on the Stack 4 Bowls task vs. 48 for the baseline; LA-Tok reaches a maximum of 89 on the Stack 2–3 Bowls tasks.
多任务联合训练Multi-task joint training
图 5：Baseline（左）vs. LA-Cond（右）在 10 项 RoboTwin 任务上的联合训练对比。绿色表示相对单任务训练的性能提升，红色表示下降。LA-Cond 消除了负迁移，实现 +20.9% 的平均提升，而 baseline 在多个任务上出现性能下降。Figure 5: Joint-training comparison of Baseline (left) vs. LA-Cond (right) on 10 RoboTwin tasks. Green indicates performance gains relative to single-task training, red indicates drops. LA-Cond eliminates negative transfer and achieves an average gain of +20.9%, whereas the baseline degrades on several tasks.
数据效率Data efficiency
LA-Tok 仅使用 50% 数据即可达到 94.0% 成功率，而 baseline 使用全量数据（100%）才达到 80%，体现出显著的数据效率优势。LA-Tok reaches a 94.0% success rate using only 50% of the data, whereas the baseline reaches just 80% with the full data (100%), demonstrating a significant data-efficiency advantage.
04局限性Limitations
Note:以下局限性均为作者在论文中明确陈述（stated by the authors）。All limitations below are explicitly stated by the authors in the paper.
本文提出的四种集成策略（LA-Align、LA-Direct、LA-Cond、LA-Tok）基于两个视角进行系统化设计，但并不能穷尽所有可能的 latent action 集成方式，存在其他设计空间尚待探索。The four integration strategies proposed in this paper (LA-Align, LA-Direct, LA-Cond, LA-Tok) are systematically designed from two perspectives, but they cannot exhaust all possible ways of integrating latent actions; other design spaces remain to be explored.
Latent Action 模型质量的影响未充分研究The impact of latent action model quality is under-studied
实验中使用的 image-based 和 action-based latent action 模型本身的质量（如 VQ-VAE 的重建精度、量化误差）对下游 VLA 性能的影响尚未系统分析，未来需进一步研究。The influence of the quality of the image-based and action-based latent action models themselves (e.g. the reconstruction accuracy and quantization error of VQ-VAE) on downstream VLA performance has not yet been systematically analyzed and requires further study.
真实世界评估平台单一A single real-world evaluation platform
真实世界实验仅在单臂 JAKA 机械臂平台上进行，未涵盖双臂或更多样化的真实硬件，策略在不同硬件平台上的泛化性有待验证。Real-world experiments are carried out only on a single-arm JAKA robot-arm platform, without covering bimanual or more diverse real hardware; the generalization of the strategies across different hardware platforms remains to be verified.
