VLA-JEPA 提出了一种基于 JEPA（Joint Embedding Predictive Architecture）的潜空间世界模型预训练框架，专为视觉-语言-动作（VLA）策略设计。通过在隐空间而非像素空间中预测未来帧的表示，模型能够学到对相机运动、光照变化和背景干扰均鲁棒的动态抽象，在 LIBERO、SimplerEnv 及真实机器人实验上均达到业界领先性能。VLA-JEPA introduces a latent world-model pretraining framework built on JEPA (Joint Embedding Predictive Architecture) and designed specifically for vision-language-action (VLA) policies. By predicting representations of future frames in latent space rather than pixel space, the model learns dynamic abstractions that are robust to camera motion, illumination changes and background distractors, reaching state-of-the-art performance on LIBERO, SimplerEnv and real-robot experiments.
VLAJEPAlatent world model机器人操作robot manipulation预训练pretrainingflow matching视觉语言动作模型vision-language-action model迁移鲁棒性transfer robustness
01动机 MotivationMotivation
现有基于潜在动作预测的 VLA 预训练方法（如 LAPA、UniVLA）将优化目标锁定在像素级变化上，而非真正与控制相关的状态转移。这使得模型对相机抖动、背景变化等无关干扰极为敏感，难以在真实环境中稳健泛化。Existing VLA pretraining methods based on latent action prediction (such as LAPA and UniVLA) anchor their optimization objective to pixel-level changes rather than to the state transitions that actually matter for control. This makes the models highly sensitive to irrelevant disturbances such as camera shake and background variation, and hard to generalize robustly in real environments.
"For embodied control, we want a value-bearing latent state that discards nuisance appearance while preserving factors governing state evolution."
图1：VLA-JEPA 整体框架概览。该系统在模拟与真实场景中均表现出色，通过隐空间世界模型预训练赋予策略对环境扰动的鲁棒性。Figure 1: Overview of the VLA-JEPA framework. The system performs well in both simulated and real-world settings, with latent world-model pretraining endowing the policy with robustness to environmental perturbations.
作者归纳了现有隐动作预训练的四类核心失效模式：The authors summarize four core failure modes of existing latent-action pretraining:
像素级偏置（Pixel-level bias）：优化目标锁定像素差异，而非动作相关的状态转移；Pixel-level bias: the objective is locked onto pixel differences rather than action-relevant state transitions;
干扰运动（Nuisance motion）：真实视频中相机运动等无关信号被放大，遮盖了真正的交互信号；Nuisance motion: irrelevant signals such as camera motion in real videos get amplified and mask the genuine interaction signal;
信息泄漏（Information leakage）：未来帧的特征在预测网络中提前泄露，形成捷径；Information leakage: features of future frames leak into the prediction network in advance, creating a shortcut;
多阶段复杂性（Multi-stage complexity）：三阶段以上的流程引入额外脆弱性。Multi-stage complexity: pipelines with three or more stages introduce extra fragility.
VLA-JEPA 采用两阶段流程：先以 JEPA 目标进行世界模型预训练，再对 action head 进行微调。骨干网络使用 Qwen3-VL，视频编码采用 V-JEPA2 encoder，动作生成采用 flow matching。整体设计无需像素重建，无需复杂的多阶段解码器。VLA-JEPA adopts a two-stage pipeline: world-model pretraining with a JEPA objective, followed by fine-tuning of the action head. The backbone is Qwen3-VL, video is encoded with a V-JEPA2 encoder, and actions are generated by flow matching. The overall design needs neither pixel reconstruction nor a complex multi-stage decoder.
图1（详细）：VLA-JEPA 模型架构。VLM 处理当前观测，通过可学习 token <latent_i> 和 <action> 生成世界状态预测与动作。Target encoder 编码未来帧，仅供损失计算，不参与前向推理。Figure 1 (detailed): the VLA-JEPA model architecture. The VLM processes the current observation and produces world-state predictions and actions through the learnable tokens <latent_i> and <action>. The target encoder encodes future frames and serves only the loss computation, never the forward inference path.
隐空间无泄漏状态预测（Leakage-Free State Prediction）Leakage-Free State Prediction in Latent Space
核心创新：Target encoder 处理未来帧，而 student 网络（VLM）仅能看到当前观测。模型在隐空间对齐预测表示与目标表示：Core idea: the target encoder processes future frames, while the student network (the VLM) can see only the current observation. The model aligns the predicted representation with the target representation in latent space: ℒ_WM = Σ(k=1 to T) 𝔼[||ŝ_tk − s_tk||²] 其中 ŝ_tk 为预测的世界状态，s_tk 为 target encoder 产生的目标表示。通过在隐空间而非像素空间预测，模型天然过滤了像素级的无关变化（光照、背景、相机运动），专注于语义状态转移。where ŝ_tk is the predicted world state and s_tk is the target representation produced by the target encoder. Predicting in latent space rather than pixel space naturally filters out pixel-level nuisance variation (illumination, background, camera motion) and concentrates on semantic state transitions.
Flow-Matching 动作头Flow-Matching Action Head
动作生成采用 flow matching 对连续轨迹分布建模：Action generation uses flow matching to model the distribution over continuous trajectories: ℒ_FM = 𝔼[||v_θ(a_t,t|z_a) − (a₀:H − ε)||₂²] 综合训练目标为：ℒ = ℒ_FM + β·ℒ_WM，β 为超参数权衡世界模型损失与动作预测损失。The overall training objective is ℒ = ℒ_FM + β·ℒ_WM, where β is a hyperparameter balancing the world-model loss against the action-prediction loss.
图2：VLA-JEPA 支持跨域联合训练。人类视频数据仅使用隐空间对齐损失（alignment loss），机器人数据同时使用对齐损失与动作预测损失，在同一框架内无缝整合。Figure 2: VLA-JEPA supports cross-domain joint training. Human video data uses only the latent alignment loss, while robot data uses both the alignment loss and the action-prediction loss, integrated seamlessly within a single framework.
跨域人类视频迁移Cross-Domain Transfer from Human Videos
框架支持无缝整合无动作标注的人类视频与有标注机器人数据。人类视频只需 world modeling loss，不需要动作标签。实验表明，人类视频主要强化"模型已有技能的鲁棒性和稳定性"，而非引入全新的动作动力学。The framework seamlessly combines action-free human videos with annotated robot data. Human videos need only the world modeling loss, no action labels. Experiments show that human videos mainly reinforce "the robustness and stability of skills the model already has" rather than introducing entirely new action dynamics.
03实验 ExperimentsExperiments
在 LIBERO（4个任务套件）、LIBERO-Plus（7种扰动维度）、SimplerEnv（Google Robot + WidowX）和真实 Franka 机器人上进行评估，与 OpenVLA-OFT、π₀、π₀.₅、UniVLA、GR00T N1 等方法对比。Evaluation is carried out on LIBERO (4 task suites), LIBERO-Plus (7 perturbation dimensions), SimplerEnv (Google Robot + WidowX) and a real Franka robot, in comparison with OpenVLA-OFT, π₀, π₀.₅, UniVLA, GR00T N1 and other methods.
图4：真实世界 Franka 机器人实验结果，对比分布内任务、任务分布外（Task OOD）和布局随机化（Layout OOD）三种场景。VLA-JEPA 在分布内表现最优，在布局随机化设置中同样表现最佳；注意力可视化显示模型更精准地关注机械臂和目标物体。Figure 4: results of the real-world Franka robot experiments, comparing three settings: in-distribution tasks, task out-of-distribution (Task OOD) and layout randomization (Layout OOD). VLA-JEPA is the strongest in distribution and likewise the best under layout randomization; the attention visualization shows that the model attends more precisely to the robot arm and the target objects.
消融实验（Ablations）Ablations
视频时间窗口（video horizon）消融（Table 4）：T=4 帧时信息不足（avg 94.8%），T=16 帧时引入冗余信息（avg 95.5%），T=8 帧取得最优（avg 96.1%）。注意力可视化（Figure 6）显示：LAPA 呈现"excessively dense visual information"，UniVLA 过度关注背景元素，而 VLA-JEPA "focus[ed] more precisely on the operation, for instance, the robotic arm, the hand, and the objects"。Video horizon ablation (Table 4): at T=4 frames the information is insufficient (avg 94.8%), at T=16 frames redundant information is introduced (avg 95.5%), and T=8 gives the best result (avg 96.1%). The attention visualization (Figure 6) shows that LAPA exhibits "excessively dense visual information", UniVLA over-attends to background elements, whereas VLA-JEPA "focus[ed] more precisely on the operation, for instance, the robotic arm, the hand, and the objects".
图6：latent action token 对 image token 注意力权重矩阵可视化。左：LAPA 注意力分散；中：UniVLA 过度关注背景；右：VLA-JEPA 精准聚焦机械臂与操作目标。Figure 6: visualization of the attention weight matrix from latent action tokens to image tokens. Left: LAPA attention is diffuse; middle: UniVLA over-attends to the background; right: VLA-JEPA focuses precisely on the robot arm and the manipulation target.图5：人类视频数据在预训练中的占比对 LIBERO-Plus 各扰动维度成功率的影响。结论：人类视频主要增强模型在干扰场景下的鲁棒性，而非引入新的操作技能；随比例增大，鲁棒性指标持续提升。Figure 5: effect of the proportion of human video data in pretraining on the success rate along each LIBERO-Plus perturbation dimension. Conclusion: human videos mainly enhance the robustness of the model under distracting scenarios rather than introducing new manipulation skills; as the proportion grows, the robustness metrics keep improving.
04局限性 LimitationsLimitations
注：论文未设专门的 Limitations 章节。以下内容部分为作者在实验讨论中明确指出（stated），部分为从设计中推断（inferred）。Note: the paper has no dedicated Limitations section. Some of the points below are explicitly made by the authors in their experimental discussion (stated), others are inferred from the design (inferred).
泛化能力弱于 π₀.₅（Stated）Generalization weaker than π₀.₅ (Stated)
论文明确指出："the generalization capability of VLA-JEPA is less robust than π₀.₅, yet it produces more stable trajectories." 在真实任务 OOD 场景中，π₀.₅ 能更准确地跟随指令定位目标，而 VLA-JEPA "is prone to grasping objects that do not align with the command"，存在指令遵循精度不足的问题。The paper states explicitly: "the generalization capability of VLA-JEPA is less robust than π₀.₅, yet it produces more stable trajectories." In real-world task OOD scenarios, π₀.₅ follows the instruction and locates the target more accurately, whereas VLA-JEPA "is prone to grasping objects that do not align with the command", i.e. its instruction-following precision is not good enough.
人类视频未能引入新技能（Stated）Human videos bring no new skills (Stated)
实验分析显示，人类视频数据主要增强已有技能的鲁棒性，"而非教会模型新的操作动力学"。如何让大规模人类视频真正迁移新的操作能力，仍是开放问题。The experimental analysis shows that human video data mainly strengthens the robustness of existing skills, "rather than teaching the model new manipulation dynamics". How to make large-scale human videos genuinely transfer new manipulation capabilities remains an open question.
相机视角变化的适应能力有限（Inferred）Limited adaptation to camera viewpoint change (Inferred)
LIBERO-Plus 的 Camera 扰动维度上，VLA-JEPA（63.3%）优于 π₀（13.8%）但仍落后于 π₀-Fast（65.1%），表明对剧烈相机运动的鲁棒性仍有提升空间。这是从 Table 3 数据推断得出，并非作者明确声明。On the Camera perturbation dimension of LIBERO-Plus, VLA-JEPA (63.3%) beats π₀ (13.8%) but still trails π₀-Fast (65.1%), indicating that robustness to drastic camera motion still has room to improve. This is inferred from the Table 3 numbers and is not stated explicitly by the authors.
依赖大型 VLM 骨干（Inferred）Reliance on a large VLM backbone (Inferred)
方法基于 Qwen3-VL + V-JEPA2 encoder，推理成本较高。论文未讨论模型压缩或小模型迁移方案。轻量化部署能力尚未验证。The method builds on Qwen3-VL plus a V-JEPA2 encoder, so inference cost is fairly high. The paper does not discuss model compression or transfer to smaller models, and lightweight deployment remains unverified.
