Fast-WAM: Do World Action Models Need Test-time Future Imagination?
推理时去掉未来预测，训练时保留视频联合训练——速度提升4倍，性能不降Drop future prediction at inference, keep video co-training at training — 4× faster with no performance loss
Tianyuan Yuan, Zibin Dong, Yicheng Liu, Hang Zhao
World Action Models（WAMs）通常在推理时生成未来观测帧，再根据生成的未来帧预测动作。Fast-WAM 提出解耦这两个因素：训练时保留视频联合训练目标，推理时直接跳过未来帧生成，仅凭当前观测的 latent 世界表征一次 forward pass 输出动作。在 LIBERO 和 RoboTwin 2.0 等模拟基准，以及真实世界折叠毛巾任务上，Fast-WAM 实现了与 imagine-then-execute 范式相当甚至更优的性能，推理延迟仅需 190ms，比同类方法快 4× 以上。World Action Models (WAMs) typically generate future observation frames at inference time and then predict actions conditioned on the generated future. Fast-WAM proposes to disentangle these two factors: the video co-training objective is kept during training, while future-frame generation is skipped entirely at inference, so actions are produced in a single forward pass from the latent world representation of the current observation alone. On simulation benchmarks such as LIBERO and RoboTwin 2.0, as well as a real-world towel-folding task, Fast-WAM matches or even surpasses the imagine-then-execute paradigm while requiring only 190ms of inference latency, more than 4× faster than comparable methods.
World Action Models机器人操作Robot Manipulation视频预测Video Prediction扩散策略Diffusion Policyflow matchingtest-time inferenceimitation learningMixture-of-Transformer
01动机Motivation
现有 World Action Models 把训练时的视频预测目标与推理时的未来帧生成捆绑在一起，无法区分二者各自的贡献。推理时的迭代去噪带来极高延迟，阻碍了实际部署。Existing World Action Models entangle the video prediction objective used during training with future-frame generation at inference time, making it impossible to separate their individual contributions. The iterative denoising performed at inference incurs very high latency and hinders practical deployment.
"It remains unclear whether explicit future imagination is actually necessary for strong action performance... existing WAM systems typically entangle these two factors, making it difficult to determine which one is actually responsible for the observed gains."
图1：三种 WAM 范式对比。
(A) Joint-modeling WAMs 同时对未来视频 token 与动作 token 去噪；
(B) Causal WAMs 先生成未来观测，再以生成结果为条件预测动作；
(C) Fast-WAM 在训练时保留视频联合训练，在推理时移除显式未来生成，仅凭当前帧 latent 特征一次 forward pass 输出动作。Figure 1: Comparison of three WAM paradigms. (A) Joint-modeling WAMs denoise future video tokens and action tokens simultaneously; (B) Causal WAMs first generate future observations and then predict actions conditioned on the generated result; (C) Fast-WAM keeps video co-training during training but removes explicit future generation at inference, producing actions in a single forward pass from the latent features of the current frame.
190msFast-WAM 推理延迟Fast-WAM inference latency
4×+相比 imagine-then-execute 的速度提升Speedup over imagine-then-execute
91.8%RoboTwin 2.0 平均成功率（无 embodied pretraining）RoboTwin 2.0 average success rate (no embodied pretraining)
97.6%LIBERO 平均成功率（无 embodied pretraining）LIBERO average success rate (no embodied pretraining)
研究的核心假设是：视频预测在 WAM 中的主要价值来自训练时改善世界表征，而非推理时提供未来观测。为验证这一假设，作者构建了受控变体——Fast-WAM-Joint、Fast-WAM-IDM 以及移除视频联合训练的消融版本——进行系统对比。The central hypothesis of this work is that the main value of video prediction in WAMs comes from improving world representations during training rather than from providing future observations at inference. To test it, the authors build controlled variants — Fast-WAM-Joint, Fast-WAM-IDM, and an ablated version with video co-training removed — and compare them systematically.
02方法Method
Fast-WAM 基于 Mixture-of-Transformer（MoT）架构，由视频 Diffusion Transformer（DiT）与动作专家 DiT 通过 shared attention 组合而成。训练时三类 token 同时参与：当前帧的 clean latent token、未来帧的 noisy video token（仅训练用）、动作 token。推理时仅保留当前帧 latent，单次前向传播直接输出动作。Fast-WAM builds on a Mixture-of-Transformer (MoT) architecture that combines a video Diffusion Transformer (DiT) with an action expert DiT through shared attention. Three kinds of tokens participate during training: clean latent tokens of the current frame, noisy video tokens of future frames (training only), and action tokens. At inference only the current-frame latent is kept, and actions are emitted directly in a single forward pass.
图2：Fast-WAM 模型架构。
(a) 模型整体设计，包含视频 DiT 与动作专家 DiT 的 shared attention 结构；
(b) 结构化 attention mask 控制三类 token 之间的信息流：clean observation token 可被所有 token 关注；noisy future video token 仅在训练时存在；action token 只能看到 observation token，防止未来信息提前泄露。训练目标为 flow matching 损失：
L = L_act + λ · L_vid，同时优化动作生成与视频预测。Figure 2: Fast-WAM model architecture. (a) The overall design, containing the shared-attention structure that links the video DiT and the action expert DiT; (b) A structured attention mask controls the information flow among the three token types: clean observation tokens can be attended to by all tokens; noisy future video tokens exist only during training; action tokens can only see observation tokens, which prevents future information from leaking early. The training objective is a flow matching loss, L = L_act + λ · L_vid, jointly optimizing action generation and video prediction.
训练阶段：视频联合训练的关键作用Training stage: the crucial role of video co-training
训练时，模型同时优化动作 flow matching 损失（L_act）和视频 flow matching 损失（L_vid），以系数 λ 加权平衡。视频预测目标驱使模型学习物理上有意义的世界表征，即使在推理时这些表征不再依赖未来帧生成也能保留其有效性。Attention mask 的设计保证 action token 不能看到 noisy future token，从而动作预测的性能不依赖于测试时的视频生成质量。During training the model jointly optimizes the action flow matching loss (L_act) and the video flow matching loss (L_vid), balanced by the weighting coefficient λ. The video prediction objective drives the model to learn physically meaningful world representations, which remain effective even when inference no longer relies on future-frame generation. The attention mask is designed so that action tokens cannot see noisy future tokens, so the quality of action prediction does not depend on the quality of video generation at test time.
推理阶段：单次前向传播直接输出动作Inference stage: actions straight from a single forward pass
推理时，"only the clean first-frame latent tokens are retained and passed through the video backbone once to produce latent world features for the action expert."这意味着完全跳过迭代式视频去噪，推理延迟从 810ms（Fast-WAM-IDM）降至 190ms，实现超过 4× 的加速，同时保持相当的动作生成质量。At inference, "only the clean first-frame latent tokens are retained and passed through the video backbone once to produce latent world features for the action expert." Iterative video denoising is therefore skipped entirely, cutting inference latency from 810ms (Fast-WAM-IDM) down to 190ms, a speedup of more than 4× while retaining comparable action generation quality.
图3：结构化 attention mask 设计。
训练时（左），action token 仅能 attend 到 clean observation token，避免从 noisy future video token 获取捷径信息；推理时（右），noisy future video token 整列被移除，模型结构天然适配，无需额外设计。Figure 3: Design of the structured attention mask. During training (left), action tokens may attend only to clean observation tokens, which keeps them from taking shortcut information out of noisy future video tokens; at inference (right), the entire column of noisy future video tokens is removed, a change the architecture accommodates naturally with no extra design.
受控变体设计Design of the controlled variants
Fast-WAM-Joint
未来视频 token 与动作 token 同时进行去噪，shared attention 使两类 token 互相关注。推理时仍需运行视频去噪，延迟较高。用于研究 joint denoising 对性能的影响。Future video tokens and action tokens are denoised simultaneously, with shared attention letting the two kinds of tokens attend to each other. Video denoising still has to be run at inference, so latency stays high. Used to study the effect of joint denoising on performance.
Fast-WAM-IDM
先完整生成未来视频帧（Causal 范式），再将生成结果作为条件输入动作预测模块。推理延迟约 810ms。用于研究显式未来生成对动作性能的作用。Future video frames are first generated in full (the causal paradigm), and the generated result is then fed as a condition into the action prediction module. Inference latency is about 810ms. Used to study the role of explicit future generation in action performance.
03实验Experiments
实验涵盖两个模拟基准（LIBERO、RoboTwin 2.0）和真实世界折叠毛巾任务（Galaxea R1 Lite 平台）。所有 Fast-WAM 变体均不使用 embodied pretraining，而部分强基线方法使用了大规模具身预训练数据。The experiments cover two simulation benchmarks (LIBERO, RoboTwin 2.0) and a real-world towel-folding task (on the Galaxea R1 Lite platform). All Fast-WAM variants are trained without embodied pretraining, whereas some of the strong baselines use large-scale embodied pretraining data.
真实世界：折叠毛巾任务与推理延迟Real world: towel-folding task and inference latency
图4：真实世界性能与推理延迟对比。
左图展示各方法在折叠毛巾任务上的成功率与完成时间；右图为推理延迟柱状图。Fast-WAM 仅需 190ms 推理延迟，远低于 Fast-WAM-IDM 的 810ms。移除视频联合训练的消融变体成功率仅 10%，揭示视频 co-training 对真实世界性能的关键重要性。Figure 4: Real-world performance and inference latency. The left panel shows the success rate and completion time of each method on the towel-folding task; the right panel is a bar chart of inference latency. Fast-WAM needs only 190ms of inference latency, far below the 810ms of Fast-WAM-IDM. The ablated variant with video co-training removed reaches a success rate of only 10%, revealing how critical video co-training is for real-world performance.
消融分析Ablation analysis
受控实验给出一致结论：去掉视频联合训练引发的性能下降，远大于 Fast-WAM 与 imagine-then-execute 变体之间的差异。具体来说：The controlled experiments give a consistent conclusion: the performance drop caused by removing video co-training is far larger than the difference between Fast-WAM and the imagine-then-execute variants. Specifically:
RoboTwin：移除视频 co-training → 性能从 91.8% 跌至 83.8%（-8.0 pp）；Fast-WAM vs Fast-WAM-Joint 差距仅 1.2 pp。RoboTwin: removing video co-training → performance falls from 91.8% to 83.8% (-8.0 pp); the gap between Fast-WAM and Fast-WAM-Joint is only 1.2 pp.
LIBERO：移除视频 co-training → 平均从 97.6% 跌至 93.5%（-4.1 pp），Spatial 子集从 98.2% 跌至 89.2%（-9.0 pp）。LIBERO: removing video co-training → the average falls from 97.6% to 93.5% (-4.1 pp), and the Spatial suite from 98.2% to 89.2% (-9.0 pp).
真实世界：移除视频 co-training → 成功率骤降至 10%，而 Fast-WAM 三个变体（Fast-WAM、Fast-WAM-Joint、Fast-WAM-IDM）之间差距极小。Real world: removing video co-training → the success rate collapses to 10%, while the three Fast-WAM variants (Fast-WAM, Fast-WAM-Joint, Fast-WAM-IDM) differ only marginally among themselves.
上述结果有力支持核心论点："the main value of video prediction in WAMs may lie in improving world representations during training rather than generating future observations at test time."These results strongly support the central claim: "the main value of video prediction in WAMs may lie in improving world representations during training rather than generating future observations at test time."
04局限性Limitations
Note:以下局限性部分来自论文明确陈述（stated），部分为从方法设计中推断（inferred）。Some of the limitations below are stated explicitly in the paper (stated); others are inferred from the method design (inferred).
规模化预训练的影响尚未研究（stated）The effect of large-scale pretraining has not been studied (stated)
论文明确指出："an important direction for future work is to study the effect of larger-scale pretraining data and model scaling on this design." 当前实验未涉及大规模具身预训练，无法确认结论在更大规模下是否依然成立。The paper states explicitly: "an important direction for future work is to study the effect of larger-scale pretraining data and model scaling on this design." The current experiments involve no large-scale embodied pretraining, so it cannot be confirmed whether the conclusions still hold at larger scale.
仅考虑单动作 chunk 生成，省略外层自回归循环（stated）Only single action-chunk generation is considered; the outer auto-regressive loop is omitted (stated)
论文明确提到"omit the outer auto-regressive loop for simplicity and controlled comparison"，实际部署中需要完整的序列控制流，本文尚未覆盖该场景。The paper explicitly mentions that it will "omit the outer auto-regressive loop for simplicity and controlled comparison"; real deployment needs the complete sequential control loop, a setting this work does not yet cover.
仅用单帧观测作为条件（inferred）Conditioning on a single observation frame only (inferred)
推理时仅使用第一帧的 clean latent token，对于需要多帧历史的任务（如部分遮挡、长时依赖场景）可能不足，而 imagine-then-execute 变体理论上能利用生成的未来帧进行多步规划。Inference uses only the clean latent tokens of the first frame, which may be insufficient for tasks that require multi-frame history (such as partial occlusion or long-horizon dependency scenarios), whereas imagine-then-execute variants could in principle exploit the generated future frames for multi-step planning.
视频生成质量本身未深入评估（inferred）Video generation quality itself is not evaluated in depth (inferred)
论文关注的核心是动作性能与推理速度，对视频 DiT 生成的未来帧质量（保真度、一致性）未做定量评估，无法判断生成质量与表征质量之间的关联。The paper centers on action performance and inference speed, and gives no quantitative evaluation of the quality (fidelity, consistency) of the future frames produced by the video DiT, so the relationship between generation quality and representation quality cannot be judged.
