Accelerating Discrete Diffusion VLA to Real-Time Performance
Wenxuan Song*, Jiayi Chen*, Shuai Chen*, Jingbo Wang, Pengxiang Ding, Han Zhao, Yikai Qin, Xinhu Zheng, Donglin Wang, Yan Wang†, Haoang Li† · HKUST(GZ)、ShanghaiTech、上海技物所、AIR Tsinghua、Westlake、Zhejiang UniversityWenxuan Song*, Jiayi Chen*, Shuai Chen*, Jingbo Wang, Pengxiang Ding, Han Zhao, Yikai Qin, Xinhu Zheng, Donglin Wang, Yan Wang†, Haoang Li† · HKUST(GZ), ShanghaiTech, Shanghai Institute of Technical Physics (CAS), AIR Tsinghua, Westlake, Zhejiang University
离散扩散 VLA（dVLA）在多模态对齐上优于流匹配架构，但推理速度远低于机器人实时控制所需的 30 Hz。Fast-dVLA 揭示了 dVLA 隐含的逐块自回归解码趋势，并借此设计分块扩散注意力与 KV 缓存复用机制，配合扩散强迫实现块间并行解码，最终以最低 2.8×、最高 4.1× 的加速比将 dVLA 推进实时域，同时保持 SOTA 级别的成功率。Discrete diffusion VLA (dVLA) outperforms flow-matching architectures on multimodal alignment, yet its inference speed falls far below the 30 Hz required for real-time robot control. Fast-dVLA reveals the block-wise autoregressive decoding tendency implicit in dVLA and exploits it to design block-wise diffusion attention together with a KV cache reuse mechanism, coupled with diffusion forcing for parallel decoding across blocks, ultimately pushing dVLA into the real-time regime at speedups of 2.8× at minimum and 4.1× at maximum while preserving SOTA-level success rates.
离散扩散 VLA（dVLA，如 Dream-VLA、DD-VLA）通过并行迭代去噪输出动作 token，天然继承 VLM 的预训练知识且无需独立动作头；但其双向注意力机制导致每步前向传播均需重新计算 Key-Value（KV）表示，无法复用 KV 缓存，从而产生极低的单次前向效率。当前 dVLA 的执行频率远低于物理机器人所需的 30 Hz 实时控制标准。Discrete diffusion VLAs (dVLA, e.g. Dream-VLA, DD-VLA) emit action tokens through parallel iterative denoising, naturally inheriting the pretrained knowledge of the VLM without needing a separate action head; their bidirectional attention, however, forces the Key-Value (KV) representations to be recomputed at every forward pass, making KV cache reuse impossible and yielding extremely low per-forward efficiency. The execution frequency of current dVLAs stays far below the 30 Hz real-time control standard that physical robots require.
"current dVLAs still suffer from a fundamental limitation. Their inference speed is slow, with an execution frequency that is far below the real-time requirements of physical robotic systems (typically around 30 Hz)."
图 1 速度与成功率权衡。左（同类对比）：在 DD-VLA 和 Dream-VLA 两个基底模型上，Fast-dVLA 分别实现 2.6× 和 3.2× 加速，同时成功率持平或略升，优于 BlockDiff 和 Fast-dLLM 等方案。右（跨范式对比）：Fast-dVLA 在 LIBERO 上同时达到与连续流匹配方法（π₀.₅）相当的频率（~30 Hz）和更高的成功率，兼具 dVLA 的内在优势。Figure 1 Speed vs. success-rate trade-off. Left (within-paradigm comparison): on the two base models DD-VLA and Dream-VLA, Fast-dVLA achieves 2.6× and 3.2× speedup respectively, while success rates stay level or rise slightly, outperforming schemes such as BlockDiff and Fast-dLLM. Right (cross-paradigm comparison): on LIBERO, Fast-dVLA simultaneously attains a frequency comparable to the continuous flow-matching method (π₀.₅) (~30 Hz) and a higher success rate, retaining the intrinsic advantages of dVLA.
4.1×最大推理加速比（DD-VLA on LIBERO）Max. inference speedup (DD-VLA on LIBERO)
30 Hz达到实时控制频率Real-time control frequency reached
96.6%LIBERO 平均成功率（Fast-dVLA + DD-VLA）LIBERO average success rate (Fast-dVLA + DD-VLA)
图 2 四类离散解码范式对比。（a）自回归 VLA：每步一个 token，前向次数多但 forward 快；（b）dVLA：全序列一次并行去噪，forward 次数少但每次 forward 慢；（c）Block Diffusion：分块串行，适中；（d）Fast-dVLA（ours）：forward 次数少且每次 forward 快，综合推理速度最快，同时保留块间并行性。Figure 2 Comparison of four discrete decoding paradigms. (a) Autoregressive VLA: one token per step, many forward passes but each forward is fast; (b) dVLA: parallel denoising of the whole sequence at once, few forward passes but each forward is slow; (c) Block Diffusion: block-wise and sequential, intermediate; (d) Fast-dVLA (ours): few forward passes and each forward is fast, giving the fastest overall inference speed while preserving inter-block parallelism.
02 Method
Fast-dVLA 的核心是将全序列双向注意力替换为分块因果注意力（block-wise causal attention），并结合扩散强迫（diffusion forcing）实现块间并行解码。训练时采用非对称蒸馏从已微调的双向 dVLA 高效迁移能力；推理时设计流水线并行调度算法，平衡解码可靠性与吞吐量。The core of Fast-dVLA is to replace full-sequence bidirectional attention with block-wise causal attention, and to combine it with diffusion forcing so that blocks can be decoded in parallel. At training time, asymmetric distillation efficiently transfers capability from an already fine-tuned bidirectional dVLA; at inference time, a pipelined parallel scheduling algorithm balances decoding reliability against throughput.
3.1 隐式块级自回归趋势的发现3.1 Discovering the Implicit Block-Level Autoregressive Tendency
通过可视化 Dream-VLA 在不同去噪步骤中各 token 位置的解码概率（Figure 3），作者发现：尽管 dVLA 使用双向注意力，其解码过程在宏观上仍呈现从左到右的块级自回归模式——时间序列中靠前的动作块倾向于在更早的去噪迭代中被解码。这一观察源于两个原因：①dVLA 骨干初始化自自回归 VLM，天然保留了自回归特性；②不同时间步的动作存在固有的时序依赖。该发现表明，经过微调的双向 dVLA 可被直接"强制"遵循分块扩散解码方式。By visualizing the decoding probability of every token position across denoising steps in Dream-VLA (Figure 3), the authors find that although dVLA uses bidirectional attention, its decoding process still exhibits, at a macroscopic level, a left-to-right block-level autoregressive pattern — action blocks that come earlier in the temporal sequence tend to be decoded at earlier denoising iterations. This observation has two causes: (i) the dVLA backbone is initialized from an autoregressive VLM and thus naturally retains autoregressive properties; (ii) actions at different time steps carry inherent temporal dependencies. The finding indicates that a fine-tuned bidirectional dVLA can be directly "forced" to follow a block-wise diffusion decoding scheme.
3.2 分块注意力与 KV 缓存复用3.2 Block-Wise Attention and KV Cache Reuse
将长度为 L 的动作 token 序列划分为 N 个等大小的块，每个块仅能 attend 到其之前所有块的 token（因果限制），同一块内 token 可互相 attend（块内双向）。一旦某块解码完成，其 KV 状态即固定不变，后续解码步骤可直接复用缓存的 KV，无需重新计算，彻底解决了双向注意力下 KV 随去噪步变化的问题（Figure 4b 验证了缓存相似度接近 1.0）。An action token sequence of length L is partitioned into N equally sized blocks; each block may attend only to the tokens of all preceding blocks (the causal restriction), while tokens inside the same block attend to one another (bidirectional within a block). Once a block has been decoded, its KV states are frozen, so subsequent decoding steps can reuse the cached KV directly without recomputation, thoroughly resolving the problem that KV vary with the denoising step under bidirectional attention (Figure 4b verifies a cache similarity close to 1.0).
为实现块间并行解码，论文借鉴扩散强迫思想，对不同块赋予单调递增的噪声水平（t₁ < t₂ < ⋯ < tₙ）：靠前的块噪声更少（信息更完整），靠后的块仍高度遮蔽，从而允许模型在精化前序块的同时并发去噪后续块，实现块间并行而不损害时序一致性。To make inter-block parallel decoding possible, the paper borrows the idea of diffusion forcing and assigns monotonically increasing noise levels to the different blocks (t₁ < t₂ < ⋯ < tₙ): earlier blocks carry less noise (more complete information) while later blocks remain heavily masked, which lets the model denoise subsequent blocks concurrently while refining the preceding ones, achieving inter-block parallelism without harming temporal consistency.
直接从头训练代价高昂。论文提出从已任务微调的双向 dVLA（教师）蒸馏 Fast-dVLA（学生）：教师以全局视角（bidirectional attention，看到所有块）预测目标，学生以块因果视角（causal block attention，仅看到前缀块）逼近教师输出，二者结构共享但注意力模式不同——这种"不对称"使蒸馏损失 ℒ_AD 能高效传递教师的整体规划能力。实验显示，ℒ_AD 仅需约 2,000 步即可收敛，是从头训练所需步骤的约 1/10（Figure 8）。蒸馏采用 LoRA（rank=32），仅训练 LoRA 分支以保留骨干的视觉语言预训练知识。Training from scratch is prohibitively expensive. The paper proposes distilling Fast-dVLA (the student) from a task-fine-tuned bidirectional dVLA (the teacher): the teacher predicts targets from a global view (bidirectional attention, seeing all blocks), while the student approximates the teacher output from a block-causal view (causal block attention, seeing only prefix blocks); the two share the same architecture but differ in attention pattern — this "asymmetry" lets the distillation loss ℒ_AD transfer the teacher holistic planning ability efficiently. Experiments show that ℒ_AD converges within only about 2,000 steps, roughly 1/10 of the steps needed to train from scratch (Figure 8). Distillation uses LoRA (rank=32), training only the LoRA branches so that the vision-language pretrained knowledge of the backbone is preserved.
推理时维护一条动态增长的块流水线（Figure 6），区分"半激活"与"全激活"两种状态：当前一块的解码完成比例超过阈值 τ_add 时，新块被引入为半激活状态；当前一块完成比例超过 τ_act 时，新块升为全激活状态。全激活块采用置信度自适应的激进解码策略（按置信度排名，每步至少解码 ⌊剩余 token/n⌋ 个），兼顾吞吐与可靠性。At inference a dynamically growing pipeline of blocks is maintained (Figure 6), distinguishing two states, "half-activated" and "fully activated": when the decoded fraction of the previous block exceeds the threshold τ_add, a new block is introduced in the half-activated state; when that fraction exceeds τ_act, the new block is promoted to fully activated. Fully activated blocks adopt a confidence-adaptive aggressive decoding strategy (ranked by confidence, decoding at least ⌊remaining tokens/n⌋ per step), balancing throughput against reliability.
图 6 Fast-dVLA 流水线并行解码示意。各列代表一个动作块，各行代表一个解码时步 T（从下到上）。深色=已缓存，蓝色=全激活，浅色=半激活，白色=未激活。新块在前驱完成比例超 τ_add = 2/7 时引入，在超 τ_act = 4/7 时升级为全激活。块间并行推进使整体 KV 缓存利用率最大化。Figure 6 Illustration of Fast-dVLA pipelined parallel decoding. Each column stands for an action block and each row for a decoding time step T (bottom to top). Dark = already cached, blue = fully activated, light = half-activated, white = inactive. A new block is introduced once the completion fraction of its predecessor exceeds τ_add = 2/7, and is promoted to fully activated once it exceeds τ_act = 4/7. Advancing blocks in parallel maximizes overall KV cache utilization.
03 Experiments
在 CALVIN（长时序操控）、LIBERO（4 个分套件）、SimplerEnv（真实视觉迁移）三个仿真 benchmark 以及真实双臂 AgileX 平台上评估。基底 dVLA 模型选取 Dream-VLA、DD-VLA（代表型 dVLA）和 UD-VLA（统一多模态 dVLA）。Evaluation covers three simulation benchmarks — CALVIN (long-horizon manipulation), LIBERO (4 sub-suites) and SimplerEnv (real-visual transfer) — as well as a real dual-arm AgileX platform. The base dVLA models are Dream-VLA and DD-VLA (representative dVLAs) plus UD-VLA (a unified multimodal dVLA).
LIBERO：加速策略对比（RQ1）LIBERO: Comparison of Acceleration Strategies (RQ1)
方法Method
Spatial
Goal
Object
Long
Avg.
速度Speed (tok/s)
Dream-VLA
0.902
0.920
0.880
0.720
0.856
98.8 (×1.0)
+ Fast-dLLM
0.884
0.894
0.834
0.702
0.828
183.2 (×1.9)
+ Block Diffusion
0.918
0.904
0.886
0.722
0.858
181.7 (×1.8)
+ Fast-dVLA (ours)
0.912
0.920
0.902
0.746
0.870
313.1 (×3.2)
DD-VLA
0.972
0.986
0.974
0.920
0.963
152.1 (×1.5)
+ Fast-dLLM
0.940
0.952
0.948
0.898
0.935
312.5 (×3.2)
+ Block Diffusion
0.976
0.986
0.972
0.932
0.967
322.1 (×3.3)
+ Fast-dVLA (ours)
0.970
0.988
0.976
0.928
0.966
402.7 (×4.1)
CALVIN ABCD→D：与 SOTA 跨范式对比（RQ2，Table 3）CALVIN ABCD→D: Cross-Paradigm Comparison with SOTA (RQ2, Table 3)
图 7 真实机器人实验结果。在双臂 AgileX 平台上评估三项任务：(a) 传送带拣取（Conveyor Picking）——以每分钟成功抓取次数衡量，Fast-dVLA 达到 13.0 次/分钟，约为 Dream-VLA（7.0）的 2 倍，π₀-FAST（5.0）的 2.6 倍；(b) 蔬菜归类（Vegetables Stowing）——成功率 57.5%，接近 Dream-VLA 的 45.0%，同时将完成时间从 ~15s 压缩至 ~14s；(c) 蔬菜取用（Vegetables Retrieving）——成功率 60.0%，维持在 Dream-VLA 水平，执行频率稳定在 30 Hz，而 π₀-FAST 约 16 Hz、Dream-VLA 约 15 Hz。Figure 7 Real-robot experiment results. Three tasks are evaluated on a dual-arm AgileX platform: (a) Conveyor Picking — measured by the number of successful grasps per minute, Fast-dVLA reaches 13.0 per minute, about two times Dream-VLA (7.0) and 2.6 times π₀-FAST (5.0); (b) Vegetables Stowing — a success rate of 57.5%, close to the 45.0% of Dream-VLA, while compressing the completion time from ~15s to ~14s; (c) Vegetables Retrieving — a success rate of 60.0%, on par with Dream-VLA, with the execution frequency stable at 30 Hz, versus about 16 Hz for π₀-FAST and about 15 Hz for Dream-VLA.
消融实验（RQ4/RQ5）Ablation Studies (RQ4/RQ5)
训练效率方面，非对称蒸馏（ℒ_AD）从微调权重出发，仅需 2,000 步即可收敛，约为从微调权重继续用 ℒ_BD 训练所需的 1/5，是从头训练步数的约 1/10（Figure 8）。On training efficiency, asymmetric distillation (ℒ_AD) starts from the fine-tuned weights and converges within only 2,000 steps, roughly 1/5 of what is needed when training continues from the same fine-tuned weights with ℒ_BD, and about 1/10 of the step count of training from scratch (Figure 8).
块大小方面，将块大小设为动作维度的倍数（本文为 7）比随机块大小在 LIBERO-Long 上平均成功率提高约 1.4 个百分点（74.7% vs 73.3%），且加速比更高（4.01× vs 3.95×）（Table 5）。On block size, setting the block size to a multiple of the action dimension (7 in this paper) raises the average success rate on LIBERO-Long by about 1.4 percentage points over random block sizes (74.7% vs 73.3%), with a higher speedup as well (4.01× vs 3.95×) (Table 5).
置信度阈值 τ_conf 方面，设为 0.5 时达到速度与成功率最优平衡，在 UD-VLA 上实现 2.8× 加速而成功率仅下降 ~2%（Figure 9）。On the confidence threshold τ_conf, a value of 0.5 attains the best balance between speed and success rate, delivering a 2.8× speedup on UD-VLA at a success-rate drop of only ~2% (Figure 9).
04 Limitations
Note:论文无独立的 Limitations 章节。以下局限性均为从论文设计与实验描述中推断（inferred from the design），非作者明确陈述。The paper has no standalone Limitations section. The limitations below are all inferred from the design and experimental descriptions of the paper (inferred from the design), not explicitly stated by the authors.
块大小需手工对齐动作维度Block size must be hand-aligned with the action dimension
Fast-dVLA 的最优块大小需为动作维度的整数倍（UD-VLA 设为 32 的倍数，DD-VLA/Dream-VLA 设为 7）。不同模型或任务需重新搜索超参，自动化块粒度选择尚未解决（inferred）。The optimal block size of Fast-dVLA must be an integer multiple of the action dimension (a multiple of 32 for UD-VLA, and 7 for DD-VLA/Dream-VLA). Different models or tasks require re-searching this hyperparameter, and automatic selection of block granularity remains unsolved (inferred).
蒸馏依赖已微调的双向 dVLADistillation relies on an already fine-tuned bidirectional dVLA
Fast-dVLA 的非对称蒸馏以双向 dVLA 的微调权重为教师起点；对于全新任务或无法获得已微调模型的场景，需额外先做标准微调，增加了准备成本（inferred）。Asymmetric distillation in Fast-dVLA takes the fine-tuned weights of a bidirectional dVLA as the teacher starting point; for entirely new tasks, or scenarios where no fine-tuned model is obtainable, an additional round of standard fine-tuning must be done first, which raises the preparation cost (inferred).
真实世界任务范围有限Limited scope of real-world tasks
真实机器人实验仅覆盖 3 类桌面操控任务（传送带拣取、蔬菜归类、蔬菜取用），所用 AgileX 双臂平台与工业环境存在差距；长时序、接触丰富或高动态任务的实时性能尚未验证（inferred）。The real-robot experiments cover only 3 categories of tabletop manipulation tasks (conveyor picking, vegetables stowing, vegetables retrieving), and the AgileX dual-arm platform used still differs from industrial environments; real-time performance on long-horizon, contact-rich or highly dynamic tasks has not yet been verified (inferred).
SimplerEnv 部分任务成功率低于基线Success rate below the baseline on some SimplerEnv tasks
在 SimplerEnv 的"Carrot on Plate"和"Eggplant in Basket"任务上，Fast-dVLA + Dream-VLA 的成功率（54.1% 和 54.1%）低于 Dream-VLA 原始值（45.8% 和 100.0%，其中"Eggplant"下降幅度较大），说明加速策略在特定任务上存在成功率代价（stated in Table 4）。On the SimplerEnv tasks "Carrot on Plate" and "Eggplant in Basket", the success rates of Fast-dVLA + Dream-VLA (54.1% and 54.1%) fall below the original values of Dream-VLA (45.8% and 100.0%, with "Eggplant" dropping considerably), showing that the acceleration strategy carries a success-rate cost on specific tasks (stated in Table 4).
