FASTer: 高效自回归视觉-语言-动作模型

01 动机Motivation

自回归 VLA 面临一个核心矛盾：为了加快推理速度，需要将连续动作序列压缩成尽可能少的 token；但过度压缩会丢失精细的动作细节，导致任务失败。Autoregressive VLA faces a core conflict: to accelerate inference, continuous action sequences must be compressed into as few tokens as possible, yet excessive compression discards fine-grained action detail and leads to task failure.

"The key challenge lies in the reconstruction fidelity vs. inference efficiency trade-off: high compression reduces inference latency but degrades action reconstruction quality, while low compression preserves fidelity but increases token count and slows inference."

现有方法（如 FAST）将动作块经 DCT 变换后量化为离散 token，虽然加快了推理速度，但存在两大缺陷： （1）codebook 利用率低（FAST 仅使用了 48.4% 的 codebook），导致大量表征空间浪费； （2）固定的 DCT 频率截断无法自适应地保留各维度动作信号中最关键的频率成分。 Existing methods such as FAST quantize action chunks into discrete tokens after a DCT transform. This accelerates inference but has two major flaws: (1) low codebook utilization (FAST uses only 48.4% of the codebook), wasting a large part of the representation space; (2) fixed DCT frequency truncation cannot adaptively retain the most critical frequency components of the action signal in each dimension.

FASTer 跨具身形态的整体对比 — **图 1：FASTer 在多具身形态上的综合表现。** FASTer 在单臂操作（LIBERO, Simpler-Bridge）和全身控制（R1Lite）任务上均超越现有最优方法，同时推理延迟大幅降低。单臂任务延迟 112 ms，全身控制延迟 237 ms（π0 FAST 为 1,100–3,000 ms）。**Figure 1: Overall performance of FASTer across embodiments.** FASTer surpasses prior state-of-the-art methods on both single-arm manipulation (LIBERO, Simpler-Bridge) and whole-body control (R1Lite), while sharply reducing inference latency. Single-arm latency is 112 ms and whole-body latency is 237 ms (π0 FAST: 1,100–3,000 ms).

97.9%LIBERO 平均成功率（↑3.7 pp vs. π0 FAST-D）LIBERO average success rate (↑3.7 pp vs. π0 FAST-D)

87.9%Simpler-Bridge 平均成功率（↑11.4 pp vs. π0 FAST-R）Simpler-Bridge average success rate (↑11.4 pp vs. π0 FAST-R)

112 ms单臂推理延迟（vs. FAST 197–556 ms）Single-arm inference latency (vs. FAST 197–556 ms)

100%FASTerVQ codebook 利用率（vs. FAST 48.4%）FASTerVQ codebook utilization (vs. FAST 48.4%)

02 方法Method

FASTer 框架由两部分组成：FASTerVQ 是一个 Transformer 架构的残差向量量化（RVQ）动作分词器，负责将动作块高效编码为离散 token；FASTerVLA 则在预训练的视觉-语言骨干网络上，通过块级自回归解码（block-wise autoregressive decoding）实现快速、高精度的动作生成。The FASTer framework has two parts: FASTerVQ is a Transformer-based residual vector quantization (RVQ) action tokenizer that efficiently encodes action chunks into discrete tokens; FASTerVLA then builds on a pretrained vision-language backbone and achieves fast, high-precision action generation through block-wise autoregressive decoding.

FASTerVQ：神经动作分词器FASTerVQ: a neural action tokenizer

**图 2：FASTerVQ 架构示意。** 动作块被编码为"单通道图像"（single-channel image），通过 Transformer 编码器捕捉全局时空依赖关系，再经三层残差向量量化（RVQ）压缩为少量离散 token。解码器通过时域 ℓ₁ 损失与 DCT 域 ℓ₁ 损失联合优化，确保重建的动作序列在时域与频域均保持高保真度。**Figure 2: FASTerVQ architecture.** An action chunk is encoded as a "single-channel image"; a Transformer encoder captures global spatio-temporal dependencies, and three residual vector quantization (RVQ) layers compress it into a small number of discrete tokens. The decoder is jointly optimized with a temporal ℓ₁ loss and a DCT-domain ℓ₁ loss, so the reconstructed action sequence stays high-fidelity in both the time and frequency domains.

非均匀动作分组（Non-uniform Grouping）Non-uniform Grouping

不同维度的动作（如关节角度、末端执行器位移）具有不同的频率特性与数值分布。 FASTerVQ 采用非均匀分组策略，按维度的分布特性分配不同的量化精度，避免高方差维度与低方差维度共享同一量化网格导致的精度损失。Actions in different dimensions (joint angles, end-effector displacements, and so on) have different frequency characteristics and value distributions. FASTerVQ adopts a non-uniform grouping strategy that allocates different quantization precision according to the distribution of each dimension, avoiding the precision loss caused by high-variance and low-variance dimensions sharing the same quantization grid.

双重重建损失（Dual Reconstruction Loss）Dual Reconstruction Loss

联合使用时域 ℓ₁ 损失（temporal ℓ₁）和 DCT 域 ℓ₁ 损失：前者直接约束动作序列的逐步误差，后者约束频率分量以保留动作的周期性结构。三层 RVQ codebook（大小 4096）实现了 100% 的 codebook 利用率（normalized entropy 0.91）。A temporal ℓ₁ loss (temporal ℓ₁) and a DCT-domain ℓ₁ loss are used jointly: the former directly constrains the step-wise error of the action sequence, the latter constrains frequency components so as to preserve the periodic structure of the action. The three-layer RVQ codebook (size 4096) achieves 100% codebook utilization (normalized entropy 0.91).

FASTerVLA：块级自回归解码FASTerVLA: block-wise autoregressive decoding

FASTerVLA 架构与块级解码 — **图 3：FASTerVLA 架构与块级自回归解码（Block-wise Autoregressive decoding，BAR）。** 每次 forward pass 同时预测 B 个 token（block），将前向传播次数从 N 次降低到 N/B 次。 token 的解码顺序按"先 codebook 层级（粗到细）、后时间步"排列，保证每步解码都具有充分的上下文。间距增广（Spacing Augmentation）防止位置过拟合，轻量级 action expert 共享主干参数，进一步节省计算。**Figure 3: FASTerVLA architecture and Block-wise Autoregressive decoding (BAR).** Each forward pass predicts B tokens (a block) at once, cutting the number of forward passes from N to N/B. Tokens are decoded in the order "codebook level first (coarse to fine), then time step", so every decoding step has sufficient context. Spacing Augmentation prevents positional overfitting, and a lightweight action expert shares the backbone parameters to save further computation.

Block-wise Autoregressive decoding (BAR)

标准自回归解码每步仅生成 1 个 token，对长动作序列（N 个 token）需 N 次 forward pass。 BAR 每步并行预测 B 个 token，forward pass 次数降低至 ⌈N/B⌉，在保持 token 间因果依赖的前提下大幅提升吞吐量。Standard autoregressive decoding emits only 1 token per step, requiring N forward passes for a long action sequence of N tokens. BAR predicts B tokens in parallel per step, lowering the number of forward passes to ⌈N/B⌉ and greatly raising throughput while preserving the causal dependencies among tokens.

Spacing Augmentation

训练时对动作块中的 token 位置施加随机间距扰动，使模型学习对位置偏移鲁棒的表征，避免在推理时因 block 边界与训练时不一致而产生位置过拟合问题。During training, random spacing perturbations are applied to token positions inside an action chunk, so the model learns representations robust to positional shifts and avoids the positional overfitting that arises when block boundaries at inference differ from those seen in training.

03 实验Experiments

在 LIBERO（四个子任务）、Simpler-Bridge（零样本跨具身）和 R1Lite 全身控制 上进行评测，并与 π0、π0 FAST-D/R、OpenVLA 等最强基线对比。推理延迟在 RTX 5090 上测量。 Evaluation is carried out on LIBERO (four sub-tasks), Simpler-Bridge (zero-shot cross-embodiment) and R1Lite whole-body control, and compared against the strongest baselines including π0, π0 FAST-D/R and OpenVLA. Inference latency is measured on an RTX 5090.

LIBERO 基准（Table 1，成功率 %）LIBERO benchmark (Table 1, success rate %)

模型Model	Spatial	Object	Goal	Long	Average
FASTer	98.6	95.4	98.6	97.9	97.9
FASTer w/o BAR	94.8	88.6	98.6	95.4	95.4
π0 FAST-D	96.0	86.8	96.0	94.2	94.2
π0	95.8	85.2	98.8	94.2	94.2

Simpler-Bridge 零样本泛化（Table 1，成功率 %）Simpler-Bridge zero-shot generalization (Table 1, success rate %)

模型Model	Spoon	Carrot	Block	Eggplant	Average
FASTer	91.7	93.3	67.5	99.2	87.9
π0 FAST-D	77.5	88.3	68.3	71.7	76.5
π0	66.7	58.3	58.3	88.3	66.7
OpenVLA	—	—	—	—	29.5

推理延迟（Table 2，RTX 5090）Inference latency (Table 2, RTX 5090)

任务Task	FASTer	π0 FAST	π0
单臂操作（LIBERO）Single-arm manipulation (LIBERO)	112 ms	197–556 ms	176 ms
全身控制（R1Lite）Whole-body control (R1Lite)	237 ms	1,100–3,000 ms	—

跨骨干网络泛化Cross-backbone generalization

跨具身形态的性能对比 — **图 4：跨骨干网络性能对比（LIBERO 基准）。** FASTer 在不同视觉-语言骨干下均显著优于 FAST 基线。以 InternVL3.5-2B 为骨干时，FAST 平均成功率为 79.35%，FASTer 提升至 96.65%（**↑17.3 pp**）。**Figure 4: Performance comparison across backbones (LIBERO benchmark).** FASTer clearly outperforms the FAST baseline under every vision-language backbone. With InternVL3.5-2B as the backbone, FAST reaches an average success rate of 79.35% while FASTer raises it to 96.65% (**↑17.3 pp**).

Codebook 利用率（Table 8，Bridge 数据集）Codebook utilization (Table 8, Bridge dataset)

指标Metric	FAST	FAST+	FASTer
Codebook 大小Codebook size	2048	2048	4096
使用率Utilization	48.4%	57.4%	100%
Normalized Entropy	0.69	0.77	0.91

消融实验（Ablations）Ablations

压缩率与重建质量的权衡 — **图 5：压缩率与重建质量（VRR）的权衡曲线。** FASTerVQ 在所有误差容忍度 σ 下均优于 FAST 及其变体，在物理意义容忍度 σ=10⁻³ 处实现了"near-lossless"动作块重建。**Figure 5: Trade-off curve between compression rate and reconstruction quality (VRR).** FASTerVQ outperforms FAST and its variants at every error tolerance σ, achieving "near-lossless" action-chunk reconstruction at the physically meaningful tolerance σ=10⁻³.

架构消融（Table 6）：TAAE 架构取得最优成功率（97.9%）和最低 ℓ₁ 损失（0.0021）。Architecture ablation (Table 6): the TAAE architecture attains the best success rate (97.9%) and the lowest ℓ₁ loss (0.0021).
Codebook 大小：4096 为最优；8192 时出现 codebook 坍塌（collapse）。Codebook size: 4096 is optimal; at 8192 the codebook collapses.
RVQ 层数：三层残差层配置最为稳定。Number of RVQ layers: the three-residual-layer configuration is the most stable.
BAR 消融：移除 BAR 后成功率从 97.9% 降至 95.4%（LIBERO 平均），验证块级解码对性能的贡献。BAR ablation: removing BAR drops the success rate from 97.9% to 95.4% (LIBERO average), confirming the contribution of block-wise decoding.

04 局限性Limitations

说明：论文未设置独立的 Limitations 章节；以下各点综合了论文中明确陈述的限制（标注"stated"）和基于方法设计合理推断的局限（标注"inferred"）。Note: The paper has no standalone Limitations section; the points below combine limitations explicitly stated in the paper (marked "stated") with limitations reasonably inferred from the method design (marked "inferred").

块级解码要求动作序列长度稳定（inferred）Block-wise decoding requires a stable action sequence length (inferred)

BAR 每步并行预测 B 个 token，要求动作块 token 数量在训练与推理阶段一致。若不同任务或具身形态的动作块长度差异较大，需要重新训练或设计变长策略，限制了跨任务零样本部署的灵活性。BAR predicts B tokens in parallel per step, which requires the token count of an action chunk to be identical at training and inference time. If action-chunk lengths differ substantially across tasks or embodiments, retraining or a variable-length strategy is needed, limiting the flexibility of zero-shot cross-task deployment.

主要在操作任务上验证，移动底座等场景评测有限（stated + inferred）Validated mainly on manipulation tasks, with limited evaluation of mobile-base settings (stated + inferred)

论文的实验以桌面操作（LIBERO、Simpler-Bridge）和全身控制（R1Lite）为主；对移动操控、户外导航等更高自由度任务的泛化性尚未充分验证。The paper's experiments centre on tabletop manipulation (LIBERO, Simpler-Bridge) and whole-body control (R1Lite); generalization to tasks with higher degrees of freedom, such as mobile manipulation and outdoor navigation, has not been fully verified.

大规模分词器训练的计算代价（inferred）Computational cost of large-scale tokenizer training (inferred)

FASTerVQ 的 Transformer 编码器与三层 RVQ 的训练需要覆盖多具身形态的大规模动作数据集；相比 DCT+简单量化的 FAST，其训练成本更高，部署新具身形态时需要重新收集数据并微调分词器。Training the FASTerVQ Transformer encoder and its three RVQ layers requires a large-scale action dataset covering multiple embodiments; compared with FAST, which uses DCT plus simple quantization, its training cost is higher, and deploying a new embodiment requires re-collecting data and fine-tuning the tokenizer.

实时控制场景的延迟下限（inferred）Latency floor for real-time control scenarios (inferred)

尽管 FASTer 在 RTX 5090 上达到 112 ms（单臂）和 237 ms（全身），对于需要 30 Hz 以上控制频率（<33 ms）的高动态任务（如高速抛接、敏捷运动），当前延迟仍存在差距，需要进一步量化或推理优化。Although FASTer reaches 112 ms (single-arm) and 237 ms (whole-body) on an RTX 5090, for highly dynamic tasks that need control frequencies above 30 Hz (<33 ms), such as high-speed throwing and catching or agile locomotion, the current latency still falls short and calls for further quantization or inference optimization.