F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actions

01 动机Motivation

现有 VLA 模型主要依赖"被动的状态到动作映射（reactive state-to-action mappings）"，缺乏对时间演化的建模，在动态场景和长时序任务中表现脆弱。F₁ 提出将视觉前瞻生成引入决策循环，让模型在执行动作前先"想象"未来的视觉状态，从而生成更具前瞻性的动作序列。Existing VLA models rely mainly on "reactive state-to-action mappings", lack any modeling of temporal evolution, and are fragile in dynamic scenes and long-horizon tasks. F₁ brings visual foresight generation into the decision loop, letting the model first "imagine" the future visual state before executing an action, so that it produces more far-sighted action sequences.

"Existing approaches primarily rely on reactive state-to-action mappings, often leading to short-sighted behaviors and poor robustness in dynamic scenes."

范式对比图 — **Figure 1：**"The comparison of varied paradigms for manipulation policies." 左：传统 state→action 的被动映射；右：F₁ 的预测引导逆动力学范式——先生成目标视觉状态，再由动作专家推导实际指令。**Figure 1:** "The comparison of varied paradigms for manipulation policies." Left: the conventional passive state→action mapping; right: the prediction-guided inverse-dynamics paradigm of F₁ — the target visual state is generated first, and the action expert then derives the actual commands.

82.2%真实机器人任务平均成功率（vs. π₀ 65.2%）average success rate on real-robot tasks (vs. π₀ 65.2%)

95.7%LIBERO 平均成功率，全场景榜首average success rate on LIBERO, first across all suites

72.9%SimplerEnv Bridge 平均成功率average success rate on SimplerEnv Bridge

330k预训练轨迹数 · 136 任务 · 5 本体pre-training trajectories · 136 tasks · 5 embodiments

02 方法Method

F₁ 采用 Mixture-of-Transformer (MoT) 架构，由三类专家协同工作：understanding expert 编码语言指令与当前观测，generation expert 预测未来视觉状态，action expert 基于多模态上下文生成动作序列。三者通过 UGA（Understanding→Generation→Action）progressive attention 串联，严格保证因果信息流，防止反向泄漏。F₁ adopts a Mixture-of-Transformer (MoT) architecture in which three kinds of experts cooperate: the understanding expert encodes the language instruction and the current observation, the generation expert predicts the future visual state, and the action expert produces the action sequence from the multimodal context. The three are chained by UGA (Understanding→Generation→Action) progressive attention, which strictly guarantees causal information flow and prevents backward leakage.

F₁ 框架总览 — **Figure 2：**"Overview of F₁ framework. It employs the Mixture-of-Transformer (MoT) architecture comprising three core components: an understanding expert, a generation expert, and an action expert." 三类专家共享 Transformer backbone，通过路由机制和 UGA attention 实现有序信息融合。**Figure 2:** "Overview of F₁ framework. It employs the Mixture-of-Transformer (MoT) architecture comprising three core components: an understanding expert, a generation expert, and an action expert." The three experts share a Transformer backbone and achieve ordered information fusion through the routing mechanism and UGA attention.

视觉前瞻生成（Visual Foresight）Visual Foresight Generation

生成专家采用多尺度残差 VQ-VAE（Residual VQ-VAE）将未来帧分解为从 16×16 到 256×256 的离散 token 序列，再以自回归 next-scale prediction 逐步生成目标视觉状态。预测结果解码为"可视化未来观测"，为动作专家提供前瞻性上下文。The generation expert uses a multi-scale Residual VQ-VAE to decompose future frames into discrete token sequences from 16×16 up to 256×256, and then generates the target visual state step by step with autoregressive next-scale prediction. The prediction is decoded into a "visualized future observation" that supplies far-sighted context to the action expert.

预测引导的逆动力学（Predictive Inverse Dynamics）Predictive Inverse Dynamics

动作专家接收当前观测、语言指令及生成专家产生的预测视觉状态，通过flow matching 预测连续动作序列。与直接 state→action 映射不同，此设计将目标状态显式纳入动作生成，使策略具备更长的时间视野。The action expert takes the current observation, the language instruction and the predicted visual state produced by the generation expert, and predicts a continuous action sequence via flow matching. Unlike a direct state→action mapping, this design writes the target state explicitly into action generation, giving the policy a longer temporal horizon.

Residual VQ-VAE 可视化 — **Figure 3：**"Visualization of Residual VQ-VAE from 16×16 to 256×256 resolution." 从粗粒度到细粒度逐级重建目标图像，展示生成专家的多尺度预测能力。**Figure 3:** "Visualization of Residual VQ-VAE from 16×16 to 256×256 resolution." The target image is reconstructed level by level from coarse to fine, showing the multi-scale prediction ability of the generation expert.

三阶段训练方案（Three-Stage Training Recipe）Three-Stage Training Recipe

Stage I：生成专家对齐。冻结预训练的 understanding expert，单独训练 generation expert 学习视觉预见生成。Stage I: aligning the generation expert. The pre-trained understanding expert is frozen while the generation expert alone is trained to learn visual foresight generation.
Stage II：大规模机器人数据预训练。在约 33 万条多本体轨迹上联合训练全模型，学习跨任务的通用操作策略。Stage II: large-scale pre-training on robot data. The full model is trained jointly on about 330000 multi-embodiment trajectories to learn a general manipulation policy across tasks.
Stage III：特定任务后训练。针对目标机械臂本体进行少量演示数据的微调，完成跨本体适应。Stage III: task-specific post-training. Fine-tuning on a small amount of demonstration data for the target robot-arm embodiment completes cross-embodiment adaptation.

训练损失为两项之和：自回归 next-scale prediction loss（生成专家）+ flow matching action prediction loss（动作专家），以加权系数平衡。The training loss is the sum of two terms: the autoregressive next-scale prediction loss (generation expert) plus the flow matching action prediction loss (action expert), balanced by a weighting coefficient.

03 实验Experiments

F₁ 在三个评测场景下与主流 VLA 基线（π₀、gr00t-N1/N1.5、π₀-Fast、OpenVLA、SpatialVLA 等）进行全面对比：真实机器人任务、LIBERO 模拟基准、SimplerEnv Bridge 基准。此外，在动态环境和长时序任务场景下额外验证鲁棒性。F₁ is compared comprehensively with mainstream VLA baselines (π₀, gr00t-N1/N1.5, π₀-Fast, OpenVLA, SpatialVLA and others) under three evaluation settings: real-robot tasks, the LIBERO simulation benchmark and the SimplerEnv Bridge benchmark. Robustness is additionally verified in dynamic environments and long-horizon task scenarios.

真实机器人任务（Table 1）Real-Robot Tasks (Table 1)

Task	π₀	gr00t-N1	gr00t-N1.5	F₁ (Ours)
Pen	66.7%	46.7%	73.3%	93.3%
Flower	66.7%	33.3%	40.0%	80.0%
Chip	86.7%	33.3%	46.6%	100.0%
Tea (Table)	86.7%	40.0%	73.5%	93.3%
Tea (Shelf)	73.3%	13.3%	26.6%	86.7%
Bread	66.7%	33.3%	53.3%	66.7%
Handover	33.3%	26.7%	60.0%	80.0%
Handover R2H	40.0%	13.3%	40.0%	93.3%
Mixture	66.7%	33.3%	66.7%	66.7%
Average	65.2%	30.4%	53.3%	82.2%

LIBERO 基准（Table 2）LIBERO Benchmark (Table 2)

Method	Spatial SR	Object SR	Goal SR	Long SR	Average SR
Diffusion Policy	78.5%	87.5%	73.5%	64.8%	76.1%
OpenVLA	84.7%	88.4%	79.2%	53.7%	76.5%
SpatialVLA	88.2%	89.9%	78.6%	55.5%	78.1%
π₀	98.0%	96.8%	94.4%	88.4%	94.4%
π₀-Fast	96.4%	96.8%	88.6%	60.2%	85.5%
gr00t-N1	94.4%	97.6%	93.0%	90.6%	93.9%
CoT-VLA	87.5%	91.6%	87.6%	69.0%	83.9%
F₁ (pretrained)	98.2%	97.8%	95.4%	91.3%	95.7%

**Figure 4：**真实机器人实验场景——涵盖笔、花、薯片、茶杯、面包、传递等9类任务，以及 Handover Robot-to-Human 等复杂交互场景。F₁ 在 Handover R2H 上从 π₀ 的 40.0% 提升至 93.3%。**Figure 4:** Real-robot experimental scenes — covering 9 task categories such as pen, flower, chips, teacup, bread and handover, together with complex interaction scenarios such as Handover Robot-to-Human. On Handover R2H, F₁ raises the success rate from 40.0% for π₀ to 93.3%.

消融实验（Ablation Studies）Ablation Studies

消融实验结果 — **Figure 5（消融）：**去除 generation expert 后成功率从 77.5% 骤降至 60.3%；冻结 generation expert（不参与 Stage II/III 训练）降至 73.8%；去除 Stage II 大规模预训练降约 3.3%。最优规划步长为 4 步。**Figure 5 (ablation):** Removing the generation expert makes the success rate plunge from 77.5% to 60.3%; freezing the generation expert (excluding it from Stage II/III training) lowers it to 73.8%; removing the Stage II large-scale pre-training costs about 3.3%. The optimal planning horizon is 4 steps.

去除 generation expert：77.5% → 60.3%（-17.2 pp）Removing the generation expert: 77.5% → 60.3% (-17.2 pp)
冻结 generation expert：77.5% → 73.8%（-3.7 pp）Freezing the generation expert: 77.5% → 73.8% (-3.7 pp)
去除 Stage II 预训练：降低约 3.3 ppRemoving Stage II pre-training: a drop of about 3.3 pp
最优前瞻规划步长：4 步Optimal foresight planning horizon: 4 steps

视觉前瞻样本（Visual Foresight Examples）Visual Foresight Examples

生成的未来图像示例 — **Figure 6：**F₁ generation expert 生成的未来观测图像示例。尽管像素级精度（image token accuracy ≈ 40–45%）有限，但生成图像已足够引导动作专家完成任务进度跟踪，且动作 token 准确率与图像 token 准确率存在显著正相关。**Figure 6:** Examples of future observations generated by the F₁ generation expert. Although pixel-level precision is limited (image token accuracy ≈ 40–45%), the generated images already suffice to guide the action expert in tracking task progress, and action token accuracy is significantly positively correlated with image token accuracy.

04 局限性Limitations

说明：以下局限性均为论文作者在 limitations 章节中明确陈述。Note: All limitations below are explicitly stated by the authors in the limitations section of the paper.

生成专家对细粒度物体细节处理不足The generation expert handles fine-grained object details poorly

由于缺乏大规模生成数据集的预训练，generation expert 在网格纹理、可形变物体等精细视觉结构上表现欠佳，生成图像的视觉保真度有限（image token accuracy 约 40–45%）。尽管如此，这一精度水平已足以为动作决策提供有效的任务级引导。Lacking pre-training on large-scale generative datasets, the generation expert performs poorly on fine visual structures such as grid textures and deformable objects, and the visual fidelity of the generated images is limited (image token accuracy about 40–45%). Even so, this level of precision is already sufficient to provide effective task-level guidance for action decision-making.

可形变物体（deformable objects）建模能力弱Weak modeling capability for deformable objects

论文明确指出 generation expert 在处理可形变物体时存在困难，这与预训练数据分布及 VQ-VAE 离散化方案的局限有关。The paper states explicitly that the generation expert has difficulty with deformable objects, which is related to the pre-training data distribution and to the limitations of the VQ-VAE discretization scheme.

视觉像素级精度与任务成功率脱耦Pixel-level visual precision is decoupled from task success rate

实验表明图像 token 准确率（~40–45%）并非关键瓶颈，任务完成率与之正相关但并不严格依赖像素级还原，说明模型实际上学习的是语义级的任务进度引导，而非精确的像素预测。这一设计选择可能限制对需要高视觉精度操作任务的适用性（作者推断）。Experiments show that image token accuracy (~40–45%) is not the key bottleneck: the task completion rate is positively correlated with it but does not strictly depend on pixel-level reconstruction, indicating that the model actually learns semantic-level guidance on task progress rather than precise pixel prediction. This design choice may limit applicability to manipulation tasks that demand high visual precision (inferred by the authors).