UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent

01 动机Motivation

Vision-Language-Action (VLA) 模型借助大规模预训练 VLM 的语义知识，大幅提升了机器人策略的泛化能力。然而，现有 VLM 以高层语义理解为主，对低层视觉特征（距离、尺寸、空间关系）理解不足，而这些恰是机器人精细操作任务所必需的。Vision-Language-Action (VLA) models exploit the semantic knowledge of large-scale pre-trained VLMs to greatly improve the generalization of robot policies. However, existing VLMs focus mainly on high-level semantic understanding and understand low-level visual features (distance, size, spatial relations) poorly, yet these are exactly what fine-grained robotic manipulation tasks require.

"VLMs often focus on high-level semantic content and neglect low-level features, limiting their ability to capture detailed visual and spatial information. These aspects, which are crucial for robotic control tasks, remain underexplored in existing pre-training paradigms."

UP-VLA overview teaser — **图 1：**UP-VLA 同时接受 multi-modal understanding 和 future prediction 两个预训练目标。左侧展示高层语义理解（VQA、指令跟随），右侧展示预测未来帧以建立低层空间感知；两者共同驱动 action 生成。**Figure 1.** UP-VLA simultaneously adopts two pre-training objectives, multi-modal understanding and future prediction. The left side shows high-level semantic understanding (VQA, instruction following); the right side shows predicting future frames to build low-level spatial perception; together they drive action generation.

Comparison of VLA paradigms — **图 2：**三类 VLA 范式对比。纯 VLM-based VLA 语义理解强但空间感知弱；纯 prediction-based 方法空间感知强但缺乏语义泛化；UP-VLA 结合两者优势，在仿真与真实操作任务上均表现最优。**Figure 2.** Comparison of three VLA paradigms. Purely VLM-based VLA is strong in semantic understanding but weak in spatial perception; purely prediction-based methods have strong spatial perception but lack semantic generalization; UP-VLA combines the advantages of both and performs best on both simulated and real manipulation tasks.

4.08Calvin ABC→D Avg. Length（UP-VLA）Calvin ABC→D Avg. Length (UP-VLA)

+33%相对 GR-1 (3.06) 的提升幅度Improvement over GR-1 (3.06)

80%真实机械臂已见任务成功率Success rate on seen tasks, real robot arm

58%未见物体 / 精细操作泛化成功率Generalization success rate on unseen objects / precise operations

02 方法Method

UP-VLA 以 Phi-1.5（1.5B 参数）作为语言骨干，将 CLIP-ViT（连续 token，用于理解）和 VQ-GAN（离散 token，用于预测）双路图像编码器的输出统一送入同一 LLM，通过三个互补目标联合训练：多模态理解（MMU）、未来图像预测（PRE）以及动作学习（ACT），模型初始化自 Show-o（1.3B，512×512）。UP-VLA adopts Phi-1.5 (1.5B parameters) as its language backbone and feeds the outputs of a dual-path image encoder — CLIP-ViT (continuous tokens, for understanding) and VQ-GAN (discrete tokens, for prediction) — into a single shared LLM, training jointly with three complementary objectives: multi-modal understanding (MMU), future image prediction (PRE) and action learning (ACT); the model is initialized from Show-o (1.3B, 512×512).

UP-VLA architecture overview — **图 3：**UP-VLA 整体架构。蓝色路径：CLIP-ViT 编码连续视觉 token，经 MLP 投影后与语言 token 一起喂入 LLM，完成 VQA / 语义理解；橙色路径：VQ-GAN 编码离散图像 token，LLM 在给定当前观测与任务指令的条件下自回归预测下一帧的离散 token；绿色路径：MAP（单层 attention）+ 线性层将 LLM 输出映射为机器人动作（6-DoF 末端位移 + 夹爪开合）。三路目标共享同一 LLM 权重。**Figure 3.** Overall architecture of UP-VLA. Blue path: CLIP-ViT encodes continuous visual tokens which, after MLP projection, are fed into the LLM together with language tokens to perform VQA / semantic understanding. Orange path: VQ-GAN encodes discrete image tokens, and the LLM autoregressively predicts the discrete tokens of the next frame conditioned on the current observation and the task instruction. Green path: a MAP (single-layer attention) plus a linear layer maps the LLM output to robot actions (6-DoF end-effector displacement + gripper open/close). The three objectives share the same LLM weights.

Unified prompting and attention mechanism — **图 4：**统一 prompt 与 attention 机制示意图。MMU 任务时，图像 token 彼此可见并预测语言 token；PRE 任务时，语言 token 条件化未来离散图像 token 的生成；ACT 任务时，两路 token 联合驱动动作输出，并同步预测未来帧。**Figure 4.** Illustration of the unified prompting and attention mechanism. In the MMU task, image tokens are mutually visible and predict language tokens; in the PRE task, language tokens condition the generation of future discrete image tokens; in the ACT task, both token paths jointly drive the action output while future frames are predicted at the same time.

三个训练目标Three Training Objectives

Multi-modal Understanding (ℒ_MMU)

给定图像连续 token u 和已生成语言 token，最大化下一个语言 token 的对数似然：Given the continuous image tokens u and the already generated language tokens, maximize the log-likelihood of the next language token:

ℒ_MMU = Σ_i log p_θ(l_i | u, l₁,…,l_i-1)

预训练数据：LLaVa-tuning-665k（665k 图文对），涵盖 VQA、描述、推理等任务，赋予模型丰富语义知识。Pre-training data: LLaVa-tuning-665k (665k image-text pairs), covering VQA, captioning, reasoning and other tasks, endowing the model with rich semantic knowledge.

Future Visual Prediction (ℒ_PRE)

以语言指令和当前观测为条件，逐位预测未来帧的 VQ-GAN 离散 token，使用 cross-entropy 损失：Conditioned on the language instruction and the current observation, predict the VQ-GAN discrete tokens of the future frame position by position, using a cross-entropy loss:

ℒ_PRE = Σ_j log p_θ(v'_j | l, v₁,…,v_M)

预训练数据：Bridge 数据集（25k 机械臂演示），强迫模型学习动作执行后的视觉后果，建立低层空间先验。Pre-training data: the Bridge dataset (25k robot-arm demonstrations), forcing the model to learn the visual consequences of executed actions and to establish low-level spatial priors.

Action Learning (ℒ_ACT) — 联合目标Joint Objective

在动作数据上同时预测下一帧图像和机器人动作，合并损失为：On action data, the next frame image and the robot action are predicted at the same time, with the combined loss:

ℒ = λ₁ℒ_MMU + λ₂ℒ_PRE + λ₃ℒ_ACT

其中where ℒ_ACT = Σ‖â_pos − a_pos‖²₂ + BCE(â_end, a_end)，分别约束末端执行器 6-DoF 相对位移（MSE）和夹爪开合状态（BCE）。, constraining respectively the 6-DoF relative displacement of the end-effector (MSE) and the gripper open/close state (BCE).

模型生成场景描述（VQA 自问自答）后将其拼接到 prompt 中，为动作推断提供明确的语义上下文，进一步利用 MMU 能力。训练 20k steps，batch size 64，前 1k steps 线性 warmup。After the model generates a scene description (self-asked, self-answered VQA), it is concatenated into the prompt to provide explicit semantic context for action inference, further exploiting the MMU capability. Training runs for 20k steps with batch size 64 and a linear warmup over the first 1k steps.

真实机器人部署Real-Robot Deployment

使用 Franka-Emika Panda 机械臂，收集 2k+ 演示（6 项技能，人工远程操控 + 脚本策略），涵盖抓取、摆放、插线等任务；测试分为已见场景、未见物体、精细操作三类，各执行 20 次。A Franka-Emika Panda arm is used and 2k+ demonstrations are collected (6 skills, human teleoperation + scripted policies), covering grasping, placing, cable plugging and other tasks; testing is split into three categories — seen scenes, unseen objects and precise operations — each executed 20 times.

Evaluation environments — **图 5：**评估环境可视化。左：Calvin 仿真环境，包含 ABC→D 和 ABCD→D 两种泛化设置；右：Franka 真实机械臂任务，包含已见任务（seen tasks）、未见物体（unseen objects）和精细操作（precise operations）三类。**Figure 5.** Visualization of the evaluation environments. Left: the Calvin simulation environment, with two generalization settings, ABC→D and ABCD→D. Right: real Franka robot-arm tasks, in three categories — seen tasks, unseen objects and precise operations.

03 实验Experiments

在仿真基准 Calvin（ABC→D / ABCD→D）和真实 Franka 机械臂任务上与 RT-1、Robo-Flamingo、GR-1 等强基线对比，评测指标为平均任务完成长度（Avg. Length，最高 5）和操作成功率。UP-VLA is compared against strong baselines such as RT-1, Robo-Flamingo and GR-1 on the simulation benchmark Calvin (ABC→D / ABCD→D) and on real Franka robot-arm tasks; the evaluation metrics are the average task completion length (Avg. Length, maximum 5) and the manipulation success rate.

Calvin ABC→D 仿真基准Simulation Benchmark

方法Method	类型Type	Avg. Len ↑	Task 1	Task 2	Task 3	Task 4	Task 5
RT-1	other	0.90	0.533	0.222	0.094	0.038	0.013
Diffusion Policy*	other	0.56	0.402	0.123	0.026	0.008	0.000
3D Diffuser Actor	other	3.35	0.938	0.803	0.662	0.533	0.412
3D-VLA	VLA	0.71	0.447	0.163	0.081	0.016	0.000
UP-VLA-RT-2*	VLA	1.44	0.612	0.389	0.236	0.138	0.062
Robo-Flamingo	VLA	2.47	0.824	0.619	0.466	0.331	0.235
Uni-Pi	Prediction	0.92	0.560	0.160	0.080	0.080	0.040
SuSIE	Prediction	2.69	0.870	0.690	0.490	0.380	0.260
GR-1	Prediction	3.06	0.854	0.712	0.596	0.497	0.401
UP-VLA-phi-w/o-mmu*	Prediction	3.13	0.844	0.705	0.604	0.520	0.430
UP-VLA	Prediction&VLA	4.08	0.928	0.865	0.815	0.769	0.699

Calvin ABCD→D 仿真基准Simulation Benchmark

方法Method	类型Type	Avg. Len ↑	Task 1	Task 2	Task 3	Task 4	Task 5
RT-1	other	2.45	0.844	0.617	0.438	0.323	0.227
Robo-Flamingo	VLA	4.09	0.964	0.896	0.824	0.740	0.660
GR-1	Prediction	4.21	0.949	0.896	0.844	0.789	0.731
UP-VLA	Prediction&VLA	4.42	0.962	0.921	0.879	0.842	0.812

消融实验（Calvin ABC→D + 真实机械臂）Ablation Study (Calvin ABC→D + Real Robot Arm)

消融变体Ablated Variant	ABC→D Avg.Len ↑	真实已见任务Real, seen tasks ↑	真实未见物体Real, unseen objects ↑
w/o MMU	3.89	0.85	0.20
w/o Bridge-Pretrain	2.74	0.65	0.30
w/o Prediction	1.44	0.65	0.35
w/o MMU-Condition	3.99	0.80	0.50
Full UP-VLA	4.08	0.80	0.58

消融结果表明：去掉 visual prediction（Avg. Len 骤降至 1.44）和 Bridge 预训练（降至 2.74）影响最大，说明低层空间预测目标是模型核心收益来源。MMU-Condition（将自生成场景描述拼接入 prompt）对未见物体的泛化贡献显著（从 0.50 → 0.58）。The ablation results show that removing visual prediction (Avg. Len drops sharply to 1.44) and the Bridge pre-training (down to 2.74) has the largest impact, indicating that the low-level spatial prediction objective is the core source of the model's gains. MMU-Condition (concatenating the self-generated scene description into the prompt) contributes markedly to generalization on unseen objects (from 0.50 → 0.58).

真实操作与视觉预测可视化Real-World Manipulation and Visual Prediction Visualizations

VQA results and predicted future images — **图 7：**左侧展示 VQA 理解结果（模型能正确识别场景中物体的位置、颜色与相对关系）；右侧展示 future prediction 生成的未来帧，模型能预测机械臂抓取动作后的视觉状态，验证了 PRE 目标赋予的空间推理能力。**Figure 7.** Left: VQA understanding results (the model correctly identifies the position, color and relative relations of objects in the scene). Right: future frames generated by future prediction; the model can predict the visual state after the robot arm's grasping action, validating the spatial reasoning ability conferred by the PRE objective.

Real-world manipulation results — **图 6：**真实 Franka 机械臂操作成功率汇总。已见任务成功率 0.80，未见物体泛化 0.58，精细操作（插线、小物体抓取）0.58，均明显优于消融基线。**Figure 6.** Summary of manipulation success rates on the real Franka robot arm. Success rate on seen tasks 0.80, generalization to unseen objects 0.58, precise operations (cable plugging, small-object grasping) 0.58 — all clearly better than the ablation baselines.

04 局限性Limitations

Note：论文未设独立 "Limitations" 小节；以下各点来自结论与结果讨论，均为作者明确指出（stated）的内容。Note: The paper has no dedicated "Limitations" section; the points below are drawn from the conclusion and the discussion of results, and are all explicitly stated by the authors.

特定物体识别准确率不稳定Unstable recognition accuracy for specific objects

由于"数据规模和骨干网络的限制（constraints in data scale and backbone）"，模型对特定物体的识别有时不准确，导致 VQA 质量参差不齐，进而影响 MMU-Condition 的效果。Because of "constraints in data scale and backbone", the model's recognition of specific objects is sometimes inaccurate, making VQA quality uneven and in turn degrading the effectiveness of MMU-Condition.

预测帧中背景颜色伪影Background color artifacts in predicted frames

在 Calvin D 环境中，预测的未来帧有时出现与当前输入帧背景颜色不一致的颜色伪影，原因是训练数据（Bridge 数据集）的视觉分布与测试环境存在偏差，模型尚未充分泛化到新场景的视觉风格。In the Calvin D environment, the predicted future frames sometimes exhibit color artifacts inconsistent with the background color of the current input frame, because the visual distribution of the training data (the Bridge dataset) is biased with respect to the test environment and the model has not yet generalized sufficiently to the visual style of the new scene.

视觉生成预训练数据不足Insufficient pre-training data for visual generation

当前 Bridge 预训练数据（25k 演示）规模有限，导致视觉预测在多样场景下不够鲁棒，限制了 PRE 目标在更广泛真实部署中的收益上限。未来引入更大规模、更多样的机器人视频数据有望突破此瓶颈。The current Bridge pre-training data (25k demonstrations) is limited in scale, making visual prediction insufficiently robust across diverse scenes and capping the benefit of the PRE objective in broader real-world deployment. Introducing larger-scale and more diverse robot video data in the future is expected to break this bottleneck.

模型规模与计算成本Model scale and computational cost

（inferred from design）采用双路图像编码器（CLIP-ViT + VQ-GAN）加上自回归未来帧预测，推理时计算量大于单路 VLA；在实时高频控制场景下延迟较高，论文未给出推理速度的具体数据。(inferred from design) With a dual-path image encoder (CLIP-ViT + VQ-GAN) plus autoregressive future-frame prediction, inference requires more computation than a single-path VLA; latency is higher in real-time high-frequency control settings, and the paper gives no concrete figures on inference speed.