Vision-Language Foundation Models as Effective Robot Imitators

01 Motivation

视觉-语言大模型（VLM）已展现出强大的多模态理解能力，但将其直接用于机器人低层控制仍面临三大挑战：① VLM 在静态图像-文本对上预训练，难以处理视频时序观测；② VLM 输出语言 token，而机器人需要连续动作信号；③ 现有高性能方案（如 RT-2）依赖私有模型与海量数据，无法被普通研究者复用。本文提出 RoboFlamingo，探索一种低成本、开源、可单卡运行的 VLM 机器人操作方案。Vision-language models (VLMs) have shown strong multimodal understanding, yet applying them directly to low-level robot control still faces three challenges: (i) VLMs are pre-trained on static image-text pairs and struggle with temporal video observations; (ii) VLMs emit language tokens, whereas robots require continuous action signals; (iii) existing high-performance solutions (e.g. RT-2) rely on proprietary models and massive data, and cannot be reused by ordinary researchers. This paper proposes RoboFlamingo, exploring a low-cost, open-source, single-GPU VLM solution for robot manipulation.

"We seek a straightforward way of making use of existing vision-language models (VLMs) with simple fine-tuning on robotics data. ... RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control."

RoboFlamingo 与现有方案对比示意图 — **图1：**RoboFlamingo 与现有视觉-语言机器人操作方案的对比。左侧方案（如 SayCan、PaLM-E）将 VLM 用作高层规划器，输出语言技能指令；中间的 RT-2 将 VLM 与低层动作联合微调，但依赖私有模型。RoboFlamingo（右）则解耦感知与决策：Flamingo 主干负责单步视觉-语言理解，显式 policy head 负责时序历史建模与动作输出，仅在机器人演示数据上微调。**Figure 1:** Comparison between RoboFlamingo and existing vision-language robot manipulation approaches. The approaches on the left (e.g. SayCan, PaLM-E) use the VLM as a high-level planner that emits language skill commands; RT-2 in the middle jointly fine-tunes the VLM with low-level actions but relies on a proprietary model. RoboFlamingo (right) instead decouples perception from decision-making: the Flamingo backbone handles single-step vision-language understanding, while an explicit policy head handles temporal history modeling and action output, fine-tuned only on robot demonstration data.

4.09Avg Len（ABCD→D，RoboFlamingo）Avg Len (ABCD→D, RoboFlamingo)

3.06Avg Len（ABCD→D，HULC 最优基线）Avg Len (ABCD→D, HULC, best baseline)

2×相对先前 SOTA 的性能提升Improvement over the previous SOTA

单张 GPUSingle GPU可完成训练与评估Training and evaluation feasible

02 Method

RoboFlamingo 的核心设计是感知与决策解耦：以 OpenFlamingo 作为视觉-语言主干（perception），负责在每一决策步对当前图像观测与语言指令进行单步理解；再附加一个轻量 policy head（decision-making），对历史特征进行建模并输出机器人动作。整个框架仅微调 resampler、gated cross-attention 模块与 policy head，其余参数冻结。The core design of RoboFlamingo is decoupling perception from decision-making: OpenFlamingo serves as the vision-language backbone (perception), performing single-step understanding of the current image observation and the language instruction at every decision step; a lightweight policy head (decision-making) is then attached to model historical features and output robot actions. The whole framework fine-tunes only the resampler, the gated cross-attention modules and the policy head, keeping all remaining parameters frozen.

RoboFlamingo 框架示意图 — **图2：**RoboFlamingo 框架总览。Flamingo backbone 由视觉编码器（ViT + perceiver resampler）和特征融合解码器（gated cross-attention + frozen LLM layers）组成，在每步将双视角图像 I_t、G_t 与语言指令 l 融合为视觉-语言联合嵌入 X_t。Policy head（默认为 LSTM + MLP）接收 X_t 并维护隐状态 h_t，输出 7-DoF 末端位姿增量与夹爪状态。**Figure 2:** Overview of the RoboFlamingo framework. The Flamingo backbone consists of a vision encoder (ViT + perceiver resampler) and a feature-fusion decoder (gated cross-attention + frozen LLM layers); at each step it fuses the dual-view images I_t, G_t and the language instruction l into a joint vision-language embedding X_t. The policy head (LSTM + MLP by default) takes X_t, maintains a hidden state h_t, and outputs the 7-DoF end-effector pose delta and the gripper state.

Flamingo Backbone：视觉-语言单步理解Flamingo Backbone: Single-Step Vision-Language Understanding

视觉编码器由 ViT 和 perceiver resampler 组成：ViT 将双视角图像编码为视觉 token 序列，resampler 通过可学习查询向量将 token 数从 N 压缩到 N_r，大幅降低后续计算量。特征融合解码器含 L 层交叉注意力（cross-attention）层与冻结 LLM（LLaMA / GPT-Neox / MPT）层：LLM 层参数全程冻结，仅微调 cross-attention 与 resampler 参数。每层解码器输出为视觉-语言联合嵌入，通过带可学习门控参数 α 的 gated cross-attention 控制视觉信息混入比例，保证训练稳定性。The vision encoder consists of a ViT and a perceiver resampler: the ViT encodes the dual-view images into a sequence of visual tokens, and the resampler compresses the token count from N to N_r with learnable query vectors, greatly reducing downstream computation. The feature-fusion decoder contains L cross-attention layers and frozen LLM (LLaMA / GPT-Neox / MPT) layers: the LLM parameters stay frozen throughout, and only the cross-attention and resampler parameters are fine-tuned. Each decoder layer outputs a joint vision-language embedding, and a gated cross-attention with a learnable gating parameter α controls how much visual information is mixed in, ensuring training stability.

Policy Head：历史建模与动作输出Policy Head: History Modeling and Action Output

论文测试了四种 policy head 变体：MLP w/o hist（仅当前帧）、MLP w hist（将历史帧送入视觉编码器）、GPT（显式 token 序列）和 LSTM（隐式记忆）。消融实验表明 GPT 与 LSTM 性能相近，最终选择 LSTM 作为默认配置（简洁高效）。以 LSTM 为例，policy head 对 X_t^L 进行 max-pooling 聚合后输入 LSTM，并通过 MLP 分别预测末端位姿增量（MSE 损失回归）与夹爪状态（BCE 分类损失）。The paper tests four policy head variants: MLP w/o hist (current frame only), MLP w hist (history frames fed into the vision encoder), GPT (explicit token sequence) and LSTM (implicit memory). Ablations show that GPT and LSTM perform comparably, so LSTM is chosen as the default configuration for its simplicity and efficiency. Taking LSTM as an example, the policy head aggregates X_t^L by max-pooling before feeding it into the LSTM, and uses MLPs to predict the end-effector pose delta (regression with an MSE loss) and the gripper state (BCE classification loss) separately.

训练策略Training Strategy

仅使用 CALVIN 中带语言标注的演示数据（约 2.4 万步，占全量数据的 1%）Uses only the language-annotated demonstrations in CALVIN (about 24000 steps, 1% of the full dataset)
冻结 LLM 层；微调 resampler、gated cross-attention 与 policy headFreeze the LLM layers; fine-tune the resampler, gated cross-attention and policy head
联合优化 MSE 末端位姿损失 + BCE 夹爪状态损失（权重 λ_gripper）Jointly optimizes the MSE end-effector pose loss + the BCE gripper state loss (weight λ_gripper)

部署灵活性（Open-loop Control）Deployment Flexibility (Open-loop Control)

感知与决策解耦使得可以预测动作序列（stacked actions），而无需每步重新推理 VLMDecoupling perception from decision-making makes it possible to predict action sequences (stacked actions) without re-running VLM inference at every step
直接开环控制性能会下降；以跳步演示（jump step demonstration）重训后可显著缓解Naive open-loop control degrades performance; retraining with jump step demonstrations alleviates this substantially
支持低算力平台部署，单张 GPU 即可完成训练与评估Supports deployment on low-compute platforms; a single GPU suffices for training and evaluation

Policy head 变体对比 — **图3：**四种 policy head 变体的架构示意（出自论文附录）：(a) MLP w/o hist，(b) MLP w hist，(c) GPT，(d) LSTM。GPT 与 LSTM 均将 Flamingo backbone 用于单帧感知，在 policy head 中独立建模时序历史信息，性能显著优于将历史送入 VLM 的 MLP w hist 方案。**Figure 3:** Architectures of the four policy head variants (from the paper's appendix): (a) MLP w/o hist, (b) MLP w hist, (c) GPT, (d) LSTM. Both GPT and LSTM use the Flamingo backbone for single-frame perception and model temporal history separately inside the policy head, clearly outperforming the MLP w hist variant that feeds history into the VLM.

03 Experiments

实验基准为 CALVIN（Composing Actions from Language and Vision）：包含 34 个操作任务，评估 1000 条指令链，每条链要求机器人连续完成最多 5 步语言指令，仅当前任务成功才进入下一步。训练数据为带语言标注的演示（仅 Lang 分割，约 1% 全量数据）。主要对比基线：MCIL、HULC、RT-1。The benchmark is CALVIN (Composing Actions from Language and Vision): 34 manipulation tasks evaluated over 1000 instruction chains, where each chain requires the robot to complete up to 5 language instructions in a row and only advances when the current task succeeds. Training data are the language-annotated demonstrations (the Lang split only, about 1% of the full data). The main baselines are MCIL, HULC and RT-1.

主要结果：Avg Len（ABCD→D，5步序列）Main Results: Avg Len (ABCD→D, 5-step sequences)

方法Method	训练数据Training Data	步骤1Step 1	步骤2Step 2	步骤3Step 3	步骤4Step 4	步骤5Step 5	Avg Len
MCIL	ABCD (Full)	0.373	0.027	0.002	0.000	0.000	0.40
HULC	ABCD (Full)	0.889	0.733	0.587	0.475	0.383	3.06
HULC	ABCD (Lang)	0.892	0.701	0.548	0.420	0.335	2.90
RT-1	ABCD (Lang)	0.844	0.617	0.438	0.323	0.227	2.45
RoboFlamingo (Ours)	ABCD (Lang)	0.964	0.896	0.824	0.740	0.660	4.09

零样本视觉泛化（ABC→D，跨环境）Zero-Shot Visual Generalization (ABC→D, Cross-Environment)

方法Method	训练数据Training Data	步骤1Step 1	步骤2Step 2	步骤3Step 3	步骤4Step 4	步骤5Step 5	Avg Len
HULC	ABC (Full)	0.418	0.165	0.057	0.019	0.011	0.67
RT-1	ABC (Lang)	0.533	0.222	0.094	0.038	0.013	0.90
RoboFlamingo (Ours)	ABC (Lang)	0.824	0.619	0.466	0.331	0.235	2.48

不同 VLM backbone 对比（ABCD→D，Best Avg Len）Comparison of VLM Backbones (ABCD→D, Best Avg Len)

Backbone	LLM 架构LLM Architecture	总参数Total Params	指令微调Instruction Tuning	COCO CIDEr (4-shot)	VQAv2 Acc (4-shot)	Best Avg Len	Mean Avg Len
M-3B	MPT	3B	否No	77.3	45.8	3.94	3.81
M-3B-IFT	MPT	3B	是Yes	82.7	45.7	4.09	4.02
G-4B	GPT-Neox	4B	否No	81.8	49.0	3.67	3.53
G-4B-IFT	GPT-Neox	4B	是Yes	85.8	49.0	3.79	3.72
L-9B	LLaMA	9B	否No	74.3	44.0	2.79	2.71
M-9B	MPT	9B	否No	89.0	54.8	3.97	3.87

质量分析示例 — **图4：**定性分析示例（出自论文 quality_ana 图）：展示 RoboFlamingo 在成功与失败案例中的视觉-语言对齐情况，帮助理解模型在不同对象位置和指令措辞下的行为差异。**Figure 4:** Qualitative analysis examples (from the paper's quality_ana figure): the vision-language alignment of RoboFlamingo in successful and failed cases, helping to understand how the model's behavior differs across object positions and instruction phrasings.

消融研究Ablation Studies

消融实验主要揭示以下结论：The ablations mainly reveal the following conclusions:

Policy head 设计：MLP w/o hist 性能最差，说明历史信息至关重要。GPT 与 LSTM 性能相近，选 LSTM 作为默认（简洁）。MLP w hist 虽引入历史帧但仍明显弱于 GPT/LSTM，推测是 OpenFlamingo 预训练时未见过连续帧序列。Policy head design: MLP w/o hist performs worst, showing that history information is essential. GPT and LSTM perform comparably, and LSTM is chosen as the default for its simplicity. Although MLP w hist does introduce history frames, it remains clearly weaker than GPT/LSTM, presumably because OpenFlamingo never saw consecutive frame sequences during pre-training.
VL 预训练的必要性：去除 OpenFlamingo 预训练权重（No VL Pre-train）或冻结 VLM 只训练 policy head（No VL Finetune）均导致性能大幅下降，证明 VL 预训练与在机器人数据上微调缺一不可。Necessity of VL pre-training: Removing the OpenFlamingo pre-trained weights (No VL Pre-train) or freezing the VLM and training only the policy head (No VL Finetune) both cause a large performance drop, proving that VL pre-training and fine-tuning on robot data are equally indispensable.
指令微调（Instruction Fine-tuning）：M-3B-IFT 相比 M-3B、G-4B-IFT 相比 G-4B 均有明显提升，说明 IFT 有助于将 VLM 的语言泛化能力迁移至机器人指令跟随。Instruction fine-tuning: M-3B-IFT improves clearly over M-3B, and G-4B-IFT over G-4B, showing that IFT helps transfer the language generalization of the VLM to robot instruction following.
模型规模与数据效率：在仅 10% 训练数据场景下，更大模型（M-9B: Avg Len 0.83）显著优于小模型（M-3B-IFT: 0.13），说明更大的 VLM 具有更强的数据效率。Model scale and data efficiency: With only 10% of the training data, the larger model (M-9B: Avg Len 0.83) clearly outperforms the smaller one (M-3B-IFT: 0.13), showing that larger VLMs are more data-efficient.
语言泛化（enriched instructions）：使用 GPT-4 生成的 50 条同义指令测试时，冻结 embedding 层（freeze-emb）可缓解性能下降（Avg Len 从 1.85 提升至 2.12）。Language generalization (enriched instructions): When tested on 50 synonymous instructions generated by GPT-4, freezing the embedding layer (freeze-emb) mitigates the performance drop (Avg Len rises from 1.85 to 2.12).

04 Limitations

注：论文结论部分明确指出第一点局限（作者原话）；其余两点为从设计出发的推断（inferred）。Note: The first limitation is stated explicitly in the paper's conclusion (the authors' own words); the other two are inferred from the design.

缺乏真实机器人部署验证（作者明确指出）No real-robot deployment validation (explicitly stated by the authors)

"Due to the lack of real-robot data, this paper does not deploy on real-world robotics." 所有实验均在 CALVIN 仿真环境中进行。作者也指出，近期大规模真实机器人数据（如 Open X-Embodiment）的出现有望解决这一问题。"Due to the lack of real-robot data, this paper does not deploy on real-world robotics." All experiments are conducted in the CALVIN simulation environment. The authors also note that the recent arrival of large-scale real-robot data (such as Open X-Embodiment) is likely to resolve this issue.

语言泛化能力受 VLM 微调影响（inferred）Language generalization is affected by VLM fine-tuning (inferred)

实验显示，在 enriched instructions（同义替换）设置下，RoboFlamingo 后续任务成功率的相对下降比 HULC 更明显。论文推断这是因为 RoboFlamingo 直接以 word token 作为输入，对同义表述更敏感；冻结 embedding 层（freeze-emb）可部分缓解，但整体语言泛化与未微调的 VLM 相比仍有差距。Experiments show that under the enriched instructions (synonym substitution) setting, the relative drop in RoboFlamingo's success rate on later tasks is more pronounced than for HULC. The paper infers that this is because RoboFlamingo takes word tokens directly as input and is therefore more sensitive to synonymous phrasings; freezing the embedding layer (freeze-emb) partially mitigates it, but overall language generalization still lags behind an un-fine-tuned VLM.

开环控制性能下降问题（inferred）Performance degradation under open-loop control (inferred)

直接对已训练模型执行 open-loop control（预测动作序列而不每步重新推理）会导致性能明显下降；需要用跳步演示数据（jump step demonstration）重新训练才能恢复性能，增加了部署灵活性的使用门槛。Running open-loop control directly on a trained model (predicting action sequences instead of re-inferring at every step) causes a clear performance drop; recovering performance requires retraining with jump step demonstration data, which raises the barrier to using this deployment flexibility.