ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model

01 动机 MotivationMotivation

现有 VLA 模型在机器人控制与多模态理解之间存在根本性矛盾：擅长底层操作的架构往往丧失视觉-语言对齐能力，而强大的 VLM 又缺乏物理交互能力。作者通过系统分析训练范式，识别出两大核心挑战。Existing VLA models face a fundamental tension between robot control and multimodal understanding: architectures that excel at low-level manipulation tend to lose their vision-language alignment, while powerful VLMs lack physical interaction ability. Through a systematic analysis of training paradigms, the authors identify two key challenges.

"Through a systematic analysis of existing training paradigms in vision-language-action models (VLA), we identify two key challenges: spurious forgetting, where robot training overwrites crucial visual-text alignments, and task interference, where competing control and understanding tasks degrade performance when trained jointly."

teaser — training data influence on VLA performance — **Figure 1.** 不同训练数据配置对 VLA 性能的影响对比。横轴为5项真实机器人任务，纵轴为6个多模态基准。实验揭示：仅用机器人数据训练会导致多模态理解能力严重下降；而朴素的联合训练（naive joint training）同样会因任务干扰损害两端性能。ChatVLA 通过分阶段训练策略有效平衡了两者。**Figure 1.** Comparison of how different training-data configurations affect VLA performance. The horizontal axis lists 5 real-robot tasks, the vertical axis 6 multimodal benchmarks. The experiments reveal that training on robot data alone severely degrades multimodal understanding, while naive joint training likewise hurts both sides through task interference. ChatVLA balances the two effectively with a phased training strategy.

6×在 MMMU 上相比 ECoT 的性能提升（37.4 vs. 5.4）Gain on MMMU over ECoT (37.4 vs. 5.4)

55/107ChatVLA 跨技能多任务成功次数（OpenVLA 仅 20/107）Cross-skill multi-task successes (OpenVLA only 20/107)

1.00Task 3 平均成功长度（满分），Octo 仅 0.11Average success length on Task 3 (perfect score); Octo only 0.11

2B模型参数量，对标 Qwen2-VL-2B 底座Model parameters, built on the Qwen2-VL-2B backbone

两大核心挑战Two Core Challenges

Spurious Forgetting（伪遗忘）Spurious Forgetting

机器人数据微调过程中，模型会覆盖预训练阶段习得的视觉-文本对齐能力。即便使用 LoRA 等参数高效方法，这种遗忘现象依然存在，导致 VLA 在多模态理解基准（如 MMMU、TextVQA）上性能几乎归零。During fine-tuning on robot data, the model overwrites the vision-text alignment acquired in pre-training. Even with parameter-efficient methods such as LoRA, this forgetting persists, driving VLA performance on multimodal understanding benchmarks (e.g. MMMU, TextVQA) to nearly zero.

Task Interference（任务干扰）Task Interference

当控制任务与理解任务同时训练时，二者的梯度方向相互冲突，导致两端性能均出现退化。这与 Dual Coding Theory（双编码理论）的预测一致——不同模态的信息在大脑中由独立通道处理。When control and understanding tasks are trained jointly, their gradient directions conflict and performance degrades on both sides. This agrees with the prediction of Dual Coding Theory: information from different modalities is processed by separate channels in the brain.

02 方法 MethodMethod

ChatVLA 提出两项互补的创新：Phased Alignment Training（分阶段对齐训练）解决伪遗忘问题，Mixture-of-Experts (MoE) 架构解决任务干扰问题。两者共同实现多模态理解与机器人控制的统一。ChatVLA introduces two complementary innovations: Phased Alignment Training resolves spurious forgetting, and a Mixture-of-Experts (MoE) architecture resolves task interference. Together they unify multimodal understanding and robot control.

**Figure 2.** 分阶段对齐训练策略示意。第一阶段（Stage 1）：仅使用机器人轨迹数据训练，使模型充分掌握具身控制能力；第二阶段（Stage 2）：引入视觉-文本数据与机器人数据联合训练（co-training），以逐步"重激活"（reactivate）被冻结的对齐链路，同时保留操作技能。**Figure 2.** The phased alignment training strategy. Stage 1: train on robot trajectory data only, so that the model fully acquires embodied control. Stage 2: co-train vision-text data together with robot data, so as to gradually "reactivate" the frozen alignment pathway while retaining manipulation skills.

Phased Alignment Training（分阶段对齐训练）Phased Alignment Training

直接将机器人数据与视觉-文本数据混合训练会导致两类任务相互拖累。ChatVLA 的解决思路是：先专注控制，再逐步引入理解。第一阶段让模型在机器人轨迹上充分优化，建立稳固的动作预测能力；第二阶段再以合理的数据比例混入视觉-文本数据（论文发现 1:3 的视觉-文本:机器人数据比例最优），"渐进式"地恢复多模态对齐能力，而不损害已学到的控制策略。Mixing robot data with vision-text data from the start makes the two task families drag each other down. ChatVLA's remedy is to focus on control first, then introduce understanding gradually. The first stage optimizes the model on robot trajectories until action prediction is solid; the second stage then mixes in vision-text data at a sensible ratio (the paper finds a 1:3 vision-text-to-robot data ratio optimal), progressively restoring multimodal alignment without damaging the control policy already learned.

Mixture-of-Experts Architecture（混合专家架构）Mixture-of-Experts Architecture

受 Dual Coding Theory 启发，ChatVLA 在每个 Transformer 块中设置两类专家（Expert）：Inspired by Dual Coding Theory, ChatVLA places two kinds of Expert in every Transformer block:

共享 Self-Attention 层：跨任务共享，使机器人控制与多模态理解可以相互借鉴（knowledge transfer）。Shared Self-Attention layers: shared across tasks, letting robot control and multimodal understanding draw on each other (knowledge transfer).
独立 MLP Expert：机器人专家仅处理动作 token，视觉-文本专家仅处理理解 token，从根源上隔离任务干扰。Separate MLP Experts: the robot expert handles only action tokens and the vision-text expert only understanding tokens, isolating task interference at its root.
动态路由（Router）：根据输入 token 的类型（机器人 vs. 视觉-文本）自动分配专家，无需手动切换。Dynamic Router: experts are assigned automatically according to the input token type (robot vs. vision-text), with no manual switching.

与 Dense MoE（所有 token 使用全部专家）不同，ChatVLA 的路由策略是确定性的——机器人 token 只走机器人专家，视觉-文本 token 只走理解专家，避免了 top-k sparse routing 带来的负载均衡问题。Unlike Dense MoE, where all tokens use all experts, ChatVLA's routing policy is deterministic: robot tokens go only to the robot expert and vision-text tokens only to the understanding expert, avoiding the load-balancing problems of top-k sparse routing.

03 实验 ExperimentsExperiments

实验分两部分：（1）在13个多模态理解基准上与 Qwen2-VL、OpenVLA、ECoT、DiVLA 对比；（2）在25项真实机器人任务（长时序操作 + 跨技能多任务）上与 Octo、OpenVLA 对比。所有机器人实验均在真实物理环境中进行，无仿真。The evaluation has two parts: (1) comparison against Qwen2-VL, OpenVLA, ECoT and DiVLA on 13 multimodal understanding benchmarks; (2) comparison against Octo and OpenVLA on 25 real-robot tasks (long-horizon manipulation plus cross-skill multi-tasking). All robot experiments are run in the real physical world, without simulation.

Table 1：多模态理解基准对比Table 1: Multimodal understanding benchmarks

Method	#Params	MMMU	MMStar	TextVQA	DocVQA	MME	OCRBench	HallBench
Qwen2-VL	2B	41.1	48.0	79.7	88.57	1872.0	809	41.7
OpenVLA	7B	0	0	0	0	0	0	0
ECoT	7B	5.4	0	0	0	0	12	0.9
DiVLA	2B	17.2	21.1	7.5	15.2	186.5	294	9.0
ChatVLA (Ours)	2B	37.4	47.2	71.2	83.3	1435.2	729	39.9

ChatVLA 在大多数理解基准上远超同等参数的 VLA 竞品（OpenVLA、ECoT、DiVLA），并接近纯 VLM 基线 Qwen2-VL。在 MMMU 上实现 37.4 vs. ECoT 的 5.4（约 6 倍提升）；在 TextVQA 上达到 71.2（OpenVLA 为 0）。ChatVLA far outperforms VLA competitors of comparable size (OpenVLA, ECoT, DiVLA) on most understanding benchmarks and comes close to the pure-VLM baseline Qwen2-VL. On MMMU it reaches 37.4 vs. 5.4 for ECoT (about six times higher); on TextVQA it reaches 71.2 (OpenVLA scores 0).

MMMU category-wise comparison — **Figure 4.** ChatVLA 与 Qwen2-VL 在 MMMU 验证集六大类别上的逐类对比。ChatVLA 在多数类别接近 Qwen2-VL（2B），但在 Art Theory、Lab Medicine、Pharmacy、Literature、Psychology 等专业领域仍有明显差距，原因是训练数据（LLaVA）中此类专业知识覆盖不足。**Figure 4.** Category-wise comparison of ChatVLA and Qwen2-VL on the six categories of the MMMU validation set. ChatVLA approaches Qwen2-VL (2B) in most categories, but a clear gap remains in specialized domains such as Art Theory, Lab Medicine, Pharmacy, Literature and Psychology, because the training data (LLaVA) covers such expert knowledge only sparsely.

Table 2：长时序机器人任务（直接指令）Table 2: Long-horizon robot tasks (direct instructions)

Method	Task 1 平均成功长度Task 1 average success length	Task 2	Task 3	Task 4
Octo	0.08	0.21	0.11	0.33
OpenVLA	0.06	0.29	0.15	0.42
ChatVLA (Ours)	0.54	0.64	1.00	0.75

Table 4：跨技能多任务（25 项真实任务）Table 4: Cross-skill multi-tasking (25 real tasks)

Method	Bathroom (Tasks 14-17)	Kitchen (Tasks 18-19)	Tabletop (Tasks 20-25)	Total
Octo	4/34	4/18	10/55	18/107
OpenVLA	3/34	5/18	12/55	20/107
ChatVLA (Ours)	16/34	12/18	27/55	55/107

real robot task settings — **Figure 5.** 真实机器人任务场景设置。涵盖长时序直接指令任务（玩具分类、抽屉开关等）、高层规划器引导任务，以及跨场景多任务（浴室、厨房、桌面）。**Figure 5.** Settings of the real-robot tasks. They cover long-horizon direct-instruction tasks (toy sorting, opening and closing drawers, and so on), tasks guided by a high-level planner, and cross-scene multi-tasking (bathroom, kitchen, tabletop).

Ablations：数据比例消融（Table 5）Ablations: data ratio (Table 5)

论文对视觉-文本数据与机器人数据的比例进行系统消融（1:1 / 3:1 / 1:3）：The paper systematically ablates the ratio of vision-text data to robot data (1:1 / 3:1 / 1:3):

Ratio (VT:Robot)	MMMU	MMStar	TextVQA	DocVQA	MME
1:1	36.1	44.7	72.6	82.9	1426.9
3:1	35.3	45.3	72.7	83.6	1399.5
1:3（最优）1:3 (optimal)	37.4	47.2	71.2	83.3	1435.2

结果表明，1:3 的视觉-文本:机器人数据比例在多数理解基准上取得最优，同时维持机器人任务成功率。增大视觉-文本比例（3:1）不仅未能进一步提升理解性能，反而对 MMMU 等指标略有损害。The results show that a 1:3 vision-text-to-robot data ratio is best on most understanding benchmarks while maintaining robot task success rates. Raising the vision-text share (3:1) does not improve understanding further and even slightly hurts metrics such as MMMU.

04 局限性 LimitationsLimitations

Note: 以下局限性部分由论文作者明确陈述（标注为"stated"），部分由设计推断（标注为"inferred"）。Some of the limitations below are explicitly stated by the paper's authors (marked "stated"), others are inferred from the design (marked "inferred").

专业领域知识覆盖不足（stated）Limited coverage of specialized-domain knowledge (stated)

论文明确指出：ChatVLA 在 MMMU 的艺术理论（Art Theory）、实验室医学（Lab Medicine）、药学（Pharmacy）、文学（Literature）、心理学（Psychology）等专业类别上与 Qwen2-VL 仍有明显差距，原因是训练所用的 LLaVA 数据集对这些专业领域的覆盖有限。作者表示"carefully curated data is crucial for mitigating spurious forgetting, a topic we plan to explore in future work"。The paper states explicitly that ChatVLA still lags clearly behind Qwen2-VL on MMMU's specialized categories such as Art Theory, Lab Medicine, Pharmacy, Literature and Psychology, because the LLaVA dataset used for training covers these domains only to a limited extent. The authors note that "carefully curated data is crucial for mitigating spurious forgetting, a topic we plan to explore in future work".

多模态理解性能尚未追平纯 VLM（stated）Multimodal understanding has not yet caught up with pure VLMs (stated)

尽管 ChatVLA（2B）在多数基准上远超同类 VLA，但与同等参数的纯视觉语言模型 Qwen2-VL-2B 相比仍有差距（例如 MMMU：37.4 vs. 41.1；TextVQA：71.2 vs. 79.7）。引入机器人控制能力本身仍会带来一定的理解能力损耗。Although ChatVLA (2B) far exceeds comparable VLAs on most benchmarks, a gap remains against the pure vision-language model Qwen2-VL-2B of the same size (e.g. MMMU: 37.4 vs. 41.1; TextVQA: 71.2 vs. 79.7). Adding robot control ability still costs some understanding performance.

数据路由的确定性假设（inferred）Deterministic assumption in data routing (inferred)

ChatVLA 的 MoE 路由采用基于 token 类型的确定性分配（机器人 token → 机器人专家，视觉-文本 token → 理解专家），这简化了训练但也限制了模型在混合任务场景下的灵活性。对于需要同时融合动作预测与语言推理的复杂指令，固定路由可能不是最优方案（inferred）。ChatVLA's MoE routing uses a deterministic assignment based on token type (robot tokens to the robot expert, vision-text tokens to the understanding expert), which simplifies training but also limits the model's flexibility in mixed-task settings. For complex instructions that require fusing action prediction with language reasoning at once, fixed routing may not be the optimal design (inferred).

真实机器人实验规模有限（inferred）Limited scale of the real-robot experiments (inferred)

尽管论文在真实物理环境（浴室、厨房、桌面）中进行了25项任务的评估，但每项任务的试验次数有限，且场景多样性相对受控。方法在更大规模、更开放环境下的泛化能力尚待验证（inferred）。Although the paper evaluates 25 tasks in real physical environments (bathroom, kitchen, tabletop), the number of trials per task is limited and scene diversity is relatively controlled. Generalization to larger-scale, more open environments remains to be verified (inferred).