10 Open Challenges Steering the Future of Vision-Language-Action Models

01 动机Motivation

VLA 模型将视觉感知、语言理解与具身动作统一到单一架构中，被视为 Embodied AI 的核心基础。然而，从实验室演示到真实世界大规模部署，这类模型仍面临多维度的系统性挑战。现有工作分散在各个子领域，缺乏对全局挑战的整合视角。VLA models unify visual perception, language understanding and embodied action within a single architecture, and are regarded as a core foundation of Embodied AI. Yet from laboratory demonstrations to large-scale real-world deployment, such models still face systematic challenges along many dimensions. Existing work is scattered across sub-fields and lacks an integrated view of the challenges as a whole.

"VLA models are central to the development of Embodied AI" — 这一判断驱动作者系统梳理十大开放问题，以期为未来研究指明方向。"VLA models are central to the development of Embodied AI" — this judgment motivates the authors to systematically survey ten open problems, so as to point out directions for future research.

VLA challenges overview — 论文概览图：VLA 模型的十大开放性挑战分布，涵盖感知、推理、数据、评估、泛化、效率、协调、安全、智能体与人机交互等维度。Paper overview figure: the distribution of the ten open challenges of VLA models, covering sensing, reasoning, data, evaluation, generalization, efficiency, coordination, safety, agents and human-robot interaction.

10开放性核心挑战Core open challenges

70+Open-X-Embodiment 数据集数量Datasets aggregated in Open-X-Embodiment

1M+Open-X-Embodiment 机器人演示集数Robot demonstrations in Open-X-Embodiment

4新兴解决路径类别Categories of emerging solution paths

VLA 模型分为两类：离散动作模型（将动作 token 化后通过 autoregressive 解码）与连续动作模型（直接回归连续控制信号）。前者推理速度慢，"making it unsuitable for" 高频控制场景；后者与大语言模型的推理能力结合难度更大。两类路径各有权衡，共同面临以下十大挑战。VLA models fall into two families: discrete-action models (which tokenize actions and decode them autoregressively) and continuous-action models (which directly regress continuous control signals). The former is slow at inference, "making it unsuitable for" high-frequency control settings; the latter is harder to combine with the reasoning ability of large language models. The two routes carry their own trade-offs and jointly face the ten challenges below.

02 方法：十大开放性挑战Method: Ten Open Challenges

作者逐一剖析每项挑战的技术本质、当前瓶颈与代表性工作，提供了一个系统性的问题地图。The authors dissect the technical nature, current bottleneck and representative work of each challenge one by one, providing a systematic map of the problem space.

Emerging VLA framework architecture — Figure 1（论文原图）: "A high-level emerging VLA framework" — 分层规划架构示意：高层 LLM/VLM 规划器生成子目标，低层动作专家负责具体控制，辅以安全护栏与多智能体协调模块。Figure 1 (from the paper): "A high-level emerging VLA framework" — a schematic of the hierarchical planning architecture: a high-level LLM/VLM planner generates sub-goals, a low-level action expert handles concrete control, complemented by safety guardrails and a multi-agent coordination module.

Challenge 01

多模态感知 Multimodal Sensing & PerceptionMultimodal Sensing & Perception

主流 VLA 缺乏显式深度信息；仅靠帧差估计深度对物体大小、相机距离高度敏感。环境噪声（反射、镜头光晕、水尘遮挡）严重影响感知质量。触觉模态缺失导致精细力控任务无法完成："touch modality would allow VLA to perform delicate tasks that require careful application of force"。代表方法：MolmoAct、SpatialVLA（引入深度学习）；SimplerEval（分布漂移测试）。Mainstream VLAs lack explicit depth information; estimating depth from frame differences alone is highly sensitive to object size and camera distance. Environmental noise (reflections, lens flare, occlusion by water or dust) severely degrades perception quality. The missing tactile modality prevents fine force-controlled tasks: "touch modality would allow VLA to perform delicate tasks that require careful application of force". Representative methods: MolmoAct, SpatialVLA (introducing depth learning); SimplerEval (distribution-shift testing).

Challenge 02

鲁棒推理 Robust ReasoningRobust Reasoning

LLM 的推理能力迁移到 VLA 后显著衰减——即便是抓取、放置、开抽屉等简单任务也存在较高错误率。长时程任务中错误率随 horizon 增加而累积："error rate on such simpler tasks must approach near perfection" 才能支撑复杂场景部署。工具理解与选择能力依然缺失。代表方法：Emma-X、CoT-VLA、MolmoAct；评估基准：LIBERO、SimplerEnv。The reasoning ability of LLMs degrades markedly once transferred to VLAs — even simple tasks such as grasping, placing and opening drawers still show fairly high error rates. In long-horizon tasks the error rate accumulates as the horizon grows: "error rate on such simpler tasks must approach near perfection" before complex-scene deployment can be supported. Tool understanding and selection remain absent. Representative methods: Emma-X, CoT-VLA, MolmoAct; evaluation benchmarks: LIBERO, SimplerEnv.

Challenge 03

训练数据质量 Quality Training DataQuality Training Data

Open-X-Embodiment 整合了约 70 个数据集、超过 100 万条演示，但模型对分布外环境依然脆弱："VLA models are often brittle to out-of-distribution environments and robot setups"。Sim2Real 差距、具身差异、摄像头位置变化以及人工采集者的不一致性共同引入大量方差与噪声，新场景通常需要额外微调数据。Open-X-Embodiment aggregates about 70 datasets and more than one million demonstrations, yet models remain brittle to out-of-distribution environments: "VLA models are often brittle to out-of-distribution environments and robot setups". The Sim2Real gap, embodiment differences, camera-placement variation and inconsistency among human collectors together inject substantial variance and noise, so new scenes usually require additional fine-tuning data.

Challenge 04

评估体系 EvaluationEvaluation

真实机器人、环境与物体资源稀缺，仿真评估与真实性能相关性差："environments simulated in such tools often fail to capture enough details of their real-life counterparts"。光照、反射、纹理、PD 参数（stiffness/damping）的失真都会导致"poor correlation between in-simulation and real-life performance"。SimplerEnv 通过随机化纹理、光照、相机位姿等改善评估可靠性，但 sim-to-real gap 根本问题仍未解决。Real robots, environments and objects are scarce resources, and simulated evaluation correlates poorly with real performance: "environments simulated in such tools often fail to capture enough details of their real-life counterparts". Distortions in lighting, reflection, texture and PD parameters (stiffness/damping) all lead to "poor correlation between in-simulation and real-life performance". SimplerEnv improves evaluation reliability by randomizing texture, lighting and camera pose, but the fundamental sim-to-real gap remains unsolved.

Challenge 05

跨机器人泛化 Cross-Robot Action GeneralizationCross-Robot Action Generalization

不同机器人平台的动作空间异质性使零样本迁移极为困难："training on action data from a fixed set of embodiments often fails to generalize to others"。自由度数量、结构差异（机械臂 vs. 四足机器人 vs. 自动驾驶车辆）以及控制接口多样性，使统一动作表征成为关键难题。Zheng 等（2025）对通用动作泛化问题进行了系统研究。The heterogeneity of action spaces across robot platforms makes zero-shot transfer extremely hard: "training on action data from a fixed set of embodiments often fails to generalize to others". The number of degrees of freedom, structural differences (robot arm vs. quadruped vs. autonomous vehicle) and the diversity of control interfaces make a unified action representation a key open problem. Zheng et al. (2025) study universal action generalization systematically.

Challenge 06

资源效率 Resource EfficiencyResource Efficiency

机器人平台存在空间与能量约束；灾难现场等断网环境无法依赖云推理："disaster zones are often cutoff from the internet and telecommunication services"。小模型性能显著落后于大模型，但大模型的在端部署成本难以承受。"striking the right balance between VLA model capacity and resource efficiency remains key"。代表系统：OCTO (2023)、OpenVLA、RT-2、MolmoAct。Robot platforms are constrained in space and energy; network-isolated settings such as disaster sites cannot rely on cloud inference: "disaster zones are often cutoff from the internet and telecommunication services". Small models lag markedly behind large ones, yet the on-device deployment cost of large models is hard to bear. "striking the right balance between VLA model capacity and resource efficiency remains key". Representative systems: OCTO (2023), OpenVLA, RT-2, MolmoAct.

Challenge 07

全身协调 Whole-Body CoordinationWhole-Body Coordination

移动操作要求在运动中同步控制底盘与末端执行器，动作空间高维耦合是核心难题："dominant challenge is the high-dimensional search space of coupled actions"。基于模型的控制（MPC）面临实时性瓶颈；基于学习的控制存在探索难度大、credit assignment 困难、以及安全保障弱的问题。代表系统：Mobile ALOHA、UMI on Legs、ACDIT、WBMPC。Mobile manipulation requires controlling the base and the end effector synchronously while moving, and the high-dimensional coupling of the action space is the core difficulty: "dominant challenge is the high-dimensional search space of coupled actions". Model-based control (MPC) faces real-time bottlenecks; learning-based control suffers from hard exploration, difficult credit assignment and weak safety assurances. Representative systems: Mobile ALOHA, UMI on Legs, ACDIT, WBMPC.

Challenge 08

安全保障 Safety AssurancesSafety Assurances

具身 AI 的错误动作可能直接造成人身或财产伤害："robot equipped with embodied AI may harm the victims while saving them due to imperfect actions"。在无人监督的复杂场景（如灾难救援）中，安全护栏的设计尤为关键。当前代表工作 SafeVLA 采用 RL-based 安全对齐，但在性能与安全之间的权衡仍是开放问题。A wrong action by embodied AI can directly cause personal injury or property damage: "robot equipped with embodied AI may harm the victims while saving them due to imperfect actions". In complex unsupervised settings (such as disaster rescue), the design of safety guardrails is especially critical. The current representative work SafeVLA adopts RL-based safety alignment, but the trade-off between performance and safety remains an open problem.

Challenge 09

智能体框架 Agentic FrameworksAgentic Frameworks

VLA 模型嵌入多智能体系统的研究尚处早期：如何分配自主级别、如何通过智能体间通信实现分布式决策、以及如何生成可信赖且可验证的工作流程，均是未解问题。Yang 等（2025）提出了包含 LLM 规划器 + VLM 验证器 + VLA 执行器的异构智能体框架作为初步探索。Research on embedding VLA models into multi-agent systems is still at an early stage: how to assign levels of autonomy, how to achieve distributed decision-making through inter-agent communication, and how to generate trustworthy and verifiable workflows are all unresolved. Yang et al. (2025) propose a heterogeneous agentic framework comprising an LLM planner + a VLM verifier + a VLA executor as a preliminary exploration.

Challenge 10

人机协作 Human-Robot CoordinationHuman-Robot Coordination

当前 VLA 模型的人机通信是单向的（人 → 机器人），机器人无法向人类传达意图或请求缺失信息。CoT-VLA 通过在动作解码前生成意图状态可视化输出，Emma-X 通过高层自然语言 rationale 作出初步探索。真正的双向交互——"reasoning traces and questions for the user seeking missing information"——仍是开放目标。Human-robot communication in current VLA models is one-way (human → robot); the robot cannot convey its intent to humans or ask for missing information. CoT-VLA takes a first step by producing a visualization of the intended state before action decoding, and Emma-X by producing a high-level natural-language rationale. Genuinely bidirectional interaction — "reasoning traces and questions for the user seeking missing information" — remains an open goal.

03 新兴解决路径Emerging Solution Paths

论文（作为综述/位置论文）不提供实验对比表格，而是系统梳理针对上述挑战的四类新兴技术路径，为未来研究提供参考。As a survey / position paper, this work provides no experimental comparison tables; instead it systematically reviews four families of emerging technical paths addressing the challenges above, as a reference for future research.

注：本文为 Position Paper，不包含可量化对比实验结果；以下内容为论文对新兴方法的分析与展望，均来源于原文论述。Note: This paper is a Position Paper and contains no quantifiable comparative experimental results; the content below is the paper's own analysis of and outlook on emerging methods, all drawn from the original text.

路径一：空间理解增强 Spatial UnderstandingPath 1: Enhanced Spatial Understanding

针对多模态感知挑战，作者建议在 RGB-D 数据上微调 VLM backbone，以发展深度感知能力。具体方案包括：利用 Locate 3D 框架从 RGB 帧合成 RGB-D 数据；当真实深度不可用时，保留独立的深度估计专家网络；借助 Veo3 等视频生成平台构建多样化场景训练数据。For the multimodal sensing challenge, the authors suggest fine-tuning the VLM backbone on RGB-D data to develop depth perception. Concrete options include: using the Locate 3D framework to synthesize RGB-D data from RGB frames; keeping a separate depth-estimation expert network when real depth is unavailable; and building diverse-scene training data with video generation platforms such as Veo3.

路径二：世界动力学建模 World Dynamics ModelingPath 2: World Dynamics Modeling

两类主要方法应对长时程推理挑战：
• 生成式建模：以当前状态与动作为条件预测下一状态，通过对比性"self-consistency"目标从预训练 VLM 初始化训练。
• 嵌入预测：V-JEPA-2 方法预测"latent embedding of the future frame"而非像素级重建，实现高效的长时程预测，同时避免像素生成的计算代价。Two main families of methods address the long-horizon reasoning challenge:
• Generative modeling: predicting the next state conditioned on the current state and action, trained from a pre-trained VLM initialization with a contrastive "self-consistency" objective.
• Embedding prediction: the V-JEPA-2 approach predicts the "latent embedding of the future frame" rather than reconstructing pixels, achieving efficient long-horizon prediction while avoiding the compute cost of pixel generation.

路径三：通用动作表征 Universal Action RepresentationPath 3: Universal Action Representation

针对跨机器人泛化挑战，作者提议学习"unified atomic representations of actions"——通过 codebook 建立原子动作库，配合机器人专属 decoder 实现适配。进一步的愿景是通过 prompting 教会 VLA 模型新动作空间，类比 few-shot prompting 在 LLM 中的能力，从而突破固定动作空间的限制。For the cross-robot generalization challenge, the authors propose learning "unified atomic representations of actions" — building a library of atomic actions via a codebook, paired with robot-specific decoders for adaptation. A further vision is to teach VLA models new action spaces through prompting, by analogy with few-shot prompting in LLMs, thereby breaking free of a fixed action space.

路径四：数据合成与后训练 Data Synthesis & Post-TrainingPath 4: Data Synthesis & Post-Training

• 数据合成：从视频中提取 latent action 实现无监督数据扩增；联合训练合成视频与真实机器人数据；将世界模型作为模拟器生成大规模合成数据集，并用专门评估器过滤低质量样本。
• 后训练：将 LLM 后训练技术移植到 VLA——以动作条件化世界模型的预测与子目标状态的对比作为奖励信号，结合 DPO、GRPO 等偏好优化方法；安全评估通过"action-conditioned world models or specialized evaluators"充当"virtual guardrails"。• Data synthesis: extracting latent actions from video for unsupervised data augmentation; jointly training on synthetic video and real robot data; using world models as simulators to generate large-scale synthetic datasets, with dedicated evaluators filtering out low-quality samples.
• Post-training: porting LLM post-training techniques to VLAs — using the contrast between an action-conditioned world model's prediction and the sub-goal state as the reward signal, combined with preference-optimization methods such as DPO and GRPO; safety evaluation uses "action-conditioned world models or specialized evaluators" as "virtual guardrails".

挑战与新兴路径对应关系Mapping from Challenges to Emerging Paths

挑战Challenge	主要新兴解决路径Main emerging solution path	代表方法Representative methods
多模态感知Multimodal sensing	RGB-D 微调、深度专家RGB-D fine-tuning, depth experts	Locate 3D, SpatialVLA, MolmoAct
鲁棒推理Robust reasoning	Chain-of-Thought、世界模型奖励Chain-of-Thought, world-model rewards	CoT-VLA, Emma-X, V-JEPA-2
数据质量Data quality	视频 latent action 提取、合成数据Latent action extraction from video, synthetic data	Veo3, Open-X-Embodiment
跨机器人泛化Cross-robot generalization	通用动作 Codebook + 专属 DecoderUniversal action Codebook + dedicated Decoder	Zheng et al. (2025)
资源效率Resource efficiency	分层规划（大模型规划 + 小模型执行）Hierarchical planning (large model plans + small model executes)	OCTO, OpenVLA, RT-2
全身协调Whole-body coordination	耦合奖励设计、MPC+学习混合Coupled reward design, MPC + learning hybrid	Mobile ALOHA, WBMPC, ACDIT
安全Safety	RL 安全对齐、世界模型虚拟护栏RL safety alignment, world-model virtual guardrails	SafeVLA
智能体框架Agentic frameworks	异构多智能体（规划+验证+执行）Heterogeneous multi-agent (planning + verification + execution)	Yang et al. (2025)
人机协作Human-robot coordination	双向通信、推理轨迹可视化Bidirectional communication, reasoning-trace visualization	CoT-VLA, Emma-X

04 局限性Limitations

注：本文为综述 / Position Paper，未设独立"Limitations"章节。以下局限性部分来自论文正文中的明确陈述（stated），部分来自论文结构与方法论的推断（inferred）。Note: This paper is a survey / Position Paper with no dedicated "Limitations" section. Some of the limitations below are explicit statements in the paper's main text (stated), while others are inferred from its structure and methodology (inferred).

仿真评估与真实世界的相关性不足（stated）Weak correlation between simulated evaluation and the real world (stated)

论文明确指出，当前仿真评估环境"often fail to capture enough details of their real-life counterparts"，sim-to-real gap 导致评估结论的可靠性存疑。SimplerEnv 等工具通过随机化改善了部分问题，但根本性 gap 仍未弥合。The paper states explicitly that current simulated evaluation environments "often fail to capture enough details of their real-life counterparts", so the sim-to-real gap makes the reliability of evaluation conclusions questionable. Tools such as SimplerEnv mitigate part of the problem through randomization, but the fundamental gap is not closed.

挑战分析缺乏定量基准支撑（inferred）Challenge analysis lacks quantitative benchmark support (inferred)

作为 Position Paper，论文对十大挑战的描述主要依赖定性分析与引文综述，缺乏统一的定量实验验证。各挑战的严重程度难以横向比较，优先级排序带有主观判断成分。As a Position Paper, its description of the ten challenges relies mainly on qualitative analysis and a survey of citations, without unified quantitative experimental validation. The severity of the individual challenges is hard to compare side by side, and their prioritization carries an element of subjective judgment.

提出路径的可行性尚待验证（inferred）Feasibility of the proposed paths remains to be verified (inferred)

文中提出的四类新兴解决路径（空间理解、世界动力学建模、通用动作表征、数据合成与后训练）多处于概念或早期探索阶段，部分方法（如基于 prompting 的动作空间扩展）尚无充分实证支持，实际效果有待后续工作检验。The four families of emerging solution paths proposed in the text (spatial understanding, world dynamics modeling, universal action representation, data synthesis and post-training) are mostly at a conceptual or early exploratory stage; some methods (such as prompting-based action-space extension) have no sufficient empirical support yet, and their practical effectiveness awaits verification by follow-up work.

覆盖范围的选择性（inferred）Selectivity of coverage (inferred)

"10 大挑战"的选取不可避免地带有作者视角的局限。例如，标定误差、感知延迟、硬件可靠性等工程层面挑战，以及多语言/跨文化人机交互等社会技术挑战，在论文中未得到充分讨论。The selection of the "10 challenges" inevitably carries the limits of the authors' perspective. For example, engineering-level challenges such as calibration error, perception latency and hardware reliability, as well as socio-technical challenges such as multilingual / cross-cultural human-robot interaction, are not discussed at length in the paper.