StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing

01 动机Motivation

当前 VLA 研究面临严重的"碎片化"问题：现有方法在模型结构、训练流程和推理假设上差异显著，导致跨方法的系统比较几乎不可能，使得"究竟是数据、架构还是训练技巧带来的提升"难以厘清。Current VLA research suffers from severe "fragmentation": existing methods differ markedly in model architecture, training pipeline and inference assumptions, which makes systematic cross-method comparison nearly impossible and leaves it unclear "whether an improvement comes from the data, the architecture or the training tricks".

"We attribute this fragmentation to the lack of a unified abstraction for VLA systems. Existing codebases are largely method-specific and do not support (i) modular composition across action-decoding paradigms, (ii) reusable training approaches across heterogeneous data sources, or (iii) standardized evaluation and deployment across benchmarks and embodiments."

Figure 1: Unified VLA Policy Conceptual Diagram — **图 1：StarVLA 统一 VLA 策略示意。** 策略 π 以多模态观测历史 *x≤t* 和语言指令 ℓ 为输入，输出动作序列 *a_t:t+k* 及辅助输出 *y_aux*。训练目标分解为 *ℒ = ℒ_action + ℒ_aux*，将动作预测损失与语言/视觉辅助监督解耦，使不同范式的 VLA 方法均可在同一接口下实例化。**Figure 1: Illustration of the StarVLA unified VLA policy.** The policy π takes the multimodal observation history *x≤t* and the language instruction ℓ as input, and outputs the action sequence *a_t:t+k* together with the auxiliary output *y_aux*. The training objective decomposes as *ℒ = ℒ_action + ℒ_aux*, decoupling the action prediction loss from the language/visual auxiliary supervision, so that VLA methods of different paradigms can all be instantiated under the same interface.

5集成主流评测基准数量Mainstream benchmarks integrated

4支持的动作解码范式Action-decoding paradigms supported

30K训练步数即可媲美竞品 175K+ 步Training steps, on par with competitors at 175K+ steps

2,200256 GPU 下最大样本吞吐量（samples/s）Peak sample throughput on 256 GPUs (samples/s)

02 方法Method

StarVLA 的核心是 backbone–action head 双向模块化：骨干网络（VLM 或世界模型）与动作解码头各自遵守统一的 I/O 协议，可独立替换而不影响另一侧。训练、推理与部署共用同一套代码，配置通过 YAML 声明式指定。The core of StarVLA is bidirectional modularity of backbone and action head: the backbone network (a VLM or a world model) and the action decoding head each obey a unified I/O protocol and can be replaced independently without affecting the other side. Training, inference and deployment share one and the same codebase, and configurations are specified declaratively through YAML.

统一抽象：Unified VLA PolicyUnified abstraction: Unified VLA Policy

策略被形式化为 π(a_t:t+k, y_aux | x_≤t, ℓ)，将多模态观测历史映射到 k 步动作序列。训练损失为 ℒ = ℒ_action + ℒ_aux，其中 ℒ_aux 作为归纳偏置，可为零（纯行为克隆）或语言对齐/空间感知损失。不同 VLA 范式均可理解为该公式在不同归纳偏置下的实例化。The policy is formalized as π(a_t:t+k, y_aux | x_≤t, ℓ), mapping the multimodal observation history to a k-step action sequence. The training loss is ℒ = ℒ_action + ℒ_aux, in which ℒ_aux serves as an inductive bias and may be zero (pure behavior cloning) or a language-alignment / spatial-awareness loss. Different VLA paradigms can all be understood as instantiations of this formulation under different inductive biases.

Figure 2: Four Action-Decoding Variants — **图 2：四种代表性动作解码范式。** *StarVLA-FAST*：自回归生成离散动作 token（基于 FAST tokenizer）； *StarVLA-OFT*：通过轻量 MLP 直接回归连续动作； *StarVLA-π (π₀)*：flow matching 迭代去噪； *StarVLA-GR00T*：双系统推理（System 1 快速反应 + System 2 慢速规划）。四种变体共享相同的训练/推理接口，仅动作头实现不同。**Figure 2: Four representative action-decoding paradigms.** *StarVLA-FAST*: autoregressively generates discrete action tokens (based on the FAST tokenizer); *StarVLA-OFT*: directly regresses continuous actions through a lightweight MLP; *StarVLA-π (π₀)*: iterative denoising with flow matching; *StarVLA-GR00T*: dual-system inference (System 1 for fast reaction + System 2 for slow planning). The four variants share the same training/inference interface and differ only in the action head implementation.

统一 I/O 接口与 Server-Client 评测Unified I/O interface and server-client evaluation

StarVLA 为所有框架组件定义了两个核心方法：forward(raw_images, str, ...) 作为训练入口， predict_action(raw_images, str, ...) 作为推理入口。推理接口接受"归一化动作（均值=0，标准差=1）"并返回预测动作块（minus ground-truth 均值后的预测）。这一设计与真实机器人传感器流镜像对应，使同一 checkpoint 可直接用于仿真评测和真实部署，无需修改代码。评测基准代码通过 server-client 模式与模型推理解耦，benchmark 代码无需感知模型内部。StarVLA defines two core methods for all framework components: forward(raw_images, str, ...) as the training entry point and predict_action(raw_images, str, ...) as the inference entry point. The inference interface accepts "normalized actions (mean=0, standard deviation=1)" and returns the predicted action chunk (the prediction after subtracting the ground-truth mean). This design mirrors real robot sensor streams, so that the same checkpoint can be used directly for simulation evaluation and real-world deployment without modifying code. Benchmark code is decoupled from model inference through a server-client pattern, so the benchmark code needs no awareness of model internals.

Figure 3: Unified System Pipeline — **图 3：StarVLA 统一系统流水线。** 左侧：统一数据流，连接异构机器人数据集、pluggable dataloader 与标准化数据表示；右侧：模型前向传播路径，VL Foundation Model 骨干网络经 action head 输出动作，再经 Server-Client 接口用于仿真评测（LIBERO、SimplerEnv、RoboTwin 等）和真实机器人部署。整个流水线由 YAML 配置驱动，支持 Unified Hamiltonians、Co-train 和 Multi-Bench 训练策略。**Figure 3: The StarVLA unified system pipeline.** Left: the unified data flow, connecting heterogeneous robot datasets, a pluggable dataloader and a standardized data representation; right: the model forward path, in which the VL Foundation Model backbone emits actions through the action head, which are then used via the server-client interface for simulation evaluation (LIBERO, SimplerEnv, RoboTwin, etc.) and for real robot deployment. The whole pipeline is driven by YAML configuration and supports the Unified Hamiltonians, Co-train and Multi-Bench training strategies.

灵活训练范式Flexible training paradigms

监督行为克隆（SFT）Supervised behavior cloning (SFT)

最直接的训练方式，ℒ_aux = 0，仅优化动作预测损失。 StarVLA 将此作为建立单基准可复现基线的标准起点，并提供了 benchmark 特定的训练/评测脚本。The most direct training scheme, with ℒ_aux = 0, optimizing only the action prediction loss. StarVLA takes this as the standard starting point for establishing reproducible single-benchmark baselines, and provides benchmark-specific training/evaluation scripts.

多目标协同训练（Co-training）Multi-objective co-training

在动作预测的同时引入 VLM 辅助目标（如空间感知 grounding 损失），保留模型的语言/视觉推理能力。实验表明，空间引导协同训练可将 Google Robot 成功率从 66.1% 提升至 86.2%，同时维持 RefCOCO-g 上 71.2 IoU@0.5 的空间感知精度。VLM auxiliary objectives (such as a spatial-awareness grounding loss) are introduced alongside action prediction, preserving the language/visual reasoning ability of the model. Experiments show that spatially guided co-training raises the Google Robot success rate from 66.1% to 86.2%, while maintaining a spatial-awareness accuracy of 71.2 IoU@0.5 on RefCOCO-g.

跨体态训练（Cross-embodiment）Cross-embodiment training

通过 mixture dataloader 混合来自不同机器人平台的数据，训练单一泛化模型，实现跨 LIBERO、SimplerEnv、RoboTwin、RoboCasa-GR1 的联合训练。A mixture dataloader blends data from different robot platforms to train a single generalist model, enabling joint training across LIBERO, SimplerEnv, RoboTwin and RoboCasa-GR1.

强化学习微调（RL fine-tuning）Reinforcement learning fine-tuning

计划中的功能，目前仍在持续集成中（"an ongoing integration effort"，论文明确说明）。A planned capability that is still being integrated ("an ongoing integration effort", as the paper explicitly states).

03 实验Experiments

StarVLA 在五大主流仿真评测基准上报告了单基准专家模型（Specialist）和跨基准泛化模型（Generalist）的性能，所有结果均基于官方评测协议，以最终任务成功率为主要指标。StarVLA reports the performance of single-benchmark specialist models and cross-benchmark generalist models on five mainstream simulation benchmarks. All results follow the official evaluation protocols, taking the final task success rate as the primary metric.

单基准专家模型（Specialist）性能Single-benchmark specialist performance

Benchmark	最佳变体Best variant	StarVLA 得分StarVLA score	说明Notes
LIBERO（4 suites 平均）LIBERO (average of 4 suites)	StarVLA-OFT	96.6%	仅 30K 训练步，媲美竞品 175K+ 步Only 30K training steps, on par with competitors at 175K+ steps
SimplerEnv WidowX	StarVLA-GR00T	65.3%	success rate
RoboCasa-GR1（平均）RoboCasa-GR1 (average)	StarVLA-OFT	48.8%	average success rate
RoboTwin 2.0 clean	StarVLA-GR00T	88.0%	clean setting
RoboTwin 2.0 random	StarVLA-GR00T	88.5%	random setting

泛化模型（Generalist）vs 专家模型对比Generalist vs. specialist comparison

跨 LIBERO、SimplerEnv、RoboTwin 和 RoboCasa-GR1 联合训练的单一泛化模型：A single generalist model jointly trained across LIBERO, SimplerEnv, RoboTwin and RoboCasa-GR1:

Benchmark	Specialist	Generalist	差距Gap
LIBERO（平均）LIBERO (average)	98.8%	97.8%	-1.0%
RoboCasa-GR1（平均）RoboCasa-GR1 (average)	48.8%	57.3%	+8.5%（泛化反超！）+8.5% (generalist ahead!)

Table 2: LIBERO Results — **表 2：LIBERO 基准各变体详细结果。** StarVLA 各变体（FAST、OFT、π、GR00T）在 LIBERO-Spatial、LIBERO-Object、LIBERO-Goal、LIBERO-Long 四个子集上的成功率，与已发布的强基线进行对比。StarVLA-OFT 在平均成功率上达到 96.6%，训练开销显著低于竞争对手（30K 步 vs. 175K+ 步）。**Table 2: Detailed per-variant results on the LIBERO benchmark.** Success rates of the StarVLA variants (FAST, OFT, π, GR00T) on the four subsets LIBERO-Spatial, LIBERO-Object, LIBERO-Goal and LIBERO-Long, compared with published strong baselines. StarVLA-OFT reaches 96.6% average success rate, with training cost markedly lower than that of competitors (30K steps vs. 175K+ steps).

计算效率与多节点扩展Computational efficiency and multi-node scaling

单节点（8×A100）测试：batch size 16 时每步 0.703 秒（72.2 samples/s）； batch size 192 时每步 2.404 秒（79.9 samples/s）；GPU 利用率 74–96%。多节点扩展（最高 256 GPU）：单步延迟在 32 GPU 以上趋于饱和（约 0.93 秒/步）；样本吞吐量从 87 samples/s 线性扩展至 2,200 samples/s；并行效率在 32 GPU 以上稳定于约 79–80%。Single-node (8×A100) tests: at batch size 16, 0.703 seconds per step (72.2 samples/s); at batch size 192, 2.404 seconds per step (79.9 samples/s); GPU utilization 74-96%. Multi-node scaling (up to 256 GPUs): per-step latency saturates beyond 32 GPUs (about 0.93 seconds per step); sample throughput scales linearly from 87 samples/s to 2,200 samples/s; parallel efficiency stabilizes at about 79-80% beyond 32 GPUs.

04 局限性Limitations

说明：以下局限性部分为作者在论文中明确陈述（标注"stated"），部分为从系统设计中推断（标注"inferred"）。Note: Some of the limitations below are explicitly stated by the authors in the paper (marked "stated"), while others are inferred from the system design (marked "inferred").

RL 微调尚未完成集成（stated）RL fine-tuning is not yet fully integrated (stated)

论文明确指出强化学习微调是"an ongoing integration effort"，当前版本不支持端到端 RL 训练。这意味着在需要在线环境交互优化的任务上，StarVLA 目前仍仅限于离线行为克隆范式。The paper explicitly states that reinforcement learning fine-tuning is "an ongoing integration effort"; the current version does not support end-to-end RL training. This means that on tasks requiring optimization through online environment interaction, StarVLA is for now confined to the offline behavior cloning paradigm.

跨基准泛化模型在部分任务上略弱于专家模型（stated）The cross-benchmark generalist is slightly weaker than specialists on some tasks (stated)

Generalist 模型在 LIBERO 上比 Specialist 低约 1.0%（97.8% vs 98.8%），表明"all-in-one 多基准训练"可行但仍存在小幅性能折损，尤其在各基准数据分布差异较大时。On LIBERO the generalist model is about 1.0% below the specialist (97.8% vs 98.8%), indicating that "all-in-one multi-benchmark training" is feasible but still incurs a small performance loss, especially when the data distributions of the individual benchmarks differ substantially.

多节点扩展存在通信开销（stated）Multi-node scaling incurs communication overhead (stated)

在 8–32 GPU 区间，多节点通信开销显著，导致扩展效率低于线性。 32 GPU 以上每步延迟趋于平稳，但并行效率稳定在 79–80%，意味着约 20% 的计算资源消耗在通信同步上。In the 8-32 GPU range, multi-node communication overhead is significant, making scaling efficiency sub-linear. Beyond 32 GPUs the per-step latency levels off, but parallel efficiency stays at 79-80%, meaning that about 20% of the compute is consumed by communication and synchronization.

真实机器人验证有限（inferred）Real-robot validation is limited (inferred)

论文的实验结果均来自仿真环境（LIBERO、SimplerEnv、RoboTwin 等），对真实机器人的验证仅限于部署接口设计层面的讨论，缺乏大规模真实场景的定量评测。sim-to-real 转移能力尚未系统验证。All experimental results in the paper come from simulation environments (LIBERO, SimplerEnv, RoboTwin, etc.); validation on real robots is limited to a discussion at the level of deployment interface design, without large-scale quantitative evaluation in real scenes. Sim-to-real transfer ability has not been systematically verified.

评测基准覆盖以操作任务为主（inferred）Benchmark coverage is dominated by manipulation tasks (inferred)

当前集成的五大基准（LIBERO、SimplerEnv、RoboTwin 2.0、RoboCasa-GR1、BEHAVIOR-1K）侧重桌面操作和室内场景，移动操作、户外导航等任务尚未纳入统一评测框架。The five currently integrated benchmarks (LIBERO, SimplerEnv, RoboTwin 2.0, RoboCasa-GR1, BEHAVIOR-1K) emphasize tabletop manipulation and indoor scenes; mobile manipulation, outdoor navigation and similar tasks are not yet covered by the unified evaluation framework.