HoloBrain-0 是一个视觉-语言-动作（VLA）基础模型，将多视角相机参数与机器人运动学信息融入架构设计，并结合测试驱动的迭代数据采集策略，大幅提升了机器人在真实世界与跨仿真基准上的操作性能。提供 0.2B 和 1.1B 两个规模的开源预训练模型及完整的 RoboOrchard 基础设施。HoloBrain-0 is a vision-language-action (VLA) foundation model that integrates multi-view camera parameters and robot kinematics information into its architectural design and combines them with a test-driven iterative data collection strategy, substantially improving robot manipulation performance in the real world and across simulation benchmarks. Open-source pretrained models at two scales, 0.2B and 1.1B, are released together with the complete RoboOrchard infrastructure.
具身智能Embodied AIVLA机器人操作Robot manipulationcross-embodimentimitation learning策略学习Policy learning多视角感知Multi-view perception测试驱动数据Test-driven data
01动机Motivation
通用机器人智能体的研发面临三大核心挑战：真实部署时出现的分布外状态（out-of-distribution states）、大规模收集高质量专家示范的高昂成本，以及在低延迟控制中部署大型模型的困难。现有 VLA 方法通常忽略相机内外参、机器人运动学等具身信息，导致在跨场景、跨具身迁移时表现下降。Developing general-purpose robotic agents faces three core challenges: out-of-distribution states that arise during real-world deployment, the high cost of collecting large-scale, high-quality expert demonstrations, and the difficulty of deploying large models under low-latency control. Existing VLA methods typically ignore embodiment information such as camera intrinsics and extrinsics and robot kinematics, which degrades performance when transferring across scenes and across embodiments.
"developing a truly general robotic agent remains a significant challenge" — 原文动机陈述，指出现有方法在现实部署的鲁棒性与泛化性方面仍存在显著差距。"developing a truly general robotic agent remains a significant challenge" — the motivating statement of the paper, pointing out that existing methods still leave a significant gap in robustness and generalization for real-world deployment.
图 1：HoloBrain-0 系统总览。模型将多视角 RGB 图像、机器人关节状态与 6D 姿态作为输入，通过 Perspective-aware Spatial Enhancer 将 2D 特征投影到统一 3D 坐标系，再经 Embodiment-Aware Action Expert 输出关节角残差与末端位姿位移，支持双臂 Piper 等多种具身形态。Figure 1: Overview of the HoloBrain-0 system. The model takes multi-view RGB images, robot joint states and 6D poses as input, projects 2D features into a unified 3D coordinate frame through the Perspective-aware Spatial Enhancer, and then outputs joint angle residuals and end-effector pose displacements through the Embodiment-Aware Action Expert, supporting a variety of embodiments such as the dual-arm Piper.
92.3%RoboTwin 2.0 随机化基准成功率（HB-QW 1.1B）Success rate on the RoboTwin 2.0 randomized benchmark (HB-QW 1.1B)
97.4%LIBERO 基准平均成功率（HB-QW）Average success rate on the LIBERO benchmark (HB-QW)
74.0%LIBERO-Plus 零样本鲁棒性 SOTA（HB-GD）SOTA zero-shot robustness on LIBERO-Plus (HB-GD)
77.2%真实世界任务成功率（HB-QW，10 任务均值）Real-world task success rate (HB-QW, mean over 10 tasks)
02方法Method
HoloBrain-0 由三个核心部分构成：① Perspective-aware Spatial Enhancer（PSE）将多视角图像特征统一投影至 3D 空间；② Embodiment-Aware Action Expert 编码具身运动学状态并生成混合动作输出；③ 迭代测试驱动数据策略（Test-Driven Data Strategy），在测试中自动发现失败模式并针对性采集恢复轨迹。HoloBrain-0 consists of three core parts: ① the Perspective-aware Spatial Enhancer (PSE), which projects multi-view image features into a unified 3D space; ② the Embodiment-Aware Action Expert, which encodes embodiment kinematic states and produces hybrid action outputs; and ③ an iterative Test-Driven Data Strategy, which automatically discovers failure modes during testing and collects targeted recovery trajectories.
图 2：HoloBrain-0 的输入状态表征与输出动作空间。输入包含多视角图像和机器人关节角（对夹爪关节角做掩码，仅保留 6D 姿态），输出为每个关节的残差增量：关节角残差（弧度）与末端连杆姿态位移（平移单位为米，旋转以四元数表示）。Figure 2: The input state representation and output action space of HoloBrain-0. The input contains multi-view images and robot joint angles (gripper joint angles are masked, keeping only the 6D pose), and the output is a residual increment for every joint: joint angle residuals (in radians) and end-link pose displacements (translation in meters, rotation expressed as quaternions).
PSE 利用相机内外参与深度图，将多视角 2D 图像特征投影到统一的 3D 坐标系。关键设计是将 3D 投影坐标系从机器人本体基座坐标系切换至固定中心相机坐标系，从而支持跨具身（cross-embodiment）训练——不同机器人的传感器布局不同，但统一到相机帧后可共享特征空间。PSE uses camera intrinsics and extrinsics together with depth maps to project multi-view 2D image features into a unified 3D coordinate frame. The key design choice is switching the 3D projection frame from the robot's own base frame to a fixed central camera frame, which enables cross-embodiment training — different robots have different sensor layouts, but once unified in the camera frame they can share the same feature space.
Embodiment-Aware Action Expert
动作专家模块对机器人状态进行编码：对夹爪关节角做掩码（masked），仅保留 6D 姿态信息，以实现跨具身兼容。输出动作为："a concatenation of joint angle residuals (in radians) and link pose displacements (in meters and quaternions)"，即同时预测关节角增量和连杆末端位姿增量，格式为 [Δθ, Δx, Δy, Δz, Δqw, Δqx, Δqy, Δqz]。训练损失包含关节位置损失、末端姿态损失、正向运动学姿态损失和深度损失四项，均采用 smooth L1 距离以缓解极端误差带来的训练不稳定。The action expert module encodes the robot state: gripper joint angles are masked, keeping only the 6D pose information, so as to achieve cross-embodiment compatibility. The output action is "a concatenation of joint angle residuals (in radians) and link pose displacements (in meters and quaternions)", i.e. it predicts both joint angle increments and end-link pose increments, in the format [Δθ, Δx, Δy, Δz, Δqw, Δqx, Δqy, Δqz]. The training loss comprises four terms — a joint position loss, an end-effector pose loss, a forward-kinematics pose loss and a depth loss — all using smooth L1 distance to mitigate the training instability caused by extreme errors.
图 3：HoloBrain-0 模型架构细节，展示了 Action Expert 中关节状态编码、multi-head attention 与输出预测头的具体连接方式。HB-GD（0.2B）与 HB-QW（1.1B）共享相同的 Action Expert（20.79M 参数），主要区别在于视觉编码器与语言模型的规模。Figure 3: Architectural details of the HoloBrain-0 model, showing how joint state encoding, multi-head attention and the output prediction heads are connected inside the Action Expert. HB-GD (0.2B) and HB-QW (1.1B) share the same Action Expert (20.79M parameters); the main difference lies in the scale of the vision encoder and the language model.
SimpleRTC 与 Teacher-forcing 训练SimpleRTC and Teacher-forcing Training
推理阶段采用 SimpleRTC：一种无梯度的软掩码（soft-masking）推理策略，将上一时刻未执行完毕的动作预测与新预测加权融合，实现异步、低延迟控制，相比同步基线可降低约 30–40% 的执行时间。训练阶段引入 Teacher Forcing：动态地将前 N 步输入噪声替换为真实动作，N 服从泊松分布采样，该比例从 0% 升至 50% 时成功率从约 70% 持续提升至约 87%。At inference the model adopts SimpleRTC: a gradient-free soft-masking inference strategy that blends the not-yet-executed action predictions from the previous step with the new predictions in a weighted manner, achieving asynchronous, low-latency control and reducing execution time by roughly 30–40% compared with a synchronous baseline. During training, Teacher Forcing is introduced: the first N steps of input noise are dynamically replaced with ground-truth actions, with N sampled from a Poisson distribution; as this ratio rises from 0% to 50%, the success rate improves steadily from about 70% to about 87%.
迭代测试驱动数据策略Iterative Test-Driven Data Strategy
后训练数据采集分两个阶段：① 主动状态扩展（Proactive State Expansion）：系统化改变光照、背景、物体实例，并将全任务采集拆解为子任务；② 测试驱动失败恢复（Test-Driven Failure Recovery）：在每轮测试中分析失败模式，针对性采集 2–3 秒的恢复轨迹，形成闭环迭代。Post-training data collection proceeds in two stages: ① Proactive State Expansion, which systematically varies lighting, backgrounds and object instances and decomposes whole-task collection into subtasks; and ② Test-Driven Failure Recovery, which analyses failure modes in each round of testing and collects targeted recovery trajectories of 2–3 seconds, forming a closed-loop iteration.
03实验Experiments
评测覆盖真实世界双臂 Piper 机器人（10 项任务，每项 20 次试验）和三个仿真基准：RoboTwin 2.0（50 任务，每任务 100 次）、LIBERO（40 任务，每任务 50 次）、GenieSim 2.2（10 人形机器人任务）。主要对比基线为 π0 和 π0.5。提供 HoloBrain-0-GD（0.2B，适合端侧部署）和 HoloBrain-0-QW（1.1B）两个变体。The evaluation covers a real-world dual-arm Piper robot (10 tasks, 20 trials each) and three simulation benchmarks: RoboTwin 2.0 (50 tasks, 100 trials each), LIBERO (40 tasks, 50 trials each) and GenieSim 2.2 (10 humanoid robot tasks). The main comparison baselines are π0 and π0.5. Two variants are provided: HoloBrain-0-GD (0.2B, suitable for on-device deployment) and HoloBrain-0-QW (1.1B).
HoloBrain-0-GD 平均进度得分 88.07%、成功率 74.81%；HoloBrain-0-QW 平均进度得分 87.32%、成功率 77.18%，相比 π0.5 分别提升 +5.65% 和 +8.02%。代表性任务结果（成功率）：HoloBrain-0-GD reaches an average progress score of 88.07% and a success rate of 74.81%; HoloBrain-0-QW reaches an average progress score of 87.32% and a success rate of 77.18%, improving over π0.5 by +5.65% and +8.02% respectively. Representative per-task results (success rate):
任务Task
π0
π0.5
HoloBrain-0
Fold towel（折叠毛巾）Fold towel
31.58%
63.16%
84.21%
Place shoe（放置鞋子）Place shoe
48.39%
54.84%
96.77%
Fold clothes（折衣服，长时程）Fold clothes (long-horizon)
15%
50%
75%
Fold paper box（折纸盒，长时程）Fold paper box (long-horizon)
图 4：真实世界评测任务套件，包含 7 项基础操作任务（fold towel、place empty cup、place shoe、stack blocks/bowls three、put bottles basket）、2 项长时程任务（fold clothes、fold paper box）和 1 项泛化抓取任务（Grasp Anything，测试未见物体的抓取泛化能力，未见物体成功率 97.5%，高于已见物体的 93.5%）。Figure 4: The real-world evaluation task suite, comprising 7 basic manipulation tasks (fold towel, place empty cup, place shoe, stack blocks/bowls three, put bottles basket), 2 long-horizon tasks (fold clothes, fold paper box) and 1 generalized grasping task (Grasp Anything, which tests grasp generalization to unseen objects; the success rate on unseen objects is 97.5%, higher than the 93.5% obtained on seen objects).
消融实验Ablation Studies
与 Grasp Anything 任务协同训练的效果（7 项任务均值）：不加入协同训练时平均成功率为 72.40%，加入后提升至 75.00%（+2.6 points），各任务提升幅度 0–13.33 个百分点。Effect of co-training with the Grasp Anything task (mean over 7 tasks): without co-training the average success rate is 72.40%, and with it the rate rises to 75.00% (+2.6 points), with per-task gains ranging from 0 to 13.33 percentage points.
图 5：SimpleRTC 与 Teacher Forcing 比例的消融实验（布料折叠任务）。同步基线约 60% 成功率；加入 SimpleRTC 且 Teacher Forcing 比例为 0% 时升至约 70%；比例 25% 时约 85%；50% 时约 87%。异步 SimpleRTC 相比同步基线减少约 30–40% 执行时间。Figure 5: Ablation on SimpleRTC and the Teacher Forcing ratio (cloth folding task). The synchronous baseline reaches about 60% success; adding SimpleRTC with a Teacher Forcing ratio of 0% raises it to about 70%; a ratio of 25% gives about 85%; and 50% gives about 87%. Asynchronous SimpleRTC reduces execution time by roughly 30–40% compared with the synchronous baseline.
04局限性Limitations
注： 论文未设专门 Limitations 章节；以下各点均由论文原文陈述（stated）整理，并在括号中注明出处章节。Note: The paper has no dedicated Limitations section; the points below are compiled from statements made in the paper itself, with the source section indicated in parentheses.
仿真到真实的迁移差距依然存在（Sim-to-Real Gap）The sim-to-real transfer gap still persists (Sim-to-Real Gap)
论文原文承认："Although the sim-to-real gap persists, simulation benchmarks remain essential"（Section Experiments）。尽管 RoboTwin 2.0 等仿真基准上表现出色，真实世界部署中仍会出现新的分布外状态，需持续迭代数据采集加以弥补。The paper acknowledges: "Although the sim-to-real gap persists, simulation benchmarks remain essential" (Section Experiments). Despite the strong results on simulation benchmarks such as RoboTwin 2.0, new out-of-distribution states still appear during real-world deployment and must be compensated for by continued iterative data collection.
真实数据采集成本高（Data Collection Cost）Real-world data collection is expensive (Data Collection Cost)
论文指出："The efficacy of post-training is often bottlenecked by the high cost of collecting high-quality, real-world data"。测试驱动策略虽有效降低采集需求，但每轮仍需人工遥操作录制恢复轨迹（约 2–3 秒/次），规模化时成本仍然显著。The paper notes: "The efficacy of post-training is often bottlenecked by the high cost of collecting high-quality, real-world data". Although the test-driven strategy effectively lowers collection demand, each round still requires human teleoperation to record recovery trajectories (about 2–3 seconds each), so the cost remains significant at scale.
精确指令跟随能力尚待评估（Instruction Following）Precise instruction following is still to be evaluated (Instruction Following)
论文原文指出："We observe that precise instruction following is still an under-evaluated ability in current VLA research"，并将开发更严格的语言指令跟随基准列为未来工作方向，当前模型在区分语义相近指令时仍有提升空间。The paper states: "We observe that precise instruction following is still an under-evaluated ability in current VLA research", and lists the development of stricter language instruction-following benchmarks as a direction for future work; the current model still has room for improvement in distinguishing semantically similar instructions.
未集成离线强化学习（Off-Policy RL）Off-policy reinforcement learning is not yet integrated (Off-Policy RL)
当前 HoloBrain-0 仅采用模仿学习（imitation learning），论文在 Future Work 中明确提到将集成 off-policy 强化学习与价值模型（value models），以进一步突破专家示范的性能上限。HoloBrain-0 currently relies solely on imitation learning; in Future Work the paper explicitly mentions integrating off-policy reinforcement learning and value models, in order to further break through the performance ceiling set by expert demonstrations.
