RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies
首个面向 VLA 模型的大规模真实机器人在线评测平台The first large-scale online real-robot evaluation platform for VLA models
Adina Yakefu, Bin Xie, Chongyang Xu 等 37 位作者 · RoboChallenge TeamAdina Yakefu, Bin Xie, Chongyang Xu and 37 authors in total · RoboChallenge Team
现有 VLA 模型评测大多依赖仿真或有限的真实机器人实验，缺乏标准化、可重现的大规模基准。
RoboChallenge 构建了一套在线真实机器人评测基础设施，配备 10 台跨 4 种平台的机器人，
并推出 Table30 基准（30 项桌面操作任务），系统地测试 π₀、CogACT 等主流 VLA 模型的真实能力。Existing evaluations of VLA models mostly rely on simulation or on limited real-robot experiments, and lack a standardized, reproducible large-scale benchmark. RoboChallenge builds an online real-robot evaluation infrastructure equipped with 10 robots spanning 4 platforms, and introduces the Table30 benchmark (30 tabletop manipulation tasks) to systematically test the real capabilities of mainstream VLA models such as π₀ and CogACT.
真实机器人测试是验证机器人控制算法的不可或缺环节，但现有评测体系面临三大挑战：
人工测试者偏差（同一任务因测试者不同，成功率可从 0% 波动至 100%）、
可重现性差（初始场景状态难以精确还原）、
以及规模化成本高（跨机器人平台、跨任务的大规模测试难以实施）。Testing on real machines is an indispensable step for validating robotic control algorithms, yet existing evaluation practice faces three major challenges: human tester bias (on the same task the success rate can swing from 0% to 100% depending on the tester), poor reproducibility (the initial scene state is hard to restore precisely), and the high cost of scaling up (large-scale testing across robot platforms and across tasks is hard to carry out).
"Testing on real machines is indispensable for robotic control algorithms."
图 2：Table30 基准的 30 项任务缩略图（左）与各 baseline 模型排行榜（右）。
涵盖精细三维定位、多阶段操作、时序推理、双臂协作及软体操作等多种难度维度。Figure 2: Thumbnails of the 30 tasks of the Table30 benchmark (left) and the leaderboard of the baseline models (right). They cover difficulty dimensions such as precise 3D localization, multi-stage manipulation, temporal reasoning, bimanual coordination and soft-body manipulation.
10台真实机器人（4 种平台）real robots (4 platforms)
30Table30 基准任务Table30 benchmark tasks
43.7%最优模型 π₀.₅ 平均 Success Rateaverage Success Rate of the best model π₀.₅
5%时序推理任务平均 SR（最难类别）average SR on temporal-reasoning tasks (hardest category)
02方法Method
RoboChallenge 采用"远程机器人（remote robot）"范式：参赛者在本地运行推理，通过低层 API
获取精确时间戳的传感器观测，并将动作指令写入机器人的 FIFO 队列，无需提交 Docker 镜像或模型权重，
彻底规避软件栈兼容性问题，同时通过 Visual Task Reproduction 机制保证场景初始状态的高度一致性。RoboChallenge adopts a "remote robot" paradigm: participants run inference locally, obtain precisely timestamped sensor observations through a low-level API, and write action commands into the robot's FIFO queue. No Docker image or model weights have to be submitted, which completely avoids software-stack compatibility problems, while the Visual Task Reproduction mechanism guarantees a highly consistent initial scene state.
图 1：RoboChallenge 在线服务架构。低层 API 提供精确时间戳的 RGB、深度及本体感知观测，
并通过 FIFO 动作队列支持异步执行。用户无需对外开放公网 API，模型在本地硬件推理。Figure 1: The online service architecture of RoboChallenge. The low-level API provides precisely timestamped RGB, depth and proprioceptive observations, and supports asynchronous execution through a FIFO action queue. Users do not need to expose a public API; the model runs inference on local hardware.
机器人平台Robot platforms
系统部署 4 种机器人平台，共 10 台：The system deploys 4 robot platforms, 10 robots in total:
UR5：6-DOF 单臂，Robotiq 夹爪，RTDE 接口控制UR5: 6-DOF single arm, Robotiq gripper, controlled through the RTDE interface
Franka Panda：7-DOF 单臂，Robotiq 夹爪，libfranka 驱动Franka Panda: 7-DOF single arm, Robotiq gripper, driven by libfranka
Cobot Magic Aloha：双 6-DOF 臂，安装于移动平台Cobot Magic Aloha: dual 6-DOF arms, mounted on a mobile platform
ARX-5：6-DOF 单臂，CAN 总线驱动ARX-5: 6-DOF single arm, driven over the CAN bus
所有机器人均配备 Intel RealSense RGBD 相机（主视角、腕部、侧视角三路），提供 RGB + 深度 + 本体感知多模态观测。
系统可提供每任务最多 1,000 条示教轨迹（存储于 Hugging Face，JSON 格式）。All robots are equipped with Intel RealSense RGBD cameras (three streams: main view, wrist and side view), providing multimodal RGB + depth + proprioceptive observations. The system can provide up to 1,000 demonstration trajectories per task (stored on Hugging Face, in JSON format).
为解决测试者偏差问题，系统采用参考图像叠加方案：将任务开始时的参考图像实时叠加在摄像头实时画面上，
要求测试者调整场景直至当前观测与参考图像精确匹配。论文将"Adaptive tester"（模型作者）识别出任务
"sweet spot" 从而显著抬高成功率的现象称为 "Sweet-spot Effect"。
该机制有效将不同测试者间的成功率差异压缩至可接受范围。To resolve the tester-bias problem, the system uses a reference-image overlay scheme: the reference image of the task start state is overlaid in real time on the live camera view, and the tester is required to adjust the scene until the current observation matches the reference image precisely. The paper calls the phenomenon in which an "Adaptive tester" (the model author) identifies the "sweet spot" of a task and thereby markedly raises the success rate the "Sweet-spot Effect". This mechanism effectively compresses the success-rate spread across different testers to an acceptable range.
图 3：同一任务、不同类型测试者（experienced / ignorant / adaptive）导致的成功率剧烈波动。Figure 3: Drastic fluctuation of the success rate on one and the same task caused by different types of testers (experienced / ignorant / adaptive).图 5：Visual Task Reproduction 用户界面。参考图像半透明叠加于实时摄像头画面，
测试者须精确还原初始场景状态。Figure 5: The Visual Task Reproduction user interface. The reference image is overlaid semi-transparently on the live camera view, and the tester must restore the initial scene state precisely.
评分机制Scoring scheme
每项任务分解为若干阶段，满分 10 分，共 10 次 rollout，总分上限 100 分。
每次重试扣除 0.5 分惩罚。基准设有 benchmark protocol（评估单一模型的稳定性）
和 comparative protocol（多模型公平排名）两种评测模式。Each task is decomposed into several stages with a full score of 10 points, and 10 rollouts are run, so the total score is capped at 100 points. Every retry incurs a penalty of 0.5 points. The benchmark defines two evaluation modes: a benchmark protocol (assessing the stability of a single model) and a comparative protocol (fair ranking across multiple models).
图 10：RoboChallenge 面向参赛者的完整工作流：本地推理 → API 调用 → 机器人执行 → 结果反馈。Figure 10: The complete workflow of RoboChallenge for participants: local inference → API call → robot execution → result feedback.
03实验Experiments
论文在 Table30 上评测了 5 种主流 VLA 实现，覆盖 Task-specific（全量任务数据训练）
与 Generalist（每任务 50 条混合训练）两种训练设置，以 Success Rate (SR) 和 Progress Score 为主要指标。The paper evaluates 5 mainstream VLA implementations on Table30, covering two training settings — Task-specific (trained on the full data of the task) and Generalist (mixed training with 50 trajectories per task) — and uses Success Rate (SR) and Progress Score as the main metrics.
主要结果（30 任务平均）Main results (averaged over the 30 tasks)
模型Model
训练设置Training setting
Success Rate (%)
Progress Score
π₀.₅
Task-specific
43.7
62.2
π₀
Task-specific
28.3
47.6
CogACT (Microsoft)
Task-specific
11.7
21.8
π₀.₅
Generalist
17.7
31.3
π₀
Generalist
9.3
20.6
按任务难度标签分析Analysis by task difficulty tag
论文为 30 项任务打上难度标签，揭示 VLA 当前能力边界（所有模型平均）：The paper assigns difficulty tags to the 30 tasks, revealing the current capability boundary of VLA models (averaged over all models):
难度标签Difficulty tag
平均 Success Rate (%)Average Success Rate (%)
平均 Progress ScoreAverage Progress Score
temporal dependence（时序依赖）temporal dependence
5
14
soft body（软体操作）soft body
3
8
multiview（多视角）multiview
5
21
bimanual（双臂协作）bimanual
8
20
precise 3D localization（精细定位）precise 3D localization
12
18
simple pick-and-place（简单抓放）simple pick-and-place
4
42
全任务平均All-task average
22
37
图 7：Table30 任务分布。按 VLA 难度、机器人类型、场景位置、操作对象类型四个维度分别标注。Figure 7: Task distribution of Table30, annotated along four dimensions: VLA difficulty, robot type, scene location and type of manipulated object.图 8：各模型在所有任务上的 SR 与 Progress Score 累积分布。π₀.₅ 在高百分位段表现显著领先。Figure 8: Cumulative distributions of SR and Progress Score of every model over all tasks. π₀.₅ leads markedly in the high-percentile range.
模型排行与典型任务结果Model ranking and representative task results
图 2（右）：各 baseline 模型排行榜。π₀.₅ Task-specific 以 43.7% SR 位居第一；
典型任务中 "stack bowls" 达 100% SR / 99.5 score，而 "fold dishcloth" 仅 20% SR / 24 score。Figure 2 (right): Leaderboard of the baseline models. π₀.₅ Task-specific ranks first with 43.7% SR; among the representative tasks, "stack bowls" reaches 100% SR / 99.5 score, whereas "fold dishcloth" reaches only 20% SR / 24 score.
鲁棒性分析Robustness analysis
论文验证了 VLA 对视觉扰动的鲁棒性：在输入图像上施加背景替换、遮挡等增强后，
模型输出动作几乎不变，说明 VLA 已具备一定视觉不变性，
环境光照与相机漂移等非受控因素对评测的影响在可接受范围内。The paper verifies the robustness of VLA models to visual perturbations: after applying augmentations such as background replacement and occlusion to the input images, the actions output by the models barely change, which shows that VLA models already possess a certain visual invariance, and that uncontrolled factors such as ambient lighting and camera drift affect the evaluation only within an acceptable range.
04局限性Limitations
Note：以下局限性均为论文作者明确陈述（stated），部分作者同时指出当前未观测到负面影响。Note: All the limitations below are explicitly stated by the authors of the paper; for some of them the authors also point out that no negative effect has been observed so far.
模型诚信性无法核验（Model Integrity）Model integrity cannot be verified (Model Integrity)
采用"用户本地推理"范式的核心代价是：
"we have no means to check whether the model actually run by the user matches the user's claim."
恶意用户可替换模型或引入人工辅助（human-in-the-loop cheating），系统当前无技术手段防范。The core price of the "inference on the user's own machine" paradigm is: "we have no means to check whether the model actually run by the user matches the user's claim." A malicious user could swap the model or bring in human assistance (human-in-the-loop cheating), and the system currently has no technical means to prevent it.
测试集过拟合风险（Test-set Overfitting）Risk of test-set overfitting (Test-set Overfitting)
固定的参考图像场景存在被针对性过拟合的风险。论文指出
"There is a chance that the model submissions 'overfit' to the particular reference test cases"，
但实验中目前未观测到此现象。The fixed reference-image scenes carry the risk of being deliberately overfitted. The paper notes "There is a chance that the model submissions 'overfit' to the particular reference test cases", but this phenomenon has not been observed in the experiments so far.
当前 VLA 均为单帧模型（Single-frame Models）Current VLA models are all single-frame models (Single-frame Models)
所有被测 VLA 均为 single-frame 推理，无法处理时序依赖任务
（"identical images may be received on different stages"），
这直接导致 temporal dependence 类任务平均成功率仅 5%，是最大能力短板之一。All evaluated VLA models perform single-frame inference and cannot handle temporally dependent tasks ("identical images may be received on different stages"), which directly leads to an average success rate of only 5% on the temporal dependence category — one of the largest capability gaps.
被测模型均工作在 224×224 低分辨率下，对精细三维定位任务（precise 3D localization）
造成显著影响，该类任务平均 SR 仅 12%。All evaluated models operate at a low resolution of 224×224, which significantly affects precise 3D localization tasks; the average SR of this category is only 12%.
传感器套件不完整（No Force/Torque Sensors）Incomplete sensor suite (No Force/Torque Sensors)
为降低系统复杂度，当前平台省略了力矩传感器。论文承认这对接触丰富（contact-rich）任务
的精细操作能力有所限制，未来版本可能补充。To reduce system complexity, the current platform omits force/torque sensors. The paper admits that this limits fine manipulation capability on contact-rich tasks, and that future versions may add them.
