RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies

01 动机 · MotivationMotivation

现有机器人评测范式要求对任务和环境进行严格标准化，难以全面衡量具备广泛能力的通用机器人策略。随着机器人策略越来越"通用"，标准化评测反而成为瓶颈——任务覆盖窄、场景固定、无法反映真实部署多样性。Existing robot evaluation paradigms demand strict standardization of tasks and environments, which makes it hard to comprehensively measure generalist robot policies with broad capabilities. As robot policies become ever more "general", standardized evaluation turns into the bottleneck instead: narrow task coverage, fixed scenes, and an inability to reflect the diversity of real deployment.

"Tightly standardized sets of environments and tasks are not well-suited to provide comprehensive performance evaluations for policies designed for broad capability."

RoboArena 总览 — 图1：RoboArena 框架总览。分布在多个机构的评测者各自选择任务和场景，对两个策略进行双盲成对评测，汇总偏好反馈后通过 Bradley-Terry 模型导出全局策略排名。与集中式方法不同，此框架无需标准化环境，天然覆盖真实世界的多样性。Figure 1: Overview of the RoboArena framework. Evaluators distributed across multiple institutions each pick their own tasks and scenes, run a double-blind pairwise evaluation of two policies, and the aggregated preference feedback yields a global policy ranking through a Bradley-Terry model. Unlike centralized approaches, this framework requires no standardized environments and naturally covers real-world diversity.

4284总 evaluation episodesTotal evaluation episodes

600+成对双盲比较 episodesPairwise double-blind comparison episodes

7参与机构数Participating institutions

94.6%VLM 任务分类准确率（448 样本）VLM task classification accuracy (448 samples)

02 方法 · MethodMethod

RoboArena 将评测分为三个核心模块：分布式评测协议（双盲成对实验）、任务感知排名算法（扩展 Bradley-Terry 模型）、定性分析流水线（VLM + LLM 自动生成策略特征报告）。RoboArena decomposes evaluation into three core modules: a distributed evaluation protocol (double-blind pairwise experiments), a task-aware ranking algorithm (an extended Bradley-Terry model), and a qualitative analysis pipeline (VLM + LLM automatically generating policy characterization reports).

图4：RoboArena 系统架构。由策略推理服务器（policy inference servers）、评测客户端（evaluation clients）、评测数据库（evaluation database）和中央评测服务器（central evaluation server）组成。评测者从中央服务器请求策略对，在本地场景中依次执行两个策略，并提交三类反馈：连续进度分（progress score ∈ [0…100]）、二元偏好标签（binary preference label）和自由文本解释（free-form explanation）。Figure 4: The RoboArena system architecture. It consists of policy inference servers, evaluation clients, an evaluation database, and a central evaluation server. An evaluator requests a policy pair from the central server, executes the two policies in turn in a local scene, and submits three kinds of feedback: a continuous progress score (progress score ∈ [0…100]), a binary preference label, and a free-form explanation.

双盲分布式评测协议Double-blind distributed evaluation protocol

评测者在自选的场景和任务中对两个匿名策略（πA 和 πB）依次执行 rollout，双盲设计防止评测者对特定策略产生偏见。反馈包含三个维度：连续进度分、二元偏好以及自由文本说明，三种信号相互印证，提升排名可靠性。In scenes and tasks of their own choosing, evaluators execute rollouts of two anonymized policies (πA and πB) in turn; the double-blind design keeps evaluators from developing a bias toward a particular policy. Feedback covers three dimensions: a continuous progress score, a binary preference, and a free-form explanation. The three signals corroborate one another and improve the reliability of the ranking.

任务感知 Bradley-Terry 排名算法Task-aware Bradley-Terry ranking algorithm

标准 Bradley-Terry 模型仅建模策略绝对能力，忽略任务难度差异。RoboArena 引入三类参数：策略对数能力 θ、任务难度 τ、以及策略-任务偏置 ψ，将胜出概率建模为：The standard Bradley-Terry model captures only the absolute capability of a policy and ignores differences in task difficulty. RoboArena introduces three kinds of parameters: policy log-capability θ, task difficulty τ, and a policy-task bias ψ, and models the winning probability as:

"p(πA > πB) = ∑_t=1^T ν_t · σ(θ_A + ψ_At − τ_t) · (1 − σ(θ_B + ψ_Bt − τ_t))"

参数通过近似最大似然期望最大化（approximate maximum likelihood expectation-maximization, EM）算法拟合，能够在样本稀疏时显著提升排名质量。The parameters are fit with an approximate maximum likelihood expectation-maximization (EM) algorithm, which substantially improves ranking quality when samples are sparse.

图2：定性分析流水线。使用视觉语言模型（VLM）对评测场景和任务进行分类，再用大语言模型（LLM）跨评测 rollout 聚合信息，最终自动生成每个策略的特征报告，辅助研究者理解策略的优势和薄弱环节。VLM 任务分类在 448 个样本上达到 94.6% 准确率。Figure 2: The qualitative analysis pipeline. A vision-language model (VLM) classifies the evaluation scenes and tasks, a large language model (LLM) then aggregates information across evaluation rollouts, and a characterization report is finally generated automatically for each policy, helping researchers understand where a policy is strong and where it is weak. VLM task classification reaches 94.6% accuracy on 448 samples.

策略池与硬件平台Policy pool and hardware platform

本次评测以 DROID 平台（Franka 机械臂 + Robotiq 夹爪 + 双目相机）为基础，评测七个通用策略变体，涵盖 π0-flow、π0-FAST、PG-flow、PG-FAST、PG-FAST+、PG-FSQ、PG-Bin，系统性覆盖了不同 action tokenization 和 flow matching 配置。This evaluation is built on the DROID platform (a Franka arm + a Robotiq gripper + a stereo camera) and assesses seven generalist policy variants, covering π0-flow, π0-FAST, PG-flow, PG-FAST, PG-FAST+, PG-FSQ, and PG-Bin, so as to systematically span different action tokenization and flow matching configurations.

03 实验 · ExperimentsExperiments

在七所机构收集共 4284 个 evaluation episodes，其中超过 600 个为成对双盲对比。将 Oracle 排名（由全量 episode 得出）作为参考标准，与传统集中式方法和多种排名算法对比。A total of 4284 evaluation episodes were collected at seven institutions, of which more than 600 are pairwise double-blind comparisons. The Oracle ranking, derived from the full set of episodes, serves as the reference standard for comparison against conventional centralized evaluation and a range of ranking algorithms.

排名相关性对比 — 图6：RoboArena 排名与 Oracle 排名的相关性。任务感知 Bradley-Terry 排名（RoboArena）与 Oracle 的 Kendall-τ 相关性显著高于传统集中式评测方法；MMRV（Mean Maximum Rank Violation）指标同样支持任务感知方法优于普通 Bradley-Terry 和简单平均成功率等基线。Figure 6: Correlation between the RoboArena ranking and the Oracle ranking. The Kendall-τ correlation with the Oracle of the task-aware Bradley-Terry ranking (RoboArena) is markedly higher than that of conventional centralized evaluation; the MMRV (Mean Maximum Rank Violation) metric likewise favors the task-aware method over baselines such as plain Bradley-Terry and simple average success rate.

排名方法Ranking method	与 Oracle 相关性Correlation with Oracle	MMRV ↓
传统集中式评测（平均成功率）Conventional centralized evaluation (average success rate)	较低Lower	较高Higher
Elo 在线更新Elo online update	中等Moderate	中等Moderate
Bradley-Terry（标准）Bradley-Terry (standard)	中等Moderate	中等Moderate
RoboArena（任务感知 BT）RoboArena (task-aware BT)	最高Highest	最低Lowest

图7：样本效率分析。RoboArena 在仅进行约 100 次成对比较后即收敛到高质量排名，与需要相同 episode 数量的传统评测方法相当，展示了分布式成对评测在样本效率上的显著优势。Figure 7: Sample efficiency analysis. RoboArena converges to a high-quality ranking after only about 100 pairwise comparisons, on par with conventional evaluation methods that need the same number of episodes, which demonstrates the marked sample-efficiency advantage of distributed pairwise evaluation.

定性分析验证Validation of the qualitative analysis

对 LLM 自动生成的策略报告进行人工校验：对于报告中声称"π0-FAST-DROID 优于、持平或劣于其他策略"的大多数类别，这些结论均与评测数据中的实际胜率一致，证明了定性分析流水线的可靠性。The policy reports generated automatically by the LLM were verified manually: for most of the categories in which a report claims that "π0-FAST-DROID is better than, on par with, or worse than another policy", the conclusions agree with the actual win rates in the evaluation data, which attests to the reliability of the qualitative analysis pipeline.

消融实验Ablation study

实验对比了四种排名算法（Elo、标准 Bradley-Terry MLE、基于进度分的 PROG 排名、任务感知 Bradley-Terry），发现任务感知扩展在各指标上均优于其他变体；进度分排名（PROG）也有一定效果，但精细程度不及任务感知方法。The experiments compare four ranking algorithms (Elo, standard Bradley-Terry MLE, progress-score-based PROG ranking, and task-aware Bradley-Terry) and find that the task-aware extension outperforms the other variants on every metric; progress-score ranking (PROG) is also effective to a degree, but is less fine-grained than the task-aware method.

04 局限性 · LimitationsLimitations

Note: 以下局限性均为论文作者明确陈述（stated by the authors）。All limitations below are explicitly stated by the authors.

跨平台/跨体态扩展（Cross-embodiment）Cross-platform / cross-embodiment extension

当前评测集中在 DROID 平台（Franka 机械臂），尚未覆盖跨体态通用策略。作者将跨平台扩展列为未来工作。The current evaluation is concentrated on the DROID platform (a Franka arm) and does not yet cover cross-embodiment generalist policies. The authors list cross-platform extension as future work.

难以隔离变量（Controlled Experimentation）Difficulty of isolating variables (controlled experimentation)

去中心化设计使"单变量对照实验"变得困难——不同机构使用不同场景和任务，无法像集中式评测那样精确控制实验条件。The decentralized design makes "single-variable controlled experiments" difficult: different institutions use different scenes and tasks, so experimental conditions cannot be controlled as precisely as in centralized evaluation.

对抗性评测者鲁棒性未验证（Adversarial Evaluators）Robustness to adversarial evaluators not verified

作者坦言"尚未研究该系统对故意对抗性评测者的鲁棒性（have not investigated its robustness to intentionally adversarial evaluators）"，系统在恶意操纵下的表现有待进一步分析。The authors state frankly that they "have not investigated its robustness to intentionally adversarial evaluators"; how the system behaves under malicious manipulation awaits further analysis.

Goodhart 定律风险（Over-optimization）Goodhart's law risk (over-optimization)

当评测指标本身成为优化目标时，它可能不再是真实能力的良好代理（"a measure ceases to be a good measure when it becomes a target"）。作者认为当前策略性能有限使得过度优化不太可能，但随着策略能力提升，这一风险会增大。Once an evaluation metric itself becomes an optimization target, it may cease to be a good proxy for true capability ("a measure ceases to be a good measure when it becomes a target"). The authors argue that the limited performance of current policies makes over-optimization unlikely, but the risk grows as policy capability improves.