CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation

01 动机 MotivationMotivation

「代码即策略」（Code-as-Policy）被视为数据密集型 VLA 方法的有力补充，但其作为自主控制器在 embodied manipulation 场景中的有效性仍严重缺乏系统性研究。现有工作大多依赖高层原语（如 stack_objs_in_order()），使得模型性能究竟来自智能体本身还是原语封装的任务先验难以区分，且未回答：当抽象层级降低时性能如何变化？增加测试时计算量是否能弥补低层接口带来的挑战？Code-as-Policy is seen as a strong complement to data-intensive VLA methods, yet its effectiveness as an autonomous controller for embodied manipulation remains severely underexplored. Most existing work relies on high-level primitives (e.g. stack_objs_in_order()), making it hard to tell whether performance comes from the agent itself or from the task priors baked into those primitives; nor does it answer how performance changes as the abstraction level drops, or whether more test-time compute can offset the challenges of low-level interfaces.

"Code-as-Policy considers how executable code can complement data-intensive Vision-Language-Action (VLA) methods, yet their effectiveness as autonomous controllers for embodied manipulation remains underexplored."

CaP-X teaser figure — （上）CaP-Bench 任务成功率随模型发布时间的变化：12个模型在7项任务上与人类专家的对比。前沿语言/视觉语言模型在编程机器人操控代码方面仍显著落后于人类。（下）CaP-Gym集成了RoboSuite、LIBERO-PRO和BEHAVIOR三个模拟器/基准，并展示了CaP-Agent0在多项任务上接近人类水平的表现。(Top) CaP-Bench task success rate against model release date: 12 models compared with human experts on 7 tasks. Frontier language / vision-language models still lag well behind humans at programming robot manipulation code. (Bottom) CaP-Gym integrates three simulators / benchmarks — RoboSuite, LIBERO-PRO and BEHAVIOR — and shows CaP-Agent0 approaching human-level performance on several tasks.

12前沿模型（开源+闭源）Frontier models (open + closed source)

7操控任务（单臂+双臂+移动）Manipulation tasks (single-arm + bimanual + mobile)

8评测层级（S1-S4, M1-M4）Evaluation tiers (S1-S4, M1-M4)

4/7CaP-Agent0 达到或超越人类水平的任务数Tasks where CaP-Agent0 matches or exceeds human level

02 方法 MethodMethod

CaP-X 由三个核心模块构成：CaP-Gym（交互环境）、CaP-Bench（多层次基准）、以及两种提升智能体性能的方法——训练无关的 CaP-Agent0 与基于 RL 微调的 CaP-RL。CaP-X consists of three core modules: CaP-Gym (the interactive environment), CaP-Bench (the multi-tier benchmark), and two ways to improve agent performance — the training-free CaP-Agent0 and the RL-fine-tuned CaP-RL.

CaP-Gym：分层控制环境CaP-Gym: A Hierarchical Control Environment

CaP-Gym 是一个建立在标准 Gymnasium 接口之上的分层控制框架，将低层物理仿真（或真实机器人）与有状态的代码执行器循环相绑定。感知原语包括语言条件分割（SAM3）、开放词汇定点（Molmo 2）以及OpenCV/Open3D等视觉库。控制原语调用运动规划器或逆运动学求解器（PyRoki），而非直接输出关节空间动作命令。任务涵盖7项核心操控：Cube Lift、Cube Stack、Spill Wipe、Peg Insertion、Cube Re-stack、Two-Arm Lift 和 Two-Arm Handover。CaP-Gym is a hierarchical control framework built on the standard Gymnasium interface, coupling low-level physics simulation (or a real robot) with a stateful code-executor loop. Perception primitives include language-conditioned segmentation (SAM3), open-vocabulary pointing (Molmo 2) and vision libraries such as OpenCV / Open3D. Control primitives call a motion planner or an inverse-kinematics solver (PyRoki) instead of emitting joint-space action commands directly. The tasks cover 7 core manipulations: Cube Lift, Cube Stack, Spill Wipe, Peg Insertion, Cube Re-stack, Two-Arm Lift and Two-Arm Handover.

CaP-Bench：8层次评测体系CaP-Bench: An 8-Tier Evaluation Suite

CaP-Bench 系统地在抽象层级与观测模态两个维度评测模型：CaP-Bench systematically evaluates models along two dimensions — abstraction level and observation modality:

单轮评测（S1–S4）Single-Turn Evaluation (S1–S4)

S1（Privileged）：使用真实仿真状态（掩码+物体位姿）S1 (Privileged): uses ground-truth simulation state (masks + object poses)
S2（Non-Privileged）：依赖真实感知模块处理原始RGB-D输入S2 (Non-Privileged): relies on real perception modules over raw RGB-D input
S3（Low-Level）：低层原语，含使用示例S3 (Low-Level): low-level primitives, with usage examples
S4（Low-Level）：低层原语，无使用示例S4 (Low-Level): low-level primitives, without usage examples

多轮评测（M1–M4）Multi-Turn Evaluation (M1–M4)

M1（Text-Only）：标准输出(stdout)和错误跟踪(stderr)M1 (Text-Only): standard output (stdout) and error traces (stderr)
M2（Multimodal）：当前RGB观测图像直接输入上下文M2 (Multimodal): the current RGB observation image is fed directly into the context
M3（VDM）：视觉差分模块将视觉观测转为结构化自然语言M3 (VDM): a visual difference module turns visual observations into structured natural language
M4（Low-Level+VDM）：结合低层原语与视觉差分M4 (Low-Level+VDM): combines low-level primitives with visual differencing

code example high-level vs low-level primitives — （左）Gemini-3-Pro 使用高层原语完成"lift the red cube"任务生成的代码示例；（右）同一模型使用低层原语实现等价功能所需的代码——需要更多关节控制与感知推理，难度显著增加。(Left) Code generated by Gemini-3-Pro for the "lift the red cube" task using high-level primitives; (Right) the code the same model needs to achieve the equivalent behaviour with low-level primitives — markedly harder, requiring far more joint control and perceptual reasoning.

CaP-Agent0：训练无关的智能体框架CaP-Agent0: A Training-Free Agent Framework

CaP-Agent0 架构：整合了自动合成的技能库（auto-synthesized skill library）、视觉差分模型（VDM）提供场景变化的文本描述，以及并行推理系统（多个编程智能体并发生成候选代码后由集成智能体合成最终方案）。该框架可在仿真和真实机器人环境中部署。The CaP-Agent0 architecture: an auto-synthesized skill library, a visual difference model (VDM) that describes scene changes in text, and a parallel reasoning system (several coding agents generate candidate code concurrently, then an ensemble agent synthesizes the final solution). The framework can be deployed both in simulation and on real robots.

CaP-Agent0 包含三个关键设计：CaP-Agent0 rests on three key designs:

多轮视觉差分（VDM）：将视觉差分模块集成为每轮的观测信息，使模型能够以文本形式感知场景变化，而非直接处理多模态图像（M2 反而会降低性能）。Multi-turn visual differencing (VDM): the visual difference module is integrated as the per-turn observation, letting the model perceive scene changes as text instead of consuming multimodal images directly (M2 actually degrades performance).
自动合成技能库（Skill Library）：通过自动化合成流水线构建与任务无关的技能库，从S3层级的成功rollout中提取函数定义，跨试次持久化使用。Auto-synthesized skill library: an automated synthesis pipeline builds a task-agnostic skill library, extracting function definitions from successful S3-tier rollouts and persisting them across trials.
并行推理（Parallel Reasoning）：在每轮并发采样候选方案：单模型配置（9次Gemini-3-Pro查询）或多模型配置（GPT-5.2、Claude Opus 4.5、Gemini-3-Pro各3次），再由集成模型合成最终代码。Parallel reasoning: candidate solutions are sampled concurrently at each turn — a single-model setting (9 Gemini-3-Pro queries) or a multi-model setting (3 queries each from GPT-5.2, Claude Opus 4.5 and Gemini-3-Pro) — after which an ensemble model synthesizes the final code.

CaP-RL：基于可验证奖励的强化学习CaP-RL: Reinforcement Learning from Verifiable Rewards

CaP-Gym 支持在策略编程智能体上直接进行在线强化学习（RLVR，使用可验证环境奖励）。具体采用 Group Relative Policy Optimization（GRPO）对 Qwen2.5-Coder-7B-Instruct 进行后训练，训练信号直接来自机器人操控环境的任务完成奖励，无需人工标注。CaP-Gym supports online reinforcement learning directly on the policy-coding agent (RLVR, using verifiable environment rewards). Concretely, Group Relative Policy Optimization (GRPO) post-trains Qwen2.5-Coder-7B-Instruct, with the training signal coming straight from task-completion rewards in the robot manipulation environment, requiring no human annotation.

03 实验 ExperimentsExperiments

在RoboSuite仿真、LIBERO-PRO 和 BEHAVIOR（移动操控）三个平台上评测，对比 12 个模型与人类专家，并在真实 Franka Emika 机械臂上验证 CaP-RL。Evaluation spans three platforms — RoboSuite simulation, LIBERO-PRO and BEHAVIOR (mobile manipulation) — comparing 12 models against human experts, with CaP-RL further validated on a real Franka Emika arm.

CaP-Bench：抽象层级对性能的影响CaP-Bench: How Abstraction Level Affects Performance

task success rate vs primitive abstraction — 开源与闭源模型的平均任务成功率随原语抽象层级提升（S4→S1）的变化。成功率呈单调递增，说明高层原语对性能有显著加持；S3-S4 的性能下降不仅来自代码正确率，还来自感知与控制难度的增加。Average task success rate of open- and closed-source models as primitive abstraction rises (S4→S1). Success increases monotonically, showing that high-level primitives give a substantial boost; the drop at S3-S4 stems not only from code correctness but also from the added perception and control difficulty.

多轮评测：视觉差分的关键作用Multi-Turn Evaluation: The Key Role of Visual Differencing

multi-turn benchmark comparison — 单轮（S2）与多轮层级（M1-M3）跨模型对比：多轮文本反馈（M1）在大多数模型上提升了任务成功率；直接多模态视觉（M2）反而降低了成功率；视觉差分转文本（M3/VDM）在开源与闭源模型上均一致地提升了任务成功率。Cross-model comparison of the single-turn tier (S2) and the multi-turn tiers (M1-M3): multi-turn text feedback (M1) raises task success on most models; feeding vision directly as multimodal input (M2) lowers it; converting visual differences into text (M3/VDM) consistently raises task success on both open- and closed-source models.

CaP-Agent0 消融实验CaP-Agent0 Ablations

CaP-Agent0 ablation study — （左）逐步叠加VDM、技能库（+SL）、单模型并行（+1M: Gemini-3-Pro）和多模型并行（+3M: Gemini-3-Pro + GPT-5.2 + Claude Opus）的消融结果，相对单轮低层API基准显著提升。（右）CaP-Agent0 在7项任务中的4项上达到或超越人类专家代码的成功率。(Left) Ablations that add VDM, the skill library (+SL), single-model parallelism (+1M: Gemini-3-Pro) and multi-model parallelism (+3M: Gemini-3-Pro + GPT-5.2 + Claude Opus) one step at a time, all clearly above the single-turn low-level API baseline. (Right) CaP-Agent0 matches or exceeds the success rate of human expert code on 4 of the 7 tasks.

LIBERO-PRO 与 VLA 方法对比LIBERO-PRO: Comparison with VLA Methods

基准 / 方法Benchmark / Method	OpenVLA	π₀	π₀.₅	CaP-Agent0
libero-object (Pos / Task)	0.00 / 0.00	0.00 / 0.00	0.17 / 0.01	0.22 / 0.18
libero-goal (Pos / Task)	0.00 / 0.00	0.00 / 0.00	0.38 / 0.00	0.26 / 0.17
libero-spatial (Pos / Task)	0.00 / 0.00	0.00 / 0.00	0.20 / 0.01	0.12 / 0.14

在 LIBERO-PRO 上，CaP-Agent0 在 Task Success 指标上全面超越 VLA 基线（包括 π₀.₅），无需任何任务专属训练数据。注意 Pos（Position Success）和 Task（Task Success）是不同评测指标，CaP-Agent0 在 Task 层面优势更为明显。On LIBERO-PRO, CaP-Agent0 beats every VLA baseline (including π₀.₅) on Task Success without any task-specific training data. Note that Pos (Position Success) and Task (Task Success) are different metrics; the advantage of CaP-Agent0 is clearer at the Task level.

CaP-RL 仿真与真实结果CaP-RL: Simulation and Real-World Results

CaP-RL pre and post RL real-world evaluation — CaP-RL 在仿真（N=100）和真实 Franka Emika 机械臂（N=25）上的评测结果，对比 Human Expert 与 RL 微调前后的 Qwen2.5-Coder-7B。CaP-RL evaluated in simulation (N=100) and on a real Franka Emika arm (N=25), compared against the Human Expert and against Qwen2.5-Coder-7B before and after RL fine-tuning.

任务Task	Human Expert	Qwen 2.5 Coder 7B（未微调）Qwen 2.5 Coder 7B (no fine-tuning)	Qwen w/ CaP-RL（仿真）Qwen w/ CaP-RL (simulation)	Qwen w/ CaP-RL（真实）Qwen w/ CaP-RL (real)
Cube Lift	93% / 92%	25% / 24%	80%	84%
Cube Stack	73% / 84%	4% / 12%	44%	76%
Spill Wipe	100% / —	30% / —	93%	—

CaP-RL 将 Qwen2.5-Coder-7B 在 Cube Lift 上的仿真成功率从 25% 提升至 80%，在真实机器人上从 24% 提升至 84%；Cube Stack 从 4% 提升至 44%（仿真）和 76%（真实），展现出强泛化能力。CaP-RL lifts the Cube Lift success of Qwen2.5-Coder-7B from 25% to 80% in simulation and from 24% to 84% on the real robot; Cube Stack rises from 4% to 44% (simulation) and 76% (real), showing strong generalization.

BEHAVIOR 移动操控（真实世界）BEHAVIOR Mobile Manipulation (Real World)

任务（N=25）Task (N=25)	指标Metric	Human Expert	S3（单轮低层）S3 (single-turn low-level)	CaP-Agent0
Pick up Radio	Navigation Success	88%	72%	80%
Pick up Radio	Task Success	36%	24%	56%
Pick up Soda Can	Navigation Success	80%	52%	84%
Pick up Soda Can	Task Success	72%	32%	72%

消融结论Ablation Takeaways

消融实验表明每个组件均有贡献：视觉差分（VDM）→ 技能库（+SL）→ 并行推理（+1M → +3M）逐步提升了任务成功率，其中多模型并行推理的增益最为显著。直接多模态输入（M2）会降低性能，说明视觉信息需经结构化转换（VDM）才能被模型有效利用。The ablations show that every component contributes: visual differencing (VDM) → skill library (+SL) → parallel reasoning (+1M → +3M) progressively raise task success, with multi-model parallel reasoning giving the largest gain. Feeding multimodal input directly (M2) degrades performance, indicating that visual information must be structurally converted (VDM) before models can exploit it.

04 局限性 LimitationsLimitations

注：本文没有独立的 Limitations 节；以下内容综合自第7节（Future Works and Conclusion）的作者陈述，并标注推断项。Note: the paper has no standalone Limitations section; the following is synthesized from the authors' statements in Section 7 (Future Works and Conclusion), with inferred items marked.

接触丰富任务（contact-rich tasks）仍然困难【作者陈述】Contact-rich tasks remain hard [author statement]

作者明确指出："Programmatic control performs well on long-horizon, reasoning-heavy tasks, but remains brittle for contact-rich behaviors that require tight visual servoing and continuous feedback (e.g., insertion or pouring)."——需要精细力控和连续视觉反馈的任务（如插孔、倒水）代码控制仍易失败。The authors state explicitly: "Programmatic control performs well on long-horizon, reasoning-heavy tasks, but remains brittle for contact-rich behaviors that require tight visual servoing and continuous feedback (e.g., insertion or pouring)." — code-driven control still fails easily on tasks that need fine force control and continuous visual feedback (such as peg insertion or pouring).

感知基础（perceptual grounding）的瓶颈【作者陈述】The perceptual grounding bottleneck [author statement]

作者将"more effective grounding of task-relevant visual information into code generation"列为重要改进方向，说明当前视觉感知原语与代码生成之间的衔接仍是性能瓶颈，尤其在低层（S3/S4）和多模态（M2）层级表现明显。The authors list "more effective grounding of task-relevant visual information into code generation" as an important direction, indicating that the link between visual perception primitives and code generation remains a bottleneck, most visibly at the low-level (S3/S4) and multimodal (M2) tiers.

测试时计算开销高【推断】High test-time compute cost [inferred]

CaP-Agent0 的并行推理（最多3个前沿模型、每模型3次查询）虽然无需训练，但在每次执行时需调用多次昂贵的大模型API，测试时计算成本较高，可能限制其在资源受限场景中的实际部署。The parallel reasoning of CaP-Agent0 (up to 3 frontier models, 3 queries each) needs no training, but every execution issues many expensive large-model API calls, so test-time compute cost is high and may limit practical deployment in resource-constrained settings.

CaP-RL 训练的泛化边界【推断】Generalization limits of CaP-RL training [inferred]

CaP-RL 目前仅在有限任务（Cube Lift、Cube Stack、Spill Wipe）上验证，且基座模型为 Qwen2.5-Coder-7B-Instruct（7B参数）。对更复杂任务或更大规模基座的扩展性尚未充分探索。CaP-RL has so far been validated only on a limited set of tasks (Cube Lift, Cube Stack, Spill Wipe), with Qwen2.5-Coder-7B-Instruct (7B parameters) as the base model. Scaling to more complex tasks or larger base models remains underexplored.