Code as Policies

01 动机Motivation

机器人需要理解自然语言指令并转化为具体行为。已有方法（如 SayCan、语义解析器）要么仅输出离散动作序列，要么需要大量标注数据训练，难以泛化到新指令。能否利用 LLM 直接生成结构化的机器人控制程序，从而利用代码的逻辑表达能力实现更强的空间推理与泛化？Robots need to understand natural-language instructions and turn them into concrete behavior. Existing approaches (e.g. SayCan, semantic parsers) either emit only discrete action sequences or require training on large amounts of annotated data, and generalize poorly to new instructions. Can an LLM directly generate structured robot control programs, so that the logical expressiveness of code yields stronger spatial reasoning and generalization?

"We find that language models trained on code-completion can be repurposed to write robot policy code, given natural language commands (formatted as comments) and few-shot examples of language instructions followed by corresponding code."

现有方法的不足Shortcomings of existing approaches

• 语义解析器（Semantic Parsers）：需要大量标注数据训练，泛化能力差。Semantic parsers: require training on large amounts of annotated data and generalize poorly.
• Chain-of-Thought / SayCan 类方法：只能输出离散动作序列，无法表达循环、条件判断等逻辑结构，也无法调用第三方数值计算库（NumPy、Shapely）进行精确空间推理。Chain-of-Thought / SayCan-style methods: can only emit discrete action sequences; they cannot express logical structure such as loops and conditionals, nor call third-party numerical libraries (NumPy, Shapely) for precise spatial reasoning.
• 端到端 RL / 模仿学习：需要大量环境交互或人类演示，且难以泛化到未见指令。End-to-end RL / imitation learning: require extensive environment interaction or human demonstrations, and generalize poorly to unseen instructions.

CaP 的核心洞察是：LLM 在代码补全任务上的训练赋予了它理解和生成 Python 程序的能力，而代码天然具备表达复杂逻辑的能力——这正是自然语言所缺失的。The core insight of CaP is that training an LLM on code completion gives it the ability to understand and generate Python programs, while code natively expresses complex logic — exactly what natural language lacks.

02 方法Method

Code as Policies（CaP）是一种以 LLM 为核心的机器人控制框架。在 Hints 中告知可用 API，在 Examples 中提供少量语言-代码对示范，然后让 LLM 为新的自然语言指令生成对应的策略代码，并直接在机器人上执行。Code as Policies (CaP) is a robot control framework built around an LLM. Hints declare the available APIs, Examples supply a handful of language-code demonstration pairs, and the LLM then generates the corresponding policy code for a new natural-language instruction, which is executed directly on the robot.

语言模型程序（Language Model Programs, LMP）Language Model Programs (LMPs)

每个 LMP 是一段由 LLM 生成的 Python 函数。LMP 的输入提示格式如下：Each LMP is a Python function generated by the LLM. The format of an LMP's input prompt is as follows:

# Place the first blue block to the left of the red block.
import numpy as np
target_pos = get_pos('red block')
place_pos  = target_pos + np.array([-0.1, 0, 0])
put_first_on_second('blue block', place_pos)

LMP 可以调用感知 API（如 detect_objects()、get_pos()）获取环境状态，调用控制 API（如 pick_place()、put_first_on_second()）执行动作。变量命名和代码结构遵循自然语言描述，使代码本身具有可读性。An LMP can call perception APIs (e.g. detect_objects(), get_pos()) to read the state of the environment, and control APIs (e.g. pick_place(), put_first_on_second()) to execute actions. Variable names and code structure follow the natural-language description, which makes the code itself readable.

层次化代码生成（Hierarchical Code Generation）Hierarchical Code Generation

当生成的代码引用了尚未定义的函数时，系统会递归地为该函数生成新的 LMP，直至所有函数均有实现。这一机制使 CaP 能处理更复杂的指令，同时保持每层提示的简洁性。When the generated code references a function that has not yet been defined, the system recursively generates a new LMP for that function until every function has an implementation. This mechanism lets CaP handle more complex instructions while keeping the prompt at each level concise.

"LMPs can be hierarchically prompted: the prompt for a high-level function e.g. parse_obj ... generates code that calls lower-level functions e.g. get_obj_bbox_area_xy, and automatically generates implementations of those lower-level functions."

层次化结构的优点：各层 LMP 专注于自身抽象层次，提示长度可控（在模型的 context window 内），同时组合产生高层行为能力。Advantages of the hierarchical structure: each LMP focuses on its own level of abstraction, prompt length stays bounded (within the model's context window), and composition yields high-level behavioral capability.

第三方库与空间推理Third-party libraries and spatial reasoning

CaP 允许代码导入并使用 NumPy（数值计算）、Shapely（几何计算）等标准 Python 库，从而无需专门训练即可完成精确的空间推理，例如"在四个对象的凸包内随机游走"等复杂几何任务。CaP lets the generated code import and use standard Python libraries such as NumPy (numerical computation) and Shapely (geometric computation), so it can perform precise spatial reasoning without dedicated training — for example complex geometric tasks such as "walk randomly inside the convex hull of the four objects".

支持的策略类型Supported policy types

• Waypoint-based policies：代码输出一系列目标位置，由底层运动规划器执行；Waypoint-based policies: the code outputs a sequence of target positions that a low-level motion planner executes;
• Reactive policies：代码以闭环方式持续读取传感器反馈并输出控制量（如阻抗控制器参数），适用于动态交互任务。Reactive policies: the code continuously reads sensor feedback in closed loop and outputs control values (e.g. impedance controller parameters), which suits dynamic interaction tasks.

03 实验Experiments

作者在三个维度上评估 CaP：(i) 新提出的 RoboCodeGen benchmark（代码生成质量）；(ii) TabletopManipulation 仿真任务（与 CLIPort 等基线比较成功率）；(iii) 真实机器人上的定性演示。同时分析层次化 code-gen 对代码质量的提升效果。The authors evaluate CaP along three axes: (i) the newly proposed RoboCodeGen benchmark (code generation quality); (ii) TabletopManipulation simulation tasks (success rate against baselines such as CLIPort); (iii) qualitative demonstrations on real robots. They also analyze how much hierarchical code-gen improves code quality.

RoboCodeGen 基准（代码生成）The RoboCodeGen benchmark (code generation)

作者提出 RoboCodeGen——一个包含 37 道机器人代码生成题目的评测集（对标 HumanEval 风格），涵盖 table-top manipulation 典型任务。指标为 pass@N（N 次采样中至少一次通过单元测试的比例）。The authors introduce RoboCodeGen, a benchmark of 37 robot code-generation problems (in the style of HumanEval) covering typical table-top manipulation tasks. The metric is pass@N (the fraction of problems for which at least one of N samples passes the unit tests).

Table I 节选（来自论文原文）：层次化代码生成在 pass@6 和 pass@11 指标上均优于或持平于 flat 策略，表明递归函数定义有效提升了代码质量。Excerpt from Table I (verbatim from the paper): hierarchical code generation matches or beats the flat strategy on both pass@6 and pass@11, indicating that recursive function definition effectively improves code quality.

TabletopManipulation 仿真任务TabletopManipulation simulation tasks

Table II 节选（来自论文原文）：CaP 在 Seen 属性上与 CLIPort 持平（53.0%），但 CLIPort 无法泛化到 Unseen 属性（需要重新训练），而 CaP 无需额外训练即可处理新属性（39.3%/39.4%）。Excerpt from Table II (verbatim from the paper): CaP matches CLIPort on Seen attributes (53.0%), but CLIPort cannot generalize to Unseen attributes (it would need retraining), whereas CaP handles new attributes with no additional training (39.3%/39.4%).

成功率对比（Table III）Success rate comparison (Table III)

Table III 节选：CLIPort 在 Spatial-Geometric 任务上得 0 分（无法处理空间推理），而 CaP 通过 NumPy/Shapely 代码达到 75.33%（P@100）。Long-Horizon 任务上 CLIPort 领先（97.28% vs 77.93%），说明对于已见简单序列任务，专门训练的模型更有优势。Excerpt from Table III: CLIPort scores 0 on the Spatial-Geometric tasks (it cannot handle spatial reasoning), while CaP reaches 75.33% (P@100) through NumPy/Shapely code. On Long-Horizon tasks CLIPort leads (97.28% vs 77.93%), showing that a specially trained model retains an edge on seen, simple sequential tasks.

消融实验（HumanEval）Ablation study (HumanEval)

作者在 HumanEval benchmark 上分析了层次化代码生成的效果：将未定义函数递归分解后，pass@1 从基线提升至 39.8%，优于同规模的 flat 策略。这一结果直接支持了层次化 code-gen 的有效性假设。The authors analyze the effect of hierarchical code generation on the HumanEval benchmark: after recursively decomposing undefined functions, pass@1 rises from the baseline to 39.8%, beating the flat strategy at the same scale. This result directly supports the hypothesis that hierarchical code-gen is effective.

"Hierarchical code generation, which incrementally generates new instructions and responses to the prompt, allows later instructions to refer back to previous instructions, like 'undo the last action'."

04 局限性Limitations