CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks

01 动机Motivation

现有视觉-语言任务数据集在序列长度、动作空间复杂度和语言多样性方面均存在明显不足，难以训练出能够在日常真实环境中与人类协作的通用机器人。如何让机器人在仅凭语言指令的条件下，自主完成由多个子技能组合而成的长序列操控任务，是迈向通用机器人的关键挑战。Existing vision-language task datasets fall clearly short in sequence length, action-space complexity and language diversity, making it hard to train general-purpose robots that can collaborate with humans in everyday real-world settings. Enabling a robot to autonomously complete long-horizon manipulation tasks composed of multiple sub-skills from language instructions alone is a key challenge on the way to general-purpose robots.

"General-purpose robots coexisting with humans in their environment must learn to relate human language to their perceptions and actions to be useful in a range of daily tasks. Moreover, they need to acquire a diverse repertoire of general-purpose skills that allow composing long-horizon tasks by following unconstrained language instructions."

CALVIN benchmark overview — **Fig. 1：CALVIN 基准总览。**训练阶段跨越四个环境（Env A–C 训练，Env D 零样本测试），操控员演示包含多种语言指令，如 "move the switch to turn on the light bulb"、"push the button"、"place the red block in the slider"、"push the blue block to the left"。最困难的评估设置要求智能体在从未见过的 Env D 中，零样本执行多步语言指令链。**Fig. 1: Overview of the CALVIN benchmark.**Training spans four environments (Env A–C for training, Env D for zero-shot testing), and operator demonstrations come with diverse language instructions such as "move the switch to turn on the light bulb", "push the button", "place the red block in the slider", "push the blue block to the left". The hardest evaluation setting requires the agent to execute multi-step language instruction chains zero-shot in the never-before-seen Env D.

34操控任务类别manipulation task categories

7-DOF连续动作空间continuous action space

400+人工采集自然语言标注human-collected natural-language annotations

4→1训练环境 → 零样本测试环境training envs → zero-shot test env

为什么现有基准不够用？Why are existing benchmarks not enough?

序列长度受限：现有数据集通常只评估单步或短序列任务，无法衡量智能体对多技能组合的掌握程度。Limited sequence length:Existing datasets usually evaluate only single-step or short-sequence tasks, and cannot measure how well an agent masters the composition of multiple skills.
语言多样性不足：现有数据集多依赖模板化语言标注，而 CALVIN 收集超过 400 条众包自然语言指令，对应 34 种精细操控任务，并提供超过 30,522 个词汇量的语料库（平均句长 7.98 词）。Insufficient language diversity:Existing datasets mostly rely on templated language annotations, whereas CALVIN collects over 400 crowdsourced natural-language instructions covering 34 fine-grained manipulation tasks, and provides a corpus with a vocabulary of more than 30,522 words (average sentence length 7.98 words).
泛化评估缺失：既有方法很少在全新环境和对象上进行零样本评估，CALVIN 将跨环境零样本泛化作为核心评估轴。Missing generalization evaluation:Prior methods rarely run zero-shot evaluation in entirely new environments and on new objects; CALVIN makes cross-environment zero-shot generalization a core evaluation axis.
传感器灵活性差：CALVIN 支持 RGB-D 图像、本体感知、视觉触觉等多种传感器组合，便于研究多模态感知对策略的影响。Poor sensor flexibility:CALVIN supports diverse sensor combinations such as RGB-D images, proprioception and visuo-tactile sensing, making it convenient to study how multimodal perception affects the policy.

02 方法Method

CALVIN 不仅是数据集，更是一个完整的研究框架，包含仿真环境、数据采集流程、语言标注协议和评估协议。论文同时提出一个 multi-context imitation learning（MCIL）基线模型作为出发点，供后续研究超越。CALVIN is not only a dataset but a complete research framework, comprising the simulation environment, the data-collection pipeline, the language-annotation protocol and the evaluation protocol. The paper also proposes a multi-context imitation learning (MCIL) baseline model as a starting point for later work to surpass.

CALVIN 仿真环境The CALVIN simulation environment

基于 PyBullet 物理引擎构建，包含一个带滑动门、按钮、LED 灯、滑块和彩色积木的桌面工作区，由 7-DOF Franka Emika Panda 机械臂操控。环境设计在外观（纹理、颜色）和物体布局上均有变体，支持在 4 个训练环境（Env A、B、C）和 1 个测试环境（Env D）之间的零样本泛化评估。动作空间为笛卡尔末端执行器位移（绝对位置 3 维 + 方向 3 维 + 夹爪 1 维），控制频率 30 Hz。Built on the PyBullet physics engine, it contains a tabletop workspace with a sliding door, buttons, an LED light, a slider and colored blocks, manipulated by a 7-DOF Franka Emika Panda arm. The environments vary in appearance (textures, colors) and in object layout, supporting zero-shot generalization evaluation between 4 training environments (Env A, B, C) and 1 test environment (Env D). The action space is the Cartesian end-effector displacement (absolute position 3 dims + orientation 3 dims + gripper 1 dim), at a control frequency of 30 Hz.

CALVIN sensors and action space — **Fig. 2：观察与动作空间。**左表列出所有支持的传感器模态及其维度： RGB 静态相机（200×200×3）、RGB 夹爪相机（84×84）、深度相机（84×84）、触觉传感器（120×160×2）、本体感知（末端执行器位置 3 维、方向 3 维、关节角度 7 维、夹爪动作 1 维）。右图展示 CALVIN 支持的丰富传感器套件示意，包括固定视角、夹爪视角和深度感知。**Fig. 2: Observation and action spaces.**The left table lists all supported sensor modalities and their dimensions: RGB static camera (200×200×3), RGB gripper camera (84×84), depth camera (84×84), tactile sensor (120×160×2), proprioception (end-effector position 3 dims, orientation 3 dims, joint angles 7 dims, gripper action 1 dim). The right figure sketches the rich sensor suite supported by CALVIN, including the fixed viewpoint, the gripper viewpoint and depth sensing.

语言标注协议Language annotation protocol

操控员（operator）在三个关键环境中远程控制机械臂完成示教（play data），采集约 2.4 万步遥操作数据。在此基础上，自动分割出对应 34 种技能的短序列片段，并由标注员为每个片段撰写自由形式的自然语言指令（平均每任务约 12 条同义表达）。最终 CALVIN 数据集包含跨四个环境的多样化演示，语言指令词汇量超过 30,522，平均句长 7.98 词，句子数量 7 词以下占多数，呈现真实人类自然语言的多样性。An operator teleoperates the arm in three key environments to record demonstrations (play data), collecting about 24000 steps of teleoperation data. On this basis, short segments corresponding to 34 skills are automatically segmented out, and annotators write free-form natural-language instructions for each segment (about 12 synonymous expressions per task on average). The final CALVIN dataset contains diverse demonstrations across four environments, with an instruction vocabulary of more than 30,522 words, an average sentence length of 7.98 words, and most sentences under 7 words, exhibiting the diversity of real human natural language.

CALVIN Challenge 评估协议The CALVIN Challenge evaluation protocol

评估时，智能体需在无示教的情况下，依据语言指令链连续完成 1,000 条指令序列，最多连续完成 5 个子任务（Multi-Task Long-Chain，MTLC）。成功率以"在 1,000 条指令链中，每条平均连续完成多少步"衡量（No. Instructions in a Row）。论文设置了三类评估难度：At evaluation time the agent must, without demonstrations, complete 1,000 instruction sequences one after another following language instruction chains, with at most 5 subtasks in a row (Multi-Task Long-Chain, MTLC). Success is measured as "how many steps are completed in a row on average per chain across the 1,000 instruction chains" (No. Instructions in a Row). The paper defines three levels of evaluation difficulty:

Single Environment（D→D）：训练与测试同一环境，单任务。Single Environment (D→D):Training and testing in the same environment, single task.
Multi-Task Long-Chain（ABCD→D）：在全部训练环境数据上训练，在新环境 D 零样本执行任务链。Multi-Task Long-Chain (ABCD→D):Train on the data of all training environments and execute task chains zero-shot in the new environment D.
Long-Horizon MTLC（LH-MTLC）：最困难设置，需在 Env D 中连续完成多步指令，测试长序列组合能力。Long-Horizon MTLC (LH-MTLC):The hardest setting, requiring multi-step instructions to be completed in a row in Env D, testing long-horizon compositional ability.

MCIL 基线模型The MCIL baseline model

作者基于 Lynch et al. 的 multi-context imitation learning 框架实现基线：使用 sequence-to-sequence 变分自编码器（VAE）学习语言条件下的目标图像表征，再训练一个 goal-conditioned 策略以闭环执行任务。模型输入为视觉观察和语言指令，输出为连续动作；训练数据来自跨所有环境的遥操作示教，编码器和解码器独立训练后联合微调。评估时语言指令直接送入语言编码器（BERT/随机窗口语言），推理阶段无需示例图像。The authors implement the baseline on top of the multi-context imitation learning framework of Lynch et al.: a sequence-to-sequence variational autoencoder (VAE) learns language-conditioned goal-image representations, and a goal-conditioned policy is then trained to execute tasks in closed loop. The model takes visual observations and language instructions as input and outputs continuous actions; the training data comes from teleoperated demonstrations across all environments, with encoder and decoder trained separately and then jointly fine-tuned. At evaluation the language instruction is fed directly into the language encoder (BERT / random-window language), and no example image is needed at inference.

03 实验Experiments

论文在 CALVIN Challenge 的三种难度设置下评估 MCIL 基线，并分析不同传感器组合对性能的影响，揭示当前方法面临的核心挑战。The paper evaluates the MCIL baseline under the three difficulty settings of the CALVIN Challenge and analyzes how different sensor combinations affect performance, revealing the core challenges faced by current methods.

CALVIN results distribution — **Fig. 3：任务子指令使用频率分布（1,000 条评估链）。** 横轴为随机采样的 1,000 条指令链中各子任务出现次数，展示不同任务在评估集中的频率差异，体现 CALVIN 任务分布的自然多样性。**Fig. 3: Frequency distribution of sub-instruction usage (1,000 evaluation chains).** The horizontal axis gives the number of occurrences of each subtask among the 1,000 randomly sampled instruction chains, showing the frequency differences between tasks in the evaluation set and reflecting the natural diversity of the CALVIN task distribution.

基线结果Baseline results

下表为 MCIL 基线在 CALVIN 各评估设置下的成功率（成功完成的 "No. Instructions in a Row"，满分 1000）。数字越高代表在连续指令链中完成的步数越多。The table below gives the success rate of the MCIL baseline under each CALVIN evaluation setting (the successfully completed "No. Instructions in a Row", out of 1000). Higher numbers mean more steps completed within a consecutive instruction chain.

CALVIN baseline results table — **Fig. 4：MCIL 基线在 CALVIN Challenge 上的完整结果。** 各列对应不同传感器组合（Static Camera / Gripper Camera / RGB / Depth / Tactile），各行对应不同评估设置（D→D 单环境；ABCD→D 多环境零样本；LH-MTLC 长序列）。结果显示：(1) 传感器组合对性能影响显著；(2) 零样本跨环境泛化（ABCD→D）较单环境（D→D）有明显下降； (3) 随连续指令步数增加，成功率急剧下降，说明长序列任务极具挑战性。**Fig. 4: Full results of the MCIL baseline on the CALVIN Challenge.** The columns correspond to different sensor combinations (Static Camera / Gripper Camera / RGB / Depth / Tactile), the rows to different evaluation settings (D→D single environment; ABCD→D multi-environment zero-shot; LH-MTLC long-horizon). The results show: (1) the sensor combination has a marked effect on performance; (2) zero-shot cross-environment generalization (ABCD→D) drops clearly relative to the single-environment setting (D→D); (3) as the number of consecutive instruction steps grows, the success rate falls sharply, showing that long-horizon tasks are highly challenging.

评估设置Setting	传感器Sensors	成功率（MTLC，34 tasks）Success rate (MTLC, 34 tasks)	说明Notes
D→D（单环境）D→D (single environment)	Static RGB	35.6%	单环境内训练测试Train and test within one environment
ABCD→D（零样本）ABCD→D (zero-shot)	Static RGB	49.7%	多环境训练 → 新环境测试Multi-environment training → new-environment test
ABCD→D（零样本）ABCD→D (zero-shot)	RGB + Gripper	37.3%	增加夹爪视角Adds the gripper viewpoint
LH-MTLC（长序列链）LH-MTLC (long chain)	Static RGB	≈2.6%	1,000 链中完成 5 步的比例极低Extremely low fraction completing 5 steps among the 1,000 chains

关键发现： 即便是表现最好的传感器组合（Static Camera 单独），MCIL 基线在 LH-MTLC 长序列设置下成功率依然极低，证明"将子技能组合为长序列任务"对现有 imitation learning 方法仍是开放挑战。论文明确指出："a baseline model based on multi-context imitation learning performs poorly on CALVIN, suggesting that there is significant room for developing innovative agents."Key finding: Even with the best-performing sensor combination (Static Camera alone), the MCIL baseline still reaches an extremely low success rate under the LH-MTLC long-horizon setting, showing that "composing sub-skills into long-horizon tasks" remains an open challenge for existing imitation learning methods. The paper states explicitly: "a baseline model based on multi-context imitation learning performs poorly on CALVIN, suggesting that there is significant room for developing innovative agents."

传感器消融Sensor ablation

实验系统比较了 Static Camera（200×200 RGB）、Gripper Camera（84×84 RGB）、Depth、Tactile 四类传感器在不同组合下的策略性能。结果显示：The experiments systematically compare four sensor types — Static Camera (200×200 RGB), Gripper Camera (84×84 RGB), Depth and Tactile — under different combinations. The results show:

仅使用 Static RGB Camera 时，MCIL 在单环境设置（D→D）下达到 35.6% 成功率；多环境设置（ABCD→D）约 49.7%。With the Static RGB Camera alone, MCIL reaches 35.6% success in the single-environment setting (D→D), and about 49.7% in the multi-environment setting (ABCD→D).
加入 Gripper Camera 不一定提升性能，某些组合下反而因输入维度增大导致下降。Adding the Gripper Camera does not necessarily improve performance; under some combinations it even degrades results because of the larger input dimensionality.
Depth 和 Tactile 传感器对当前基线帮助有限，但 CALVIN 框架为未来研究多模态感知提供了接口。Depth and Tactile sensors help the current baseline only marginally, but the CALVIN framework provides an interface for future research on multimodal perception.
本体感知（proprioceptive state）对精细操控任务（如 grasp）贡献明显。Proprioceptive state contributes clearly to fine-grained manipulation tasks such as grasping.

04 局限性Limitations

说明：论文在 Discussion 和 Conclusion 部分明确指出了以下局限性，以下各点均为作者明确陈述，并附有推断标注。Note:The paper explicitly points out the following limitations in its Discussion and Conclusion sections; every item below is explicitly stated by the authors, with inference labels attached.

仅限仿真环境，未涉及真实机器人Simulation only, no real robot involved

CALVIN 完全基于 PyBullet 仿真，尚未在真实机器人平台上验证。仿真与现实之间的 sim-to-real gap 可能使基准上的结论难以直接迁移至实体机器人。作者未在论文中提供真实场景验证（作者明确指出）。CALVIN is built entirely on PyBullet simulation and has not been validated on a real robot platform. The sim-to-real gap may make conclusions drawn on the benchmark hard to transfer directly to physical robots. The authors provide no real-world validation in the paper (explicitly stated by the authors).

基线模型性能较弱，尚无强竞争对手The baseline model is weak, and no strong competitor exists yet

论文发布时仅提供 MCIL 一种基线，且该基线在长序列任务上表现极差，难以作为有力参照。更多竞争性方法的加入依赖社区后续跟进（作者明确指出需要社区投入）。At publication only the single MCIL baseline is provided, and that baseline performs very poorly on long-horizon tasks, so it is hard to use as a strong reference. Adding more competitive methods depends on follow-up work by the community (the authors explicitly note the need for community effort).

任务设计局限于桌面精细操控Task design is confined to tabletop fine manipulation

34 种任务均聚焦于单一桌面工作区内的精细操控（推拉积木、开关灯、滑动门等），场景多样性和任务复杂度（如多物体交互、工具使用）仍有限（从设计推断）。All 34 tasks focus on fine-grained manipulation inside a single tabletop workspace (pushing and pulling blocks, switching lights, sliding doors, etc.), so scene diversity and task complexity (such as multi-object interaction and tool use) remain limited (inferred from the design).

语言指令来源仍有偏差The language instructions still carry sourcing bias

尽管通过众包收集了 400 余条自然语言标注，但标注者群体和指令风格可能存在统计偏差，未覆盖真实人机交互中的所有表达方式（从方法推断）。Although more than 400 natural-language annotations were collected by crowdsourcing, the annotator population and the instruction style may carry statistical bias and do not cover every way of speaking in real human-robot interaction (inferred from the method).

评估指标单一A single evaluation metric

当前主要以"连续完成子指令数"（No. Instructions in a Row）作为唯一核心指标，未涵盖效率、安全性、自然性等多维度评估（从实验设计推断）。At present the "number of sub-instructions completed in a row" (No. Instructions in a Row) is the only core metric; efficiency, safety, naturalness and other dimensions are not covered (inferred from the experimental design).