AGNOSTOS & X-ICM：视觉语言动作模型跨任务泛化极限探索

01 动机Motivation

通用机器人操作的核心挑战在于：模型能否将学到的技能迁移到从未见过的任务上？现有 VLA 模型（包括 π0、OpenVLA、RDT 等）的跨任务泛化能力几乎未被系统研究，缺乏统一的评估框架。The core challenge of general-purpose robot manipulation is whether a model can transfer learned skills to tasks it has never seen. The cross-task generalization of existing VLA models (including π0, OpenVLA and RDT) has barely been studied systematically, and a unified evaluation framework is missing.

"The cross-task generalization capabilities of existing VLA models remain significantly underexplored."

AGNOSTOS benchmark overview — **Figure 1：AGNOSTOS 基准概览。**Level-1 包含 13 个未见任务，与训练任务有部分语义相似性（共享对象或动作原语）；Level-2 包含 10 个完全陌生的场景，既无已知对象也无已知动作，要求更强的泛化推断能力。右侧展示了具体任务示例。**Figure 1: Overview of the AGNOSTOS benchmark.** Level-1 contains 13 unseen tasks that share partial semantic similarity with the training tasks (shared objects or action primitives); Level-2 contains 10 entirely unfamiliar scenarios with neither known objects nor known actions, demanding stronger generalization and inference. Concrete task examples are shown on the right.

23未见测试任务总数unseen test tasks in total

13 + 10Level-1 + Level-2

18已见训练任务数（RLBench）seen training tasks (RLBench)

12+评测的 VLA 模型类别categories of VLA models evaluated

现有跨任务评估基准（如 Colosseum、GemBench）缺乏对 foundation model 与人类视频预训练模型的系统比较，且未区分难度级别。AGNOSTOS 通过双难度设计和多类别 VLA 评测，首次全面揭示了当前模型的跨任务能力边界。Existing cross-task evaluation benchmarks (such as Colosseum and GemBench) lack a systematic comparison between foundation models and models pre-trained on human videos, and do not separate difficulty levels. With its two-level design and its evaluation across VLA categories, AGNOSTOS is the first to reveal comprehensively where the cross-task ability of current models ends.

已评测 VLA 模型类别VLA model categories evaluated

域内模型：PerAct, RVT, RVT2, Sigma-Agent, InstantPolicyIn-domain models: PerAct, RVT, RVT2, Sigma-Agent, InstantPolicy
人类视频预训练：R3M, D4R 系列Human-video pre-training: R3M and the D4R series
Foundation model：OpenVLA, RDT, π0, VoxPoser, SAM2Act, 3D-LOTUS++Foundation models: OpenVLA, RDT, π0, VoxPoser, SAM2Act, 3D-LOTUS++

核心发现Key findings

所有现有 VLA 基线在 23 个未见任务中均表现欠佳All existing VLA baselines perform poorly on the 23 unseen tasks
多个模型在 ≥8 个任务上完全失败（成功率 0%）Several models fail completely on ≥8 tasks (0% success rate)
Level-2 难度对所有模型均构成显著挑战Level-2 difficulty is a significant challenge for every model
X-ICM 是唯一在全部 23 个任务中均有成功的方法X-ICM is the only method that succeeds on all 23 tasks

02 方法Method

X-ICM（Cross-Task In-Context Manipulation）将跨任务泛化问题转化为 LLM 的上下文学习问题：将已见任务的示例"文本化"后作为 prompt，驱动大语言模型直接预测未见任务的动作序列。该方法无需对 VLA 进行额外微调，完全在推理阶段运行。X-ICM (Cross-Task In-Context Manipulation) recasts cross-task generalization as in-context learning for an LLM: demonstrations of seen tasks are "textualized" into a prompt that drives the large language model to directly predict action sequences for unseen tasks. The method needs no extra fine-tuning of the VLA and runs entirely at inference time.

X-ICM method overview — **Figure 2：X-ICM 方法总览。**左侧为*动态引导示例选择*模块：训练 dynamics diffusion model 从初始观测和语言描述预测任务完成后的最终视觉状态，提取特征后通过余弦相似度检索最相关的 K=18 条已见示例。右侧为*跨任务上下文预测*模块：将选出的示例构造为"语言/对象上下文 → 归一化 3D 动作坐标"映射的文本 prompt，输入 LLM（Qwen2.5-7B/72B）生成未见任务的动作。**Figure 2: Overview of X-ICM.** Left, the *dynamics-guided sample selection* module: a dynamics diffusion model is trained to predict, from the initial observation and the language description, the final visual state after task completion; the extracted features then retrieve the K=18 most relevant seen demonstrations by cosine similarity. Right, the *cross-task in-context prediction* module: the selected demonstrations are cast into a text prompt that maps "language / object context → normalized 3D action coordinates" and fed to an LLM (Qwen2.5-7B/72B) to generate actions for the unseen task.

模块一：Dynamics-Guided Sample Selection（动态引导示例选择）Module 1: Dynamics-Guided Sample Selection

直接用语义相似性选示例效果有限，因为跨任务的关键在于动作动态而非表面语义。X-ICM 训练一个基于 InstructPix2Pix 的 dynamics diffusion model（记为 𝒢），以初始视觉观测和语言描述为条件，预测任务完成时的最终视觉状态。提取其 language feature（f_lang）与 visual feature（f_vis.in）的组合作为动态特征表示，然后通过余弦相似度从所有已见示例中检索 K=18 条最相关的跨任务演示。该模块显著提升了示例的相关性，降低了预测方差。Selecting demonstrations by semantic similarity alone works poorly, because what matters across tasks is action dynamics rather than surface semantics. X-ICM trains a dynamics diffusion model based on InstructPix2Pix (denoted 𝒢) that, conditioned on the initial visual observation and the language description, predicts the final visual state at task completion. The combination of its language feature (f_lang) and visual feature (f_vis.in) serves as the dynamics feature representation, and cosine similarity then retrieves the K=18 most relevant cross-task demonstrations among all seen examples. This module markedly improves the relevance of the demonstrations and reduces prediction variance.

模块二：Cross-Task In-Context Prediction（跨任务上下文预测）Module 2: Cross-Task In-Context Prediction

将选出的 K 条示例"文本化"为结构化 prompt：每条示例表示为（语言描述, 对象位置信息）→（归一化 3D 动作坐标序列）的映射。将 prompt 输入 LLM（Qwen2.5-7B 或 Qwen2.5-72B），要求其推断未见任务的动作模式并输出相应的动作坐标。整个流程不修改 VLA 模型权重，推理时间开销主要来自 LLM forward pass。The K selected demonstrations are "textualized" into a structured prompt: each one is written as a mapping (language description, object position information) → (normalized 3D action coordinate sequence). The prompt is fed to an LLM (Qwen2.5-7B or Qwen2.5-72B), which is asked to infer the action pattern of the unseen task and to output the corresponding action coordinates. The whole pipeline leaves the VLA model weights untouched; the inference-time cost comes mainly from the LLM forward pass.

设计核心：通过"动态特征"而非"表面语义"匹配示例，使 LLM 能够在已见任务的动作模式中发现与未见任务最相关的跨任务规律，从而实现真正意义上的零样本操作泛化。Core design: matching demonstrations by "dynamics features" rather than by "surface semantics" lets the LLM discover, within the action patterns of seen tasks, the cross-task regularities most relevant to an unseen task, achieving zero-shot manipulation generalization in the true sense.

03 实验Experiments

在 AGNOSTOS 基准的 23 个未见任务上进行零样本跨任务评测，并在 5 个真实物理场景中验证 X-ICM 的迁移能力。所有成功率均为多次运行的均值 ± 标准差。Zero-shot cross-task evaluation is run on the 23 unseen tasks of the AGNOSTOS benchmark, and the transfer ability of X-ICM is further verified in 5 real physical scenarios. All success rates are means ± standard deviations over multiple runs.

主要结果（Table 2：跨任务零样本操作成功率）Main results (Table 2: cross-task zero-shot manipulation success rate)

方法Method	Level-1（13 任务）Level-1 (13 tasks)	Level-2（10 任务）Level-2 (10 tasks)	Overall（23 任务）Overall (23 tasks)
VoxPoser	20.9% ± 0.3	8.0% ± 0.3	15.6% ± 0.2
π0（prior best）π0 (prior best)	21.7% ± 0.4	11.5% ± 0.5	17.5% ± 0.4
SAM2Act	14.4% ± 0.5	15.9% ± 1.3	15.1% ± 0.8
X-ICM（7B）X-ICM (7B)	28.6% ± 1.9	16.9% ± 1.3	23.5% ± 1.6
X-ICM（72B）X-ICM (72B)	37.6% ± 1.4	20.3% ± 1.7	30.1% ± 1.0

X-ICM（72B）在 Overall 上超越 π0 约 +6.0%，超越 VoxPoser 约 +7.9%。更重要的是，X-ICM（72B）是唯一在全部 23 个任务中均获得非零成功率的方法，而其他基线模型在 ≥8 个任务上完全失败。On Overall, X-ICM (72B) exceeds π0 by about +6.0% and VoxPoser by about +7.9%. More importantly, X-ICM (72B) is the only method with a non-zero success rate on all 23 tasks, whereas the other baselines fail completely on ≥8 tasks.

Ablation：动态引导选择模块的作用（Table 3）Ablation: effect of the dynamics-guided selection module (Table 3)

配置Configuration	Level-1	Level-2	Overall
X-ICM（72B）无选择模块X-ICM (72B) without the selection module	30.7% ± 4.7	18.0% ± 2.2	25.2% ± 3.2
X-ICM（72B）含选择模块X-ICM (72B) with the selection module	37.6% ± 1.4	20.3% ± 1.7	30.1% ± 1.0

动态引导选择模块将 Overall 从 25.2% 提升至 30.1%（+4.9%），同时将标准差从 ±3.2 显著降至 ±1.0，表明该模块不仅提升了性能，还大幅增强了预测的稳定性。此外，上下文示例数量 K 在超过 12 条后性能趋于饱和。The dynamics-guided selection module raises Overall from 25.2% to 30.1% (+4.9%) and at the same time cuts the standard deviation markedly from ±3.2 to ±1.0, showing that it improves not only performance but also the stability of the predictions. Moreover, performance saturates once the number of in-context demonstrations K goes beyond 12.

effect of demo count — **Figure 3：上下文示例数量的影响。**随着 K 增加，X-ICM 性能持续提升，但在 K≈12 后趋于平稳，说明过多示例不再带来额外收益。**Figure 3: Effect of the number of in-context demonstrations.** X-ICM keeps improving as K grows, but levels off beyond K≈12, showing that additional demonstrations bring no further gain.

真实世界实验Real-world experiments

real-world results — **Figure 4：五个真实物理任务的零样本跨任务测试结果。**X-ICM 在单步任务上表现合理，但在长流程任务（"clean the table"，需多步骤串联）上仅达到 5% 成功率，源于各子步骤错误的累积放大效应。**Figure 4: Zero-shot cross-task results on five real physical tasks.** X-ICM performs reasonably on single-step tasks, but reaches only a 5% success rate on the long-horizon task ("clean the table", which chains several steps), because errors of the sub-steps accumulate and amplify.

04 局限性Limitations

说明：以下局限性均为论文作者明确陈述（stated）。Note: every limitation below is explicitly stated by the authors of the paper.

视觉信息利用不足（Limited Visual Understanding）Limited Visual Understanding

X-ICM 将视觉信息压缩为对象坐标文本，原文指出 "the use of visual information is limited to textualizing object information, which may ignore important visual context in the raw data"。这意味着纹理、形状、场景上下文等精细视觉线索无法被 LLM 直接利用，对于需要精细视觉感知的任务（如区分相似外观物体）效果受限。X-ICM compresses visual information into object-coordinate text; the paper points out that "the use of visual information is limited to textualizing object information, which may ignore important visual context in the raw data". Fine-grained visual cues such as texture, shape and scene context therefore cannot be used directly by the LLM, which limits performance on tasks that require fine visual perception (for example telling apart objects with similar appearance).

LLM 外推能力的上限（Extrapolation Limits of LLMs）Extrapolation Limits of LLMs

原文指出 "X-ICM's performance on many unseen tasks...remains limited due to LLMs' challenges in extrapolating beyond pre-training data"。当未见任务的概念完全超出 LLM 的预训练分布时，其动作推断能力将显著退化，特别是 Level-2 中具有全新对象和动作原语的任务。The paper points out that "X-ICM's performance on many unseen tasks...remains limited due to LLMs' challenges in extrapolating beyond pre-training data". When the concepts of an unseen task fall entirely outside the pre-training distribution of the LLM, its action inference degrades markedly, especially for Level-2 tasks with brand-new objects and action primitives.

长流程任务的累积失败（Cascading Failures in Long-Horizon Tasks）Cascading Failures in Long-Horizon Tasks

真实世界"clean the table"任务的整体成功率仅为 5%，原因是多步骤串联任务中各子步骤的错误相互叠加。这是 in-context 预测框架的固有缺陷：无法在线修正中间步骤的偏差。The overall success rate on the real-world "clean the table" task is only 5%, because the errors of the sub-steps in a multi-step chain pile onto one another. This is an inherent flaw of the in-context prediction framework: it cannot correct the deviations of intermediate steps online.

全新语义组合的挑战（Novel Semantic Combinations）Novel Semantic Combinations

Level-2 中同时包含新对象与新动作原语的任务对所有方法均构成最大挑战。X-ICM（72B）在 Level-2 的 20.3% 成功率虽优于基线，但仍远低于 Level-1 的 37.6%，反映出当前 LLM 的跨任务外推能力边界。Tasks in Level-2 that contain both new objects and new action primitives are the hardest for every method. The 20.3% success rate of X-ICM (72B) on Level-2 beats the baselines but stays far below the 37.6% on Level-1, which reflects the boundary of the cross-task extrapolation ability of current LLMs.