AHA: 检测与推理机器人操作失败的视觉语言模型

01 动机 MotivationMotivation

机器人在开放世界环境中执行操作任务时，不仅需要完成任务本身，还需要能够检测并学习自身失败。现有 VLM 和 LLM 虽然提升了机器人的空间推理与问题求解能力，但在失败识别上依然薄弱，限制了其实际应用价值。When robots carry out manipulation tasks in open-world environments, they must not only complete the task itself but also be able to detect and learn from their own failures. Existing VLMs and LLMs have improved robots' spatial reasoning and problem-solving abilities, yet they remain weak at failure recognition, which limits their practical value.

"How can we enable these models to autonomously detect and reason about their own failures, particularly in robotics, where interactions and environments are stochastic and unpredictable?"

现有方法将失败检测视为二元分类问题（成功 / 失败），无法解释为何失败。AHA 将其重新定义为自由形式推理任务：先判断子任务是否成功，若失败则生成简洁的自然语言解释，说明失败的原因与发生方式。这使得 AHA 能够跨越不同机器人形态、相机视角、任务和环境进行泛化，并无缝集成到利用 VLM/LLM 的下游机器人应用中。Existing methods treat failure detection as a binary classification problem (success / failure) and cannot explain why a failure occurred. AHA reformulates it as a free-form reasoning task: first judge whether a subtask succeeded, and if it failed, generate a concise natural-language explanation of why and how the failure happened. This lets AHA generalize across different robot embodiments, camera viewpoints, tasks and environments, and integrate seamlessly into downstream robotic applications built on VLMs/LLMs.

AHA 应用场景概览 — **图 1：AHA 应用场景。**AHA 是一个专为机器人操作失败检测与推理设计的视觉语言模型，可嵌入三类利用 VLM 的机器人应用：（上）VLM 子任务验证（Manipulate-Anything 框架）；（左下）VLM 奖励函数生成（Eureka 框架）；（右下）VLM 任务规划生成（Task and Motion Planning）。在各框架中，AHA 提供的失败反馈能够加速并提升任务执行性能。**Figure 1: AHA application scenarios.** AHA is a vision-language model designed for failure detection and reasoning in robotic manipulation, and can be embedded into three classes of robotic applications that leverage VLMs: (top) VLM subtask verification (the Manipulate-Anything framework); (bottom left) VLM reward-function generation (the Eureka framework); (bottom right) VLM task-plan generation (Task and Motion Planning). In each framework, the failure feedback provided by AHA accelerates and improves task execution performance.

+10.3%超越 GPT-4o ICL
（第二佳模型）over GPT-4o ICL
(second-best model)

+35.3%超越六模型平均
（含五个 SOTA VLM）over the six-model average
(including five SOTA VLMs)

+21.4%下游三任务
平均成功率提升average success-rate gain
on three downstream tasks

49K+AHA 数据集
失败轨迹图文对AHA dataset
failure-trajectory image-text pairs

02 方法 MethodMethod

AHA 的方法包含两个核心组成部分：一是 FailGen——用于程序化生成失败演示数据的可扩展数据生成框架；二是基于 LLaVA 架构的指令微调流程，使 VLM 能够完成失败推理任务。AHA's approach comprises two core components: first, FailGen—a scalable data-generation framework for procedurally generating failure demonstrations; second, an instruction-tuning pipeline built on the LLaVA architecture that equips the VLM for failure reasoning.

AHA 整体流程图 — **图 2：AHA 整体流程概览。**（上）数据生成：以正常任务轨迹为输入，通过 FailGen 在仿真中对所有关键帧进行程序化扰动，系统性地生成失败演示，并同步生成对应的问答提示对用于微调。（下）指令微调：遵循 LLaVA-v1.5 相同的微调流程，仅更新语言模型（LLaMA-2-13B）和投影层权重，冻结图像编码器与分词器。**Figure 2: Overview of the AHA pipeline.** (Top) Data generation: taking normal task trajectories as input, FailGen procedurally perturbs every keyframe in simulation to systematically generate failure demonstrations, together with the corresponding question-answer prompt pairs used for fine-tuning. (Bottom) Instruction tuning: following the same fine-tuning procedure as LLaVA-v1.5, only the language model (LLaMA-2-13B) and the projection-layer weights are updated, while the image encoder and the tokenizer are frozen.

失败推理问题定式（Failure Reasoning Formulation）Failure Reasoning Formulation

AHA 将机器人操作轨迹定义为一系列子任务 {S₀, S₁, S₂, …, Sₜ}，每个子任务由两个连续关键帧表示。对于每个子任务，模型接收：（1）描述当前子任务的文本提示；（2）包含多视角关键帧轨迹的结构化图像矩阵 I（行为相机视角 V₀…Vₙ，列为时序关键帧）。模型需要输出：子任务是否成功（"Yes"/"No"），若失败则给出简洁的自然语言解释。这种格式化输入支持跨不同数据集和视角对数据的统一处理。AHA defines a robotic manipulation trajectory as a sequence of subtasks {S₀, S₁, S₂, …, Sₜ}, each subtask being represented by two consecutive keyframes. For every subtask the model receives: (1) a text prompt describing the current subtask; (2) a structured image matrix I holding multi-view keyframe trajectories (rows are camera viewpoints V₀…Vₙ, columns are temporal keyframes). The model must output whether the subtask succeeded ("Yes"/"No"), and, if it failed, a concise natural-language explanation. This formatted input enables unified processing of data across different datasets and viewpoints.

FailGen：失败数据自动生成框架FailGen: an automatic framework for generating failure data

FailGen 是一个可应用于任意机器人仿真器的环境包装器，通过对关键帧进行扰动将成功轨迹转化为失败轨迹。论文定义了七种失败模式的分类体系：FailGen is an environment wrapper that can be applied to any robot simulator, turning successful trajectories into failed ones by perturbing keyframes. The paper defines a taxonomy of seven failure modes:

以对象为中心（Object-centric）Object-centric

No_Grasp：夹爪到达目标抓取位姿但未闭合，无法完成抓取No_Grasp: the gripper reaches the target grasp pose but does not close, so the grasp is never completed
Slip：成功抓取后在移动过程中夹持力不足导致物体滑落Slip: after a successful grasp, insufficient gripping force during motion lets the object slip away
Wrong_Object：夹爪操作了错误的目标物体Wrong_Object: the gripper manipulates the wrong target object

以动作为中心（Action-centric）Action-centric

Translation：关键帧沿 X/Y/Z 轴存在位移偏移Translation: the keyframe carries a translational offset along the X/Y/Z axes
Rotation：关键帧在 roll/yaw/pitch 上存在旋转偏差Rotation: the keyframe carries a rotational deviation in roll/yaw/pitch
No_Rotation：到达平移位姿但未完成必要旋转No_Rotation: the translational pose is reached but the required rotation is not performed
Wrong_Action：关键帧执行顺序错误Wrong_Action: the keyframes are executed in the wrong order

FailGen 通过遍历 RLBench 中每个任务的所有关键帧，针对七种失败模式穷举所有可能配置，生成 YAML 格式配置文件，并结合语言模板自动标注问答对。最终在 79 个仿真任务中生成了包含 49K 图像-文本对的 AHA 数据集。FailGen 还成功扩展到 ManiSkill 仿真器，证明了其跨仿真环境的通用性。By traversing every keyframe of every task in RLBench, FailGen exhaustively enumerates all possible configurations for the seven failure modes, emits YAML configuration files, and automatically annotates question-answer pairs with language templates. It ultimately produces the AHA dataset, containing 49K image-text pairs, over 79 simulated tasks. FailGen has also been extended successfully to the ManiSkill simulator, demonstrating its generality across simulation environments.

指令微调数据混合（Co-finetuning Data Mix）Co-finetuning Data Mix

为保留 VLM 的通用视觉理解能力，AHA 采用数据混合策略：将 AHA 数据集（49K 失败推理对）与 VQA 数据集（665K 对话对）和 LVIS 数据集（100K 目标检测实例）联合用于微调。微调时，仅更新 LLM（LLaMA-2-13B）参数和 2 层投影 MLP，图像编码器与分词器保持冻结。To preserve the VLM's general visual understanding, AHA adopts a data-mixing strategy: the AHA dataset (49K failure-reasoning pairs) is fine-tuned jointly with a VQA dataset (665K conversation pairs) and the LVIS dataset (100K object-detection instances). During fine-tuning, only the LLM (LLaMA-2-13B) parameters and a 2-layer projection MLP are updated, while the image encoder and the tokenizer stay frozen.

03 实验 ExperimentsExperiments

AHA 在三个评测数据集上与六个 SOTA VLM（含开源和闭源模型）进行了对比，采用四个评估指标；同时在三项下游机器人任务中验证其实用价值。AHA is compared with six SOTA VLMs (both open-source and closed-source) on three evaluation datasets under four metrics, and its practical value is further verified on three downstream robotic tasks.

评测基准与指标Benchmarks and metrics

三个评测数据集均未出现在微调数据中，覆盖不同机器人形态、任务和环境：None of the three evaluation datasets appear in the fine-tuning data, and together they span different robot embodiments, tasks and environments:

AHA dataset (Test)：11K 图像-问题对，来自 10 个 RLBench 任务（与训练集无重叠），测试同域泛化AHA dataset (Test): 11K image-question pairs drawn from 10 RLBench tasks (no overlap with the training set), testing in-domain generalization
ManiSkill-Fail：130 对，由 FailGen 在 ManiSkill 仿真器上生成，测试跨仿真域泛化ManiSkill-Fail: 130 pairs generated by FailGen on the ManiSkill simulator, testing cross-simulator generalization
RoboFail：来自真实 UR5 机器人的失败数据，测试仿真-到-现实泛化RoboFail: failure data from a real UR5 robot, testing sim-to-real generalization

四个评估指标：ROUGE-L（文本质量）、Cosine Similarity（语义相似度）、Binary Success（成功检测准确率）、LLM Fuzzy Match（使用 claude-3-sonnet 评估语义匹配）。Four evaluation metrics: ROUGE-L (text quality), Cosine Similarity (semantic similarity), Binary Success (success-detection accuracy), and LLM Fuzzy Match (semantic matching judged with claude-3-sonnet).

定量评测结果 — **表 2：失败检测与推理的定量评测。**AHA-13B 在几乎所有评测数据集和指标上均超越所有对比 VLM（AHA 测试集上仅 GPT-4o 在 Binary Success 指标上与 AHA 差距小于 3%）。AHA-13B 在 ManiSkill-Fail 上以 LLM Fuzzy Match 0.633 超越 GPT-4o ICL 的 0.630；在 RoboFail 上以 0.465 超越 GPT-4o ICL 的 0.418。**Table 2: Quantitative evaluation of failure detection and reasoning.** AHA-13B surpasses every compared VLM on almost all evaluation datasets and metrics (on the AHA test set, only GPT-4o comes within 3% of AHA on the Binary Success metric). On ManiSkill-Fail, AHA-13B attains an LLM Fuzzy Match of 0.633, above GPT-4o ICL's 0.630; on RoboFail it attains 0.465, above GPT-4o ICL's 0.418.

主要量化结论Main quantitative findings

对比维度Comparison	基线Baseline	AHA-13B	提升幅度Improvement
vs. 第二佳模型（GPT-4o ICL）vs. second-best model (GPT-4o ICL)	—	超越outperforms	平均 +10.3%+10.3% on average
vs. 六模型平均vs. six-model average	—	超越outperforms	平均 +35.3%+35.3% on average
vs. LLaVA-v1.5-13B（基础模型）vs. LLaVA-v1.5-13B (base model)	0.351 (RoboFail LLM Fuzzy)	0.465	+43.0%（平均跨数据集）+43.0% (averaged across datasets)
ManiSkill-Fail Binary Success	GPT-4o ICL: 0.971	1.000	完美检测率perfect detection rate

通用 VQA 能力保留Retention of general VQA ability

在标准 VQA 基准测试中，AHA-13B 与 LLaVA-v1.5-13B（基础模型）表现相当（如 MMBench 65.20 vs. 67.70，TextVQA 65.20 vs. 67.40，VizWiz 53.45 vs. 53.01），证明微调后 AHA 保留了通用 VLM 能力。On standard VQA benchmarks, AHA-13B performs on par with LLaVA-v1.5-13B (the base model) (e.g. MMBench 65.20 vs. 67.70, TextVQA 65.20 vs. 67.40, VizWiz 53.45 vs. 53.01), showing that AHA retains general VLM ability after fine-tuning.

数据缩放规律与下游任务性能 — **图 3：（左）AHA 数据集的缩放规律。**使用 3k、6k、12k、34k、48k、60k 规模数据微调，模型在 ManiSkill-Fail 上的平均二次拟合梯度为 0.0022，表明进一步扩大数据可持续提升性能。**（右）下游机器人任务性能。**AHA-13B 在奖励函数生成（Eureka）、任务规划（TAMP/PRoC3S）和子任务验证（Manipulate-Anything）三类应用中，均超越 GPT-4o，平均任务成功率提升 21.4%。**Figure 3: (Left) Scaling law of the AHA dataset.** Fine-tuning with data at scales of 3k, 6k, 12k, 34k, 48k and 60k, the model's mean quadratic-fit gradient on ManiSkill-Fail is 0.0022, indicating that scaling the data further keeps improving performance. **(Right) Downstream robotic task performance.** Across three classes of applications—reward-function generation (Eureka), task planning (TAMP/PRoC3S) and subtask verification (Manipulate-Anything)—AHA-13B surpasses GPT-4o, raising the average task success rate by 21.4%.

下游机器人应用Downstream robotic applications

奖励函数生成（Eureka + ManiSkill）：在 5 个强化学习任务上，AHA 提供的失败推理反馈帮助改进稠密奖励函数，任务成功率超越 GPT-4o 达 22.34%。Reward-function generation (Eureka + ManiSkill): on 5 reinforcement-learning tasks, the failure-reasoning feedback provided by AHA helps refine dense reward functions, exceeding GPT-4o in task success rate by 22.34%.
任务规划（PRoC3S/TAMP）：在 3 个任务各 10 次试验中，AHA 识别失败原因并反馈至 LMP 重规划环节，超越 GPT-4o 达 36.7%。Task planning (PRoC3S/TAMP): over 3 tasks with 10 trials each, AHA identifies the cause of failure and feeds it back into the LMP replanning stage, exceeding GPT-4o by 36.7%.
子任务验证（Manipulate-Anything）：替换原 GPT-4V 子任务验证模块后，在 4 个 RLBench 任务（各 25 幕）中整体任务成功率平均提升 5%。Subtask verification (Manipulate-Anything): after replacing the original GPT-4V subtask-verification module, the overall task success rate rises by 5% on average over 4 RLBench tasks (25 episodes each).

04 局限性 LimitationsLimitations

说明：以下局限性内容均由论文作者在 Section 6 (Conclusion) 中明确陈述。Note: All the limitations below are explicitly stated by the authors in Section 6 (Conclusion).

失败推理与微调数据分布高度相关Failure reasoning is tightly coupled to the fine-tuning data distribution

"AHA currently outputs language reasoning that is closely aligned with the failure scenarios in the fine-tuning data."（AHA 目前的语言推理输出与微调数据中的失败场景高度对齐。）这意味着对于超出七种失败模式分类体系之外的开放式失败，AHA 的输出能力受到限制，泛化范围有边界。"AHA currently outputs language reasoning that is closely aligned with the failure scenarios in the fine-tuning data." This means that for open-ended failures lying outside the seven-mode failure taxonomy, AHA's output ability is constrained and its generalization has a boundary.

FailGen 依赖仿真器中的成功演示蒸馏FailGen relies on distilling successful demonstrations inside a simulator

"FailGen systematically curates failure data from simulations, distilling large pretrained policies to perform diverse tasks in simulation and sampling failure modes would allow us to generate more open-ended failure examples, potentially enhancing the instruction-tuned performance of AHA."（FailGen 从仿真中系统地整理失败数据，若能从大型预训练策略中蒸馏多样化任务并在仿真中采样失败模式，可生成更具开放性的失败示例，进一步提升 AHA 的微调表现。）目前数据来源仍主要依赖程序化扰动而非策略自然失败。"FailGen systematically curates failure data from simulations, distilling large pretrained policies to perform diverse tasks in simulation and sampling failure modes would allow us to generate more open-ended failure examples, potentially enhancing the instruction-tuned performance of AHA." For now the data still comes mainly from procedural perturbation rather than from the natural failures of a policy.

开放式失败覆盖范围有待扩展（推断）Coverage of open-ended failures remains to be extended (inferred)

仅覆盖七类预定义失败模式，真实世界中可能出现的其他失败类型（如传感器噪声、环境光变化、动态遮挡等）尚未被 AHA 数据集系统性涵盖。这是从论文设计中推断出的潜在局限，论文中未单独列举。Only the seven predefined failure modes are covered; other failure types that may arise in the real world (sensor noise, changing ambient light, dynamic occlusion, and so on) are not yet systematically covered by the AHA dataset. This is a potential limitation inferred from the paper's design and is not listed separately in the paper.