ASkDAgger：面向交互式模仿学习的
主动技能级数据聚合ASkDAgger: Active Skill-level Data Aggregation
for Interactive Imitation Learning

01 动机（Motivation）Motivation

交互式模仿学习（Interactive Imitation Learning, IIL）允许人类在机器人执行任务时提供示范与纠错，是解决协方差偏移（covariate shift）问题的有效手段。然而，人类的教学代价是 IIL 大规模落地的核心瓶颈。Interactive Imitation Learning (IIL) lets humans provide demonstrations and corrections while the robot is performing a task, and is an effective way to address the covariate shift problem. However, human teaching effort is the core bottleneck for applying IIL at scale.

"Human teaching effort is a significant bottleneck for the broader applicability of interactive imitation learning." —— ASkDAgger 摘要—— ASkDAgger abstract

现有方法的不足Shortcomings of existing approaches

现有主动 DAgger 方法（如 SafeDAgger、ThriftyDAgger）在向教师发出查询时，只是简单地移交控制权，完全抛弃了菜鸟的计划动作。这造成两个损失：When existing active DAgger methods (such as SafeDAgger and ThriftyDAgger) query the teacher, they simply hand over control and completely discard the novice's planned action. This incurs two losses:

• 无法利用菜鸟动作所揭示的不确定性分布来动态调节查询阈值；the uncertainty distribution revealed by the novice action cannot be used to adjust the query threshold dynamically;
• 即便菜鸟动作是正确的，也不能将其复用为示范，白白浪费了潜在的训练信号。even when the novice action is correct, it cannot be reused as a demonstration, so a potential training signal is wasted.

ASkDAgger 的核心思路The core idea of ASkDAgger

ASkDAgger 让菜鸟在不确定时说："I plan to do this, but I am uncertain."——教师随即对该计划进行验证（validate）、重标注（relabel）或拒绝（reject），并将反馈信息用于三个方面：自适应门控、示范数据集构建、优先回放。ASkDAgger lets the novice say, when it is uncertain: "I plan to do this, but I am uncertain." — the teacher then validates, relabels, or rejects that plan, and the feedback is used in three ways: adaptive gating, demonstration dataset construction, and prioritized replay.

02 方法（Method）Method

ASkDAgger 在每个时间步让菜鸟以一定概率向教师提交计划动作，教师给出反馈后，框架通过三个组件分别利用该反馈：自适应调节查询门控（SAG）、将计划复用为示范（FIER）、按优先级回放（PIER）。At every time step ASkDAgger lets the novice submit its planned action to the teacher with a certain probability; once the teacher has given feedback, the framework exploits that feedback through three components: adaptively adjusting the query gate (SAG), reusing the plan as a demonstration (FIER), and prioritized replay (PIER).

问题设定Problem setting

形式上，ASkDAgger 处理交互式模仿学习（IIL）问题：机器人菜鸟与人类教师交替交互，菜鸟学习从观测 o 到技能参数 a 的映射（即目标条件化的可供性 goal-conditioned affordance）。每个时间步 t，菜鸟输出动作 a_t 及对应的不确定性 u_t。若不确定性超过自适应阈值 γ，则以概率 p_query 向教师发出查询，请教师评价菜鸟的计划动作。教师反馈 r_t 指示该动作是否适用于当前目标 g（成功/失败），用于更新 SAG 阈值与 PIER 优先级。Formally, ASkDAgger addresses the interactive imitation learning (IIL) problem: the robot novice and the human teacher interact in turn, and the novice learns a mapping from observation o to skill parameters a (that is, a goal-conditioned affordance). At each time step t the novice outputs an action a_t together with the corresponding uncertainty u_t. If the uncertainty exceeds the adaptive threshold γ, a query is issued to the teacher with probability p_query, asking the teacher to evaluate the novice's planned action. The teacher feedback r_t indicates whether the action is applicable to the current goal g (success/failure), and is used to update the SAG threshold and the PIER priorities.

与基线方法的关键区别Key differences from the baseline methods

SafeDAgger（Zhang & Cho, 2017）使用固定启发式阈值；ThriftyDAgger（Hoque et al., 2022）使用固定查询率；现有主动 DAgger 变体在查询时均放弃菜鸟动作。ASkDAgger 则首次系统性地利用了查询时菜鸟动作所携带的信息，在三个层面（门控、数据集构建、回放）同时获益。SafeDAgger (Zhang & Cho, 2017) uses a fixed heuristic threshold; ThriftyDAgger (Hoque et al., 2022) uses a fixed query rate; all existing active DAgger variants throw away the novice action when they query. ASkDAgger is the first to systematically exploit the information carried by the novice action at query time, gaining on three levels at once (gating, dataset construction, replay).

03 实验（Experiments）Experiments

论文在三类实验平台上验证了四条核心主张（C1–C4）：MNIST 数字分类（验证 SAG）、CLIPort 语言条件化桌面操作仿真（验证 C1–C3）、以及真实世界机器人装配与分拣任务（验证 C1–C4 的实际适用性）。The paper validates four core claims (C1–C4) on three kinds of experimental platform: MNIST digit classification (validating SAG), CLIPort language-conditioned tabletop manipulation in simulation (validating C1–C3), and real-world robot assembly and sorting tasks (validating the practical applicability of C1–C4).

CLIPort 仿真基准对比（Claims C1–C3）CLIPort simulation benchmark comparison (Claims C1–C3)

在 CLIPort（Shridhar et al., 2021）框架下，训练 10 个智能体，采集 300 条交互式示范，在 4 个任务上与以下基线对比：active DAgger（无 FIER 和 PIER）、SafeDAgger、ThriftyDAgger，以及 ASkDAgger w/o FIER、ASkDAgger w/o PIER 消融组。超参数：mode = sensitivity，σ_des = 0.9，N_min = 15，p_rand = 0.2；PIER：α = 1.5，b = 10，β = 1，λ = 0.5。Under the CLIPort (Shridhar et al., 2021) framework, 10 agents are trained and 300 interactive demonstrations are collected, and on 4 tasks the method is compared with the following baselines: active DAgger (without FIER and PIER), SafeDAgger, ThriftyDAgger, plus the ASkDAgger w/o FIER and ASkDAgger w/o PIER ablations. Hyperparameters: mode = sensitivity, σ_des = 0.9, N_min = 15, p_rand = 0.2; PIER: α = 1.5, b = 10, β = 1, λ = 0.5.

主张Claim	实验证据Evidence	结论Conclusion
C1：SAG 追踪指定度量C1: SAG tracks the specified metric	MNIST 9 种目标值 × 10 次重复MNIST, 9 target values × 10 repetitions	实测指标与 σdes 高度吻合measured metric closely matches σdes
C2：FIER 减少标注量C2: FIER reduces annotation effort	CLIPort 4 任务，ASkDAgger vs. active DAggerCLIPort, 4 tasks, ASkDAgger vs. active DAgger	ASkDAgger 标注示范量显著更低ASkDAgger needs markedly fewer annotated demonstrations
C3：FIER 提升泛化C3: FIER improves generalization	CLIPort unseen 任务集（颜色/形状/物体未见）CLIPort unseen task suites (unseen colors / shapes / objects)	relabeling 带来对 unseen 目标的更优表现relabeling brings better performance on unseen goals
C4：PIER 加速域迁移C4: PIER speeds up domain transfer	三阶段域迁移（seen-shapes → unseen-shapes → seen-google-objects）three-stage domain transfer (seen-shapes → unseen-shapes → seen-google-objects)	ASkDAgger 成功率更高，标注更少ASkDAgger reaches a higher success rate with fewer annotations

结果摘要（来自论文 Fig. 5–6）：在所有任务上，ASkDAgger 的累积奖励表现等于或优于所有基线，同时所需教师标注示范（annotation tuples）数量显著更少——因为 ASkDAgger 将大量菜鸟验证动作（validation）和重标注动作（relabeling）直接用作示范，不需要额外的教师标注。Summary of the results (from Fig. 5–6 of the paper): on every task the cumulative reward of ASkDAgger matches or exceeds all baselines, while the number of teacher-annotated demonstrations (annotation tuples) required is markedly lower — because ASkDAgger uses a large number of validated and relabeled novice actions directly as demonstrations, which need no additional teacher annotation.

域迁移实验（Claim C4）Domain transfer experiment (Claim C4)

在三阶段域迁移设置下（每阶段 150 条示范），ASkDAgger（含 PIER）相比 ASkDAgger w/o PIER：在 packing-seen-shapes、packing-unseen-shapes、packing-seen-google-objects-seq 三个连续任务上均实现更高的系统成功率，并在后期阶段收集更少的 annotation tuples（更多来自验证示范）。SAG 在整个训练过程中维持了设定的 sensitivity 水平。In the three-stage domain transfer setting (150 demonstrations per stage), ASkDAgger (with PIER) compared with ASkDAgger w/o PIER reaches a higher system success rate on all three consecutive tasks — packing-seen-shapes, packing-unseen-shapes, and packing-seen-google-objects-seq — and collects fewer annotation tuples in the later stages (more of them coming from validated demonstrations). SAG maintains the specified sensitivity level throughout training.

真实世界实验Real-world experiments

在 Franka Panda 机器人上进行引擎装配任务（4 种颜色螺栓 × 7 个安装位置，150 条示范）：系统成功率维持在较高水平，同时随训练进行，验证示范（validation）比例逐渐增大，标注示范（annotation）比例降低——表明菜鸟策略在实际场景中逐渐无需人工干预即可独立完成任务。此外，在 Boston Dynamics Spot 机器人上的分拣任务也展示了 ASkDAgger 与内置原始技能（primitive skills）集成的能力。An engine assembly task was run on a Franka Panda robot (4 bolt colors × 7 mounting positions, 150 demonstrations): the system success rate stayed at a high level, while as training proceeded the share of validated demonstrations grew and the share of annotated demonstrations fell — showing that in a real setting the novice policy gradually completes the task on its own without human intervention. In addition, a sorting task on a Boston Dynamics Spot robot demonstrated the ability of ASkDAgger to integrate with built-in primitive skills.

消融分析Ablation analysis

消融实验（Fig. 8）表明：去除 FIER（ASkDAgger w/o FIER）后，unseen 任务集上的性能明显下降，因为失去了通过 relabeling 获取 unseen 场景示范的途径；去除 PIER（ASkDAgger w/o PIER）后，域迁移阶段的自适应速度变慢，所需标注量更多。两个组件缺一不可。The ablation study (Fig. 8) shows that removing FIER (ASkDAgger w/o FIER) clearly degrades performance on the unseen task suites, because the route to obtaining demonstrations for unseen scenarios through relabeling is lost; removing PIER (ASkDAgger w/o PIER) slows down adaptation during domain transfer and requires more annotations. Neither component can be dispensed with.

04 局限性（Limitations）Limitations

仅适用于中/高层稀疏奖励控制，不适用于高频低层控制Applies only to mid- and high-level sparse-reward control, not to high-rate low-level control

"ASkDAgger is designed for tasks with sparse rewards and learning mid- to high-level control. While this covers many problems, it does not extend to applications requiring high-rate feedback from the teacher." 因此，ASkDAgger 最适合机器人已有预定义原始技能（primitive skills），需要学习技能选择与参数化的场景，而对于需要连续高频反馈的低层控制任务，其适用性受限。"ASkDAgger is designed for tasks with sparse rewards and learning mid- to high-level control. While this covers many problems, it does not extend to applications requiring high-rate feedback from the teacher." ASkDAgger is therefore best suited to settings in which the robot already has predefined primitive skills and has to learn skill selection and parameterization, whereas for low-level control tasks that need continuous high-rate feedback its applicability is limited.

FIER 依赖"失败动作可被重标注为其他目标的成功示范"这一前提FIER relies on the premise that failed actions can be relabeled as successful demonstrations for other goals

"FIER significantly improves performance, it relies on recasting failures as successes, which can be challenging in some applications." 若任务结构不允许同一动作对应多个目标（如动作空间缺乏重叠），FIER 的优势会大打折扣。"FIER significantly improves performance, it relies on recasting failures as successes, which can be challenging in some applications." If the task structure does not allow one action to correspond to several goals (for instance when the action space lacks overlap), the advantage of FIER is greatly diminished.

要求教师在执行前预先验证或重标注菜鸟计划Requires the teacher to validate or relabel the novice plan before execution

"ASkDAgger assumes the teacher can validate or relabel actions before execution. This may not always be feasible." 在实时性要求极高或教师不方便在线干预的场景下，执行前反馈可能无法实现。作者指出，在此情况下，验证和重标注可在执行后进行（post-execution），但会影响交互流程。"ASkDAgger assumes the teacher can validate or relabel actions before execution. This may not always be feasible." In settings with extremely tight real-time requirements, or where the teacher cannot conveniently intervene online, pre-execution feedback may be infeasible. The authors note that in such cases validation and relabeling can be carried out post-execution, but this affects the interaction flow.

真实世界中超参数调优仍有挑战Hyperparameter tuning in the real world is still challenging

"Tuning hyperparameters in real-world settings is challenging. However, our experiments show that a single set of hyperparameters, without extensive tuning, generalizes effectively across different tasks, suggesting that hyperparameter sensitivity is not strongly dependent on task descriptions." 作者指出问题的存在，但也提供了缓解证据。"Tuning hyperparameters in real-world settings is challenging. However, our experiments show that a single set of hyperparameters, without extensive tuning, generalizes effectively across different tasks, suggesting that hyperparameter sensitivity is not strongly dependent on task descriptions." The authors point out that the problem exists, but also provide evidence that it is mitigated.

实验主要基于 CLIPort 架构，策略架构的泛用性待进一步验证The experiments are mainly based on the CLIPort architecture; generality across policy architectures awaits further validation

"Our evaluations primarily used policies with a CLIPort architecture, but ASkDAgger is not limited to this choice. It can be applied to any policy architecture where a teacher can determine the success of actions and provide demonstrations." ASkDAgger 原则上架构无关，但在其他策略架构（如扩散策略、Transformer 等）上的实证验证尚不充分。"Our evaluations primarily used policies with a CLIPort architecture, but ASkDAgger is not limited to this choice. It can be applied to any policy architecture where a teacher can determine the success of actions and provide demonstrations." ASkDAgger is architecture-agnostic in principle, but empirical validation on other policy architectures (diffusion policies, Transformers and so on) is still insufficient.