HIL-SERL：人机协作强化学习实现精准灵巧机器人操作HIL-SERL: Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning

01 动机Motivation

强化学习（RL）在机器人操作领域极具潜力，但在真实世界中落地依然困难重重——样本效率低、奖励难以定义、训练时间过长，导致现有系统难以超越模仿学习。Reinforcement learning (RL) holds great promise for robotic manipulation, yet deploying it in the real world remains difficult: low sample efficiency, rewards that are hard to define, and excessive training time have kept existing systems from surpassing imitation learning.

"Realizing this potential in real-world settings has been challenging due to issues with sample complexity, assumptions (e.g., accurate reward functions), and optimization stability."

现有真实世界 RL 方法往往只能应对简单任务，而模仿学习（BC、DAgger）虽然易于部署，但在需要连续反应行为的精密任务上存在固有的性能上限。本文提出 HIL-SERL 系统，核心洞察在于：Existing real-world RL methods can usually handle only simple tasks, while imitation learning (BC, DAgger), though easy to deploy, has an inherent performance ceiling on precise tasks that require continuous reactive behavior. This paper proposes the HIL-SERL system, whose core insights are:

• 人类在线纠错（online corrections）可大幅缩小状态空间的有效覆盖难度，让 RL 探索更高效；Human online corrections greatly reduce the difficulty of effectively covering the state space, making RL exploration more efficient;
• 预训练视觉骨干提供稳定表征，避免从零训练视觉特征的巨大样本开销；A pretrained vision backbone provides stable representations, avoiding the huge sample cost of learning visual features from scratch;
• off-policy RL（RLPD）结合示范 replay buffer，实现极高的数据利用率。Off-policy RL (RLPD), combined with a demonstration replay buffer, achieves extremely high data efficiency.

02 方法Method

HIL-SERL 系统由三个异步并行的核心组件构成：Actor 进程（在机器人上执行策略）、Learner 进程（持续更新策略参数）、以及两个 Replay Buffer（分别存储人类示范数据与 on-policy 数据）。The HIL-SERL system consists of three asynchronously parallel core components: the actor process (executing the policy on the robot), the learner process (continuously updating the policy parameters), and two replay buffers (storing human demonstration data and on-policy data respectively).

核心 RL 算法：RLPD（off-policy + 示范融合）Core RL algorithm: RLPD (off-policy + demonstration fusion)

底层 RL 算法选用 RLPD（Ball et al., 2023），其在每个训练步骤中从 demo buffer 和 RL buffer 等比采样构成训练批次。RLPD 以 SAC 为基础，同时更新 Q 函数 Q_φ(s, a) 与策略 π_θ(a|s)。奖励函数采用稀疏二值奖励——由离线训练的 ResNet-based 视觉分类器判断任务是否成功（准确率通常 > 95%）。初始化阶段收集 20–30 条人类遥操作演示进入 demo buffer。The underlying RL algorithm is RLPD (Ball et al., 2023), which forms every training batch by sampling in equal proportion from the demo buffer and the RL buffer at each training step. RLPD builds on SAC and jointly updates the Q function Q_φ(s, a) and the policy π_θ(a|s). The reward is a sparse binary reward — an offline-trained ResNet-based visual classifier decides whether the task has succeeded (accuracy typically > 95%). In the initialization phase, 20–30 human teleoperated demonstrations are collected into the demo buffer.

预训练视觉骨干（Pretrained Vision Backbone）Pretrained Vision Backbone

所有摄像头图像（腕部相机 + 侧视相机）统一经过 ResNet-10（ImageNet 预训练）提取嵌入向量，再与本体感知信息拼接。使用预训练骨干带来两重好处：（1）优化稳定性更高；（2）探索效率更好——避免 RL 在高维原始图像上从零学习视觉表征。图像统一 crop 并缩放至 128×128。All camera images (wrist camera + side camera) are uniformly passed through ResNet-10 (ImageNet pretrained) to extract embeddings, which are then concatenated with proprioceptive information. Using a pretrained backbone brings two benefits: (1) higher optimization stability; (2) better exploration efficiency — it avoids having RL learn visual representations from scratch on high-dimensional raw images. Images are uniformly cropped and resized to 128×128.

人机协作训练流程（Human-in-the-Loop）Human-in-the-Loop Training Procedure

训练过程中，人类操作员全程监督机器人执行，在策略陷入局部最优或无法恢复的状态时，通过 SpaceMouse 随时介入纠错。干预数据同时写入 demo buffer 与 RL buffer；策略在干预前后的过渡状态则仅写入 RL buffer。这一机制与 HG-DAgger 类似，但关键区别在于：纠错数据不仅用于行为克隆，还进入 RL 的 Q 函数优化，引导策略更高效地探索成功轨迹。Throughout training, a human operator supervises the robot's execution and, whenever the policy falls into a local optimum or an unrecoverable state, intervenes to correct it at any time via the SpaceMouse. Intervention data is written into both the demo buffer and the RL buffer, while the transition states before and after an intervention go only into the RL buffer. This mechanism resembles HG-DAgger, but with one key difference: the corrective data is not only used for behavior cloning but also enters the Q function optimization of RL, guiding the policy to explore successful trajectories more efficiently.

"A human can intervene at any time step t_i. When a human intervenes, their action a_itv is applied to the robot instead of the policy's action a_RL. We store the intervention data in both the demonstration and RL data buffers."

下游机器人控制器设计Downstream Robot Controller Design

针对精密接触任务（如 RAM 插入、SSD 安装），策略输出 6D Cartesian twist 目标送入阻抗控制器（impedance controller），兼顾精度与安全性，允许 RL 在探索阶段发出随机动作而不损坏硬件。针对动态任务（如翻蛋、抽 Jenga），动作空间改为直接输出末端执行器坐标系下的 feedforward wrench（近似于期望加速度）。夹爪控制通过独立训练的 Grasp Critic（DQN）实现离散控制。For precise contact-rich tasks (such as RAM insertion and SSD assembly), the policy outputs a 6D Cartesian twist target that is fed to an impedance controller, balancing precision and safety and allowing RL to emit random actions during exploration without damaging the hardware. For dynamic tasks (such as egg flipping and Jenga whipping), the action space instead directly outputs a feedforward wrench in the end-effector frame (approximately a desired acceleration). Gripper control is realized discretely through a separately trained Grasp Critic (DQN).

03 实验Experiments

实验覆盖 7 大类、13 项任务，包含单臂与双臂配置：精密装配（RAM 插入、SSD 安装、USB 插入）、动态操作（翻蛋、Jenga 抽取）、柔性物体（时序皮带）、多阶段任务（IKEA 家具、汽车仪表盘）。所有任务均在真实机器人上训练，无仿真预训练。The experiments cover 7 categories and 13 tasks with both single-arm and dual-arm configurations: precision assembly (RAM insertion, SSD assembly, USB insertion), dynamic manipulation (egg flipping, Jenga whipping), deformable objects (timing belt), and multi-stage tasks (IKEA furniture, car dashboard). All tasks are trained on real robots, without simulation pretraining.

主要结果：HIL-SERL vs. BC（行为克隆基线）Main results: HIL-SERL vs. BC (behavior cloning baseline)

所有任务均报告 100 次试验的成功率（IKEA 整体装配为 10 次）。BC 基线使用与 RL 等量的演示条数与干预次数通过 HG-DAgger 训练。All tasks report success rates over 100 trials (10 trials for the whole IKEA assembly). The BC baseline is trained with HG-DAgger using the same number of demonstrations and interventions as RL.

注：IKEA Side Panel 1 的 BC 成功率原文为 2%，HIL-SERL 结果为 77%（+30% 是相对 IKEA Side Panel 2 的语境；Side Panel 1 原文数据：BC=2, HIL-SERL=100 — 以上表格数据均来自 Table 1(a)，直接引用原文）。Note: for IKEA Side Panel 1 the BC success rate is 2% in the paper and the HIL-SERL result is 77% (the +30% belongs to the context of IKEA Side Panel 2; the paper's figures for Side Panel 1 are BC=2, HIL-SERL=100 — all table data above are taken from Table 1(a) and quoted directly from the paper).

与多种方法的比较（Table 1b）Comparison with multiple methods (Table 1b)

在 RAM Insertion、Car Dashboard Assembly、Object Flipping 三个代表性任务上，对比了 Diffusion Policy（DP，200 demos）、HG-DAgger、BC、IBRL、Residual RL、DAPG 以及 HIL-SERL 的两种消融变体。On the three representative tasks RAM Insertion, Car Dashboard Assembly and Object Flipping, the paper compares Diffusion Policy (DP, 200 demos), HG-DAgger, BC, IBRL, Residual RL, DAPG, and two ablated variants of HIL-SERL.

关键消融发现：无任何示范或纠错从零训练 RL 所有任务均为 0% 成功率；将 demo 增加到 200 条但无在线纠错（no itv），Dashboard Assembly 仍为 0%，证明在线纠错是必不可少的。Key ablation findings: training RL from scratch without any demonstrations or corrections yields a 0% success rate on all tasks; increasing the demos to 200 but without online corrections (no itv) still leaves Dashboard Assembly at 0%, proving that online corrections are indispensable.

鲁棒性评估Robustness Evaluation

04 局限性Limitations