Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
ALOHA 低成本双臂遥操作系统 + ACT 算法：仅用 10 分钟演示数据学会精细操作ALOHA low-cost bimanual teleoperation system + ACT algorithm: learning fine manipulation from only 10 minutes of demonstration data
Tony Z. Zhao · Vikash Kumar · Sergey Levine · Chelsea Finn | Stanford University · UC Berkeley · Meta
精细操作（如插电池、开调味品杯）历来需要昂贵的高精度机器人。本文提出 ALOHA——一套预算在 $20k 以内的开源双臂遥操作系统，以及配套的模仿学习算法 ACT（Action Chunking with Transformers）。ACT 以 Transformer + CVAE 为骨干，将未来 k 步动作作为一个"chunk"整体预测，有效缓解了复合误差（compounding error）和人类演示中的非 Markov 噪声，最终在真实世界 6 项精细操作任务上实现 80–90% 成功率。Fine manipulation (e.g. slotting a battery, opening a condiment cup) has traditionally required expensive high-precision robots. This paper presents ALOHA, an open-source bimanual teleoperation system with a budget under $20k, together with the matching imitation learning algorithm ACT (Action Chunking with Transformers). Built on a Transformer + CVAE backbone, ACT predicts the next k actions as a single "chunk", which effectively mitigates compounding error and the non-Markovian noise in human demonstrations, and finally reaches 80–90% success rates on 6 real-world fine manipulation tasks.
精细操作任务（opening a lid of a condiment cup、slotting a battery）需要毫米级精度、接触力的精细协调以及视觉闭环反馈，传统上依赖昂贵的高端机器人与精密传感器。本文的核心问题是：能否让低成本、本身不够精确的硬件，通过学习来完成精细操作？Fine manipulation tasks (opening a lid of a condiment cup, slotting a battery) demand millimeter-level precision, fine coordination of contact forces and closed-loop visual feedback, and have traditionally relied on expensive high-end robots and precision sensors. The core question of this paper is: can low-cost, inherently imprecise hardware learn to perform fine manipulation?
"Can learning enable low-cost and imprecise hardware to perform these fine manipulation tasks?"
低成本硬件的不精确性使感知与规划难度更高。人类虽同样没有工业级本体感觉（proprioception），却能通过学习和视觉闭环反馈完成精密任务。因此，作者设计了端到端像素到动作（pixel-to-action）的策略，并搭建了一套低成本但具备高度灵巧性的遥操作系统来采集高质量演示数据。The imprecision of low-cost hardware makes perception and planning harder. Humans likewise lack industrial-grade proprioception, yet accomplish precise tasks through learning and closed-loop visual feedback. The authors therefore design an end-to-end pixel-to-action policy and build a low-cost but highly dexterous teleoperation system to collect high-quality demonstration data.
图 1：ALOHA 系统概览。左：用户通过反向驱动（backdriving）小型 leader 机器人（WidowX）实现遥操作，follower 机器人（ViperX 6-DoF）镜像跟随。右：ALOHA 可完成穿扎带、开调味品杯、插 RAM 等高精度、接触丰富和动态任务。整套系统预算低于 $20k。Figure 1: overview of the ALOHA system. Left: the user teleoperates by backdriving a small leader robot (WidowX), while the follower robot (ViperX 6-DoF) mirrors it. Right: ALOHA can thread a cable tie, open a condiment cup, insert RAM and perform other high-precision, contact-rich and dynamic tasks. The whole system costs under $20k.
$20k整套系统总预算上限Budget cap of the whole system
80–90%真实任务成功率（6 项任务平均）Real-world success rate (average over 6 tasks)
10 min每个任务所需演示数据量Demonstration data needed per task
50 Hz控制与数据采集频率Control and data collection rate
02方法Method
本文提出两大核心贡献：（1）ALOHA 遥操作硬件系统，以及（2）ACT（Action Chunking with Transformers）模仿学习算法。两者协同作用，使低成本硬件能够从少量真实演示中学会精细双臂操作技能。The paper makes two core contributions: (1) the ALOHA teleoperation hardware system, and (2) the ACT (Action Chunking with Transformers) imitation learning algorithm. Working together, they let low-cost hardware learn fine bimanual manipulation skills from a small number of real-world demonstrations.
ALOHA：低成本双臂遥操作系统ALOHA: a low-cost bimanual teleoperation system
ALOHA 以两条 ViperX 6-DoF 机械臂（~$5,600/臂）作为 follower，两条 WidowX 机械臂（~$3,300/臂）作为 leader，采用 joint-space mapping 进行遥操作——用户反向驱动 leader，follower 实时镜像。相比 task-space（末端执行器）映射，joint-space 映射在奇点附近控制更稳定、延迟更低。系统配备 4 路 Logitech C922x 网络摄像头（分辨率 480×640，帧率 30fps），包括顶部、正面和两个腕部摄像头，以 50Hz 采集数据。ALOHA uses two ViperX 6-DoF arms (~$5,600 per arm) as followers and two WidowX arms (~$3,300 per arm) as leaders, and adopts joint-space mapping for teleoperation: the user backdrives the leader and the follower mirrors it in real time. Compared with task-space (end-effector) mapping, joint-space mapping gives more stable control near singularities and lower latency. The system is equipped with 4 Logitech C922x webcams (480×640 resolution, 30fps), including top, front and two wrist cameras, recording data at 50Hz.
ACT：Action Chunking with TransformersACT: Action Chunking with Transformers
针对模仿学习的两大难题——复合误差（compounding error）和人类演示的非 Markov 噪声，ACT 提出了以下设计：To address the two major difficulties of imitation learning — compounding error and the non-Markovian noise of human demonstrations — ACT introduces the following designs:
Action Chunking：策略每次预测未来 k 步的动作序列（而非单步动作），将任务的有效时间跨度压缩 k 倍，大幅缓解复合误差。Action Chunking: the policy predicts a sequence of the next k actions at a time (rather than a single-step action), compressing the effective horizon of the task by a factor of k and greatly mitigating compounding error.
Temporal Ensembling：每一时间步都查询策略，对多个重叠 action chunk 的预测结果按指数权重（wi = exp(−m · i)）加权平均，使轨迹更平滑，且无需额外训练代价。Temporal Ensembling: the policy is queried at every timestep, and the predictions of several overlapping action chunks are averaged with exponential weights (wi = exp(−m · i)), which makes trajectories smoother at no extra training cost.
CVAE 训练目标：将策略建模为 Conditional VAE，通过"style variable" z 捕捉人类演示中的多模态性（multimodality）和随机性。CVAE encoder 在测试时丢弃，z 设为先验均值（即零向量）以确定性解码。CVAE training objective: the policy is modeled as a Conditional VAE, using a "style variable" z to capture the multimodality and stochasticity of human demonstrations. The CVAE encoder is discarded at test time and z is set to the prior mean (the zero vector) for deterministic decoding.
具体实现：CVAE encoder 采用 BERT-style Transformer encoder，输入为 [CLS] token + 关节位置 + 目标动作序列（长度 k+2）；CVAE decoder（即策略）使用 ResNet18 图像编码器 + Transformer encoder-decoder，处理 4 路 480×640 RGB 图像及关节位置，输出 k×14 维动作序列（双臂绝对关节位置）。使用 L1 重建损失 + KL 散度正则化，共约 80M 参数，在单张 RTX 2080 Ti 上训练约 5 小时。Implementation details: the CVAE encoder is a BERT-style Transformer encoder whose input is a [CLS] token + joint positions + the target action sequence (length k+2); the CVAE decoder (i.e. the policy) uses a ResNet18 image encoder plus a Transformer encoder-decoder, processes 4 RGB images of 480×640 together with joint positions, and outputs a k×14 action sequence (absolute joint positions of both arms). It is trained with an L1 reconstruction loss plus KL divergence regularization, totals about 80M parameters, and takes about 5 hours on a single RTX 2080 Ti.
图 4：ACT 架构。左（训练时）：CVAE encoder 将动作序列与关节观测压缩为 style variable z 的均值和方差。右（训练 + 推断）：CVAE decoder（策略）融合多视角图像、关节位置和 z，通过 Transformer encoder 合成特征，再由 Transformer decoder 解码为 k 步动作序列。测试时 z=0（先验均值）。Figure 4: the ACT architecture. Left (training): the CVAE encoder compresses the action sequence and the joint observations into the mean and variance of the style variable z. Right (training + inference): the CVAE decoder (the policy) fuses multi-view images, joint positions and z, synthesizes features through a Transformer encoder, and then decodes a k-step action sequence with a Transformer decoder. At test time z=0 (the prior mean).
数据采集Data collection
每个真实任务收集 50 条演示（Thread Velcro 任务收集 100 条），每条演示耗时 8–14 秒（400–700 步@50Hz），总数据量约 10–20 分钟/任务。演示具有固有随机性——例如空中换手的位置每次都略有不同——这要求策略学习任务的本质规律而非死记演示。Each real-world task collects 50 demonstrations (100 for the Thread Velcro task); each demonstration takes 8–14 seconds (400–700 steps at 50Hz), for a total of roughly 10–20 minutes of data per task. The demonstrations carry inherent stochasticity — for example the mid-air handover position differs slightly every time — which forces the policy to learn the essence of the task rather than memorize the demonstrations.
03实验Experiments
实验覆盖 2 个 MuJoCo 仿真任务（Cube Transfer、Bimanual Insertion）和 6 个真实任务（Slide Ziploc、Slot Battery、Open Cup、Thread Velcro、Prep Tape、Put On Shoe），与 BC-ConvMLP、BeT、RT-1、VINN 四条基线对比。仿真任务报告 3 个随机种子、各 50 次评测的平均成功率；真实任务报告 1 个种子、25 次评测的成功率。The evaluation covers 2 MuJoCo simulation tasks (Cube Transfer, Bimanual Insertion) and 6 real-world tasks (Slide Ziploc, Slot Battery, Open Cup, Thread Velcro, Prep Tape, Put On Shoe), compared against four baselines: BC-ConvMLP, BeT, RT-1 and VINN. Simulation tasks report the average success rate over 3 random seeds with 50 evaluations each; real-world tasks report the success rate of 1 seed over 25 evaluations.
ACT 在每个任务上都以大幅优势超越所有基线。仿真任务中，ACT 较第二好方法的成功率领先幅度分别为 59%、49%、29% 和 20%。其他方法虽然能完成前 1–2 个子任务，但最终成功率均低于 30%。ACT surpasses all baselines by a wide margin on every task. In simulation, ACT leads the second-best method in success rate by 59%, 49%, 29% and 20% respectively. Other methods can finish the first 1–2 subtasks, but their final success rates all stay below 30%.
剩余真实任务（Table II，与 BeT 对比）Remaining real-world tasks (Table II, compared with BeT)
任务Task
BeT (最终成功率)BeT (final success rate)
ACT (最终成功率)ACT (final success rate)
Open Cup
0
84%
Thread Velcro
0
20%
Prep Tape
0
64%
Put On Shoe
0
92%
BeT 在上述 4 项高难度任务的最终成功率均为 0，而 ACT 展现出显著能力（Thread Velcro 相对较低是因为该任务需要在空中完成毫米级插入，视觉定位极为困难）。BeT reaches a final success rate of 0 on all 4 of these hard tasks, while ACT shows substantial capability (Thread Velcro is relatively low because the task requires a millimeter-level insertion in mid-air, which makes visual localization extremely difficult).
图 6：6 个真实世界任务的定义与子任务流程。每个任务的初始物体位置在 15cm 白色参考线范围内随机，子任务逐步递进，要求精细的双臂协调。Figure 6: definitions and subtask flows of the 6 real-world tasks. The initial object position of each task is randomized within a 15cm white reference line, and the subtasks advance step by step, demanding fine bimanual coordination.
消融实验Ablations
图 8：消融实验。(a) chunk size k 的影响：成功率从 k=1（无 chunking）的 1% 提升至 k=100 时的 44%，过大的 k（接近完全开环）性能略有下降。BC-ConvMLP 和 VINN 加入 chunking 后也显著提升，验证了 action chunking 的通用性。(b) Temporal Ensemble (TE) 对参数化方法（ACT +3.3%，BC-ConvMLP +4%）有益，对非参数化的 VINN 反而下降 20%。(c) CVAE 目标的必要性：在人类演示数据上，移除 CVAE 后成功率从 35.3% 骤降至 2%；而对确定性脚本演示影响甚微。(d) 用户研究：将遥操作频率从 50Hz 降至 5Hz 导致任务完成时间平均增加 62%（p<0.001）。Figure 8: ablations. (a) effect of chunk size k: the success rate rises from 1% at k=1 (no chunking) to 44% at k=100, while an overly large k (close to fully open-loop execution) degrades performance slightly. BC-ConvMLP and VINN also improve markedly once chunking is added, confirming the generality of action chunking. (b) Temporal Ensemble (TE) helps parametric methods (ACT +3.3%, BC-ConvMLP +4%) but costs the non-parametric VINN 20%. (c) necessity of the CVAE objective: on human demonstration data, removing the CVAE drops the success rate from 35.3% to 2%; on deterministic scripted demonstrations the effect is negligible. (d) user study: lowering the teleoperation rate from 50Hz to 5Hz increases task completion time by 62% on average (p<0.001).
04局限性Limitations
Note：以下限制部分由作者在论文 Appendix F 及 Section VII 中明确陈述（标注为 stated），部分为设计本身隐含的局限（标注为 inferred）。Note: some of the limitations below are explicitly stated by the authors in Appendix F and Section VII of the paper (marked stated), while others are implied by the design itself (marked inferred).
超出硬件与算法能力的任务 （stated）Tasks beyond the capability of the hardware and the algorithm (stated)
论文明确指出："there exist tasks that are beyond the capability of either the robots or the learning algorithm, such as buttoning up a dress shirt." 此类任务需要更复杂的手指自由度或更长程的规划，当前 parallel-jaw gripper 设计无法胜任。The paper states explicitly: "there exist tasks that are beyond the capability of either the robots or the learning algorithm, such as buttoning up a dress shirt." Such tasks need more finger degrees of freedom or longer-horizon planning, which the current parallel-jaw gripper design cannot deliver.
Thread Velcro 任务成功率较低 （stated）Low success rate on the Thread Velcro task (stated)
Thread Velcro 最终成功率仅 20%，每个子任务成功率约减半。主要失败模式：（1）右臂空中夹持过早，无法抓住扎带尾端；（2）插入阶段不够精准，错过 3mm×25mm 的塑料环。根本原因是黑色扎带与背景对比度低，仅占图像极小比例，视觉定位困难。The final success rate of Thread Velcro is only 20%, and the success rate roughly halves at each subtask. The main failure modes: (1) the right arm closes its gripper too early in mid-air and fails to grasp the tail of the cable tie; (2) the insertion stage is not precise enough and misses the 3mm×25mm plastic loop. The root cause is that the black cable tie has low contrast against the background and occupies only a tiny fraction of the image, making visual localization difficult.
每个任务独立训练，无任务泛化能力 （inferred）Each task is trained independently, with no cross-task generalization (inferred)
ACT 对每个任务从零训练独立策略（约 5 小时/任务），模型并不具备跨任务泛化或 few-shot 能力。对新任务需重新采集 50 条演示并重训，这限制了系统在实际部署中的灵活性。ACT trains an independent policy from scratch for each task (about 5 hours per task), and the model has no cross-task generalization or few-shot ability. A new task requires collecting 50 fresh demonstrations and retraining, which limits the flexibility of the system in real deployment.
演示量与场景多样性有限 （inferred）Limited number of demonstrations and scene diversity (inferred)
每个任务仅收集 50 条演示（Thread Velcro 为 100 条），物体的位置随机化仅限于 15cm 白色参考线范围内，背景和光照条件相对固定。在更复杂或更多变的真实场景中泛化能力尚未验证。Each task collects only 50 demonstrations (100 for Thread Velcro), object position randomization is confined to a 15cm white reference line, and background and lighting conditions stay relatively fixed. Generalization to more complex or more variable real-world scenes has not been verified.
单用户遥操作，缺乏多操作员鲁棒性 （inferred）Single-user teleoperation, no multi-operator robustness (inferred)
所有演示均由单一操作员采集，人类演示的风格和策略具有操作员依赖性。不同操作员的数据混合可能引入更大的多模态噪声，CVAE 是否足以建模此类噪声尚未验证。All demonstrations were collected by a single operator, and the style and strategy of human demonstrations are operator-dependent. Mixing data from different operators may introduce larger multimodal noise, and whether the CVAE suffices to model such noise has not been verified.
