Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

01 动机 MotivationMotivation

机器人操作任务需要对多样化的物体、空间位置和语言指令进行泛化——而现有方法要么依赖目标物体的显式检测与位姿估计，要么只能处理 2D 图像输入，难以有效利用 3D 空间结构先验。如何用单一模型、在极少演示下学会多种 6-DoF 操作任务？Robotic manipulation tasks require generalization across diverse objects, spatial locations and language instructions, yet existing methods either rely on explicit detection and pose estimation of the target object, or can only handle 2D image input, making it hard to exploit 3D spatial structure priors. How can a single model learn a variety of 6-DoF manipulation tasks from very few demonstrations?

"Rather than explicitly detecting or tracking objects, we want to learn perceptual representations of actions conditioned on language goals."

PerAct 任务多样性展示 — 图 1：PerAct 在 RLBench 18 个操作任务（249 种变体）上的展示，涵盖抓取、推拉、开关门抽屉、堆叠、倒液体等多种交互类型。图中展示了语言指令（如 "open the middle drawer"、"put the green block in the green bowl"）与对应的操作场景。Figure 1: PerAct on 18 RLBench manipulation tasks (249 variations), covering interaction types such as grasping, pushing and pulling, opening and closing doors and drawers, stacking, and pouring liquid. The figure pairs language instructions (e.g. "open the middle drawer", "put the green block in the green bowl") with the corresponding manipulation scenes.

18RLBench 仿真任务RLBench simulated tasks

249任务变体数量Task variations

34×vs Image-BC 基线提升vs Image-BC baseline gain

2.8×vs C2FARM-BC 提升vs C2FARM-BC gain

核心问题Core Problem

传统机器人操作方法存在以下主要挑战：Conventional robotic manipulation methods face the following main challenges:

图像平面局限性：Image-BC（CNN/ViT）等方法将 RGB 图像直接映射到动作，缺乏 3D 空间推理能力，在精细 6-DoF 操作任务上表现差。Image-plane limitations: Methods such as Image-BC (CNN/ViT) map RGB images directly to actions, lack 3D spatial reasoning, and perform poorly on fine-grained 6-DoF manipulation tasks.
特定任务设计：许多方法依赖目标检测、位姿估计等中间表示，无法在任务间共享泛化能力。Task-specific design: Many methods rely on intermediate representations such as object detection and pose estimation, and cannot share generalization across tasks.
Transformer 对高维输入的处理瓶颈：标准 Transformer 的 self-attention 在百万维度体素输入上计算量为 O(N²)，不可行。Transformer bottleneck on high-dimensional input: The self-attention of a standard Transformer costs O(N²) on million-dimensional voxel input, which is infeasible.
演示数据稀缺：收集大量机器人演示数据代价高昂，需要能从极少演示（每任务 10–100 条）中学习的方法。Scarcity of demonstration data: Collecting large amounts of robot demonstration data is expensive, so methods must be able to learn from very few demonstrations (10-100 per task).

02 方法 MethodMethod

PerAct 将操作任务转化为"体素动作检测"问题：给定多视角 RGB-D 图像与语言指令，在离散化的 3D 体素空间中预测末端执行器的下一个最优位置与方向，输出包含平移、旋转、开合状态和碰撞规避在内的完整 6-DoF 动作。PerAct casts manipulation as a "voxel action detection" problem: given multi-view RGB-D images and a language instruction, it predicts the next best position and orientation of the end-effector in a discretized 3D voxel space, producing a complete 6-DoF action that covers translation, rotation, gripper open/close state and collision avoidance.

图 2：PerAct 整体架构。左侧：4 台 RGB-D 相机的点云融合为 100×100×100 体素网格，每个体素携带 RGB、占据和法向量特征；底部：语言指令通过预训练语言编码器（CLIP）转化为 token 序列。中部：Voxel Encoder 将体素序列与语言 token 拼接，输入 PerceiverIO Transformer，通过 cross-attention 压缩至 2048×512 latent 向量，再经多轮 self-attention 深度处理。右侧：Voxel Decoder 通过 cross-attention 输出每个体素的动作得分，Maxpool 后分别预测平移（Q_trans）、旋转（Q_rot）、开合（Q_open）和碰撞（Q_collide）。Figure 2: Overall PerAct architecture. Left: point clouds from 4 RGB-D cameras are fused into a 100x100x100 voxel grid, each voxel carrying RGB, occupancy and normal features; bottom: the language instruction is turned into a token sequence by a pretrained language encoder (CLIP). Middle: the Voxel Encoder concatenates the voxel sequence with the language tokens and feeds them into a PerceiverIO Transformer, which compresses them into a 2048x512 latent array via cross-attention and then processes them with several rounds of self-attention. Right: the Voxel Decoder outputs a per-voxel action score via cross-attention and, after maxpooling, predicts translation (Q_trans), rotation (Q_rot), gripper open/close (Q_open) and collision (Q_collide).

体素化观测表示（Voxelized Observation）Voxelized Observation

来自 4 台 RGB-D 相机的点云被融合并离散化为 100×100×100 的体素网格（voxel grid），空间分辨率约 1cm/voxel。每个体素存储 RGB 颜色、占据状态（0/1）与法向量，构成结构化的 3D 场景表示。体素按 Vision Transformer 方式切分为 patch，展平后与位置编码（positional embedding）相加，输入 PerceiverIO Transformer。Point clouds from 4 RGB-D cameras are fused and discretized into a 100x100x100 voxel grid, at a spatial resolution of about 1cm per voxel. Each voxel stores RGB color, occupancy state (0/1) and a normal vector, forming a structured 3D scene representation. The voxels are split into patches in the manner of a Vision Transformer, flattened, added to positional embeddings, and fed into the PerceiverIO Transformer.

图 3：PerceiverIO Transformer 结构。输入体素 token（100万+维度）先通过 cross-attention 压缩至小型 latent 数组（2048×512），随后多轮 self-attention 在 latent 空间高效处理，最后再次通过 cross-attention 解码输出体素级别的动作分数。该设计将 self-attention 的计算复杂度从 O(N²) 降至 O(NM)（N≫M）。Figure 3: Structure of the PerceiverIO Transformer. The input voxel tokens (1000000+ dimensions) are first compressed by cross-attention into a small latent array (2048x512); several rounds of self-attention then process them efficiently in latent space, and a final cross-attention decodes per-voxel action scores. This design lowers the computational complexity of self-attention from O(N²) to O(NM), with N far larger than M.

语言条件化（Language Conditioning）Language Conditioning

自然语言指令（如 "open the middle drawer"）通过预训练的 CLIP 语言编码器转化为 token 序列，与体素 token 序列拼接后共同输入 Perceiver。此设计允许 agent 通过语言指令区分任务变体（如颜色、位置、类别等），实现对未见过的物体实例的零样本泛化。语言特征与视觉特征在同一 latent 空间中通过 cross-attention 融合，无需额外的跨模态对齐模块。A natural-language instruction (e.g. "open the middle drawer") is converted into a token sequence by a pretrained CLIP language encoder and concatenated with the voxel token sequence before being fed into the Perceiver. This design lets the agent distinguish task variations (color, position, category, and so on) through the instruction, achieving zero-shot generalization to unseen object instances. Language and visual features are fused by cross-attention in a shared latent space, with no extra cross-modal alignment module.

动作输出（Action Representation）Action Representation

PerAct 将 6-DoF 末端执行器动作分解为：PerAct decomposes the 6-DoF end-effector action into:

平移（Q_trans）：在体素网格上预测最优体素位置（100×100×100 = 100 万类分类）；Translation (Q_trans): predicting the best voxel position on the voxel grid (100x100x100 = a 1000000-way classification);
旋转（Q_rot）：俯仰/横滚/偏航三个轴各独立预测，每轴离散化为 5° 精度（72 类）；Rotation (Q_rot): pitch / roll / yaw are predicted independently, each axis discretized at 5-degree resolution (72 classes);
夹爪开合（Q_open）：二元分类；Gripper open/close (Q_open): binary classification;
碰撞规避（Q_collide）：二元分类，控制是否规避碰撞。Collision avoidance (Q_collide): binary classification, controlling whether collisions are avoided.

训练目标为所有输出的交叉熵损失之和：L = L_trans + L_rot_x + L_rot_y + L_rot_z + L_open + L_collide。The training objective is the sum of the cross-entropy losses of all outputs: L = L_trans + L_rot_x + L_rot_y + L_rot_z + L_open + L_collide.

训练细节Training Details

使用 keyframe-based 的 behavior cloning：每条演示轨迹提取关键帧（速度接近零且夹爪状态改变处），以此作为监督信号。数据增强包括对体素网格施加随机颜色 jitter 和随机平移。模型在单台 GPU 上训练，不同任务共享一个多任务模型（single multi-task agent），每次输入的语言指令决定当前任务。Keyframe-based behavior cloning is used: keyframes are extracted from every demonstration trajectory (where the velocity is near zero and the gripper state changes) and serve as the supervision signal. Data augmentation applies random color jitter and random translation to the voxel grid. The model is trained on a single GPU, different tasks share one multi-task model (a single multi-task agent), and the language instruction fed in each time determines the current task.

03 实验 ExperimentsExperiments

实验在 RLBench 仿真环境（CoppeliaSim + PyRep）中进行，使用 4 台 RGB-D 相机；并在真实 Franka Panda 机械臂上进行了 7 个现实任务的验证。评估指标为每个任务的成功率（success rate），以 100 次评估为准（仿真），每次最多 25 步。Experiments are run in the RLBench simulator (CoppeliaSim + PyRep) with 4 RGB-D cameras, plus validation on a real Franka Panda arm across 7 real-world tasks. The evaluation metric is the per-task success rate, measured over 100 evaluations in simulation with at most 25 steps each.

基线方法Baselines

Image-BC (CNN)：以多视角 RGB 图像直接预测动作，卷积神经网络主干；Image-BC (CNN): predicts actions directly from multi-view RGB images, with a convolutional network backbone;
Image-BC (ViT)：同上，使用 Vision Transformer 主干；Image-BC (ViT): the same, with a Vision Transformer backbone;
C2FARM-BC：基于 3D 卷积网络的 coarse-to-fine action regression 方法，当时 state-of-the-art 的 3D 基线；C2FARM-BC: a coarse-to-fine action regression method built on 3D convolutional networks, the state-of-the-art 3D baseline at the time;
PerAct (ours)：本文方法。PerAct (ours): the method of this paper.

PerAct 在 RLBench 18 个任务上的多任务性能对比 — 图 4（Table 1）：RLBench 18 个任务的多任务成功率（%）对比。每个任务以 10 或 100 个 demonstrations 训练，25 次评估取平均（以 episodes 为单位）。PerAct 在绝大多数任务中显著领先，整体平均成功率从 C2FARM-BC 的约 26% 提升至约 49%（100 demos）。Image-BC (CNN) 在多数任务成功率接近 0，C2FARM-BC 是最具竞争力的基线。Figure 4 (Table 1): multi-task success rates (%) on the 18 RLBench tasks. Each task is trained with 10 or 100 demonstrations and averaged over 25 evaluations (counted in episodes). PerAct leads clearly on the large majority of tasks, raising the overall average success rate from about 26% for C2FARM-BC to about 49% (100 demos). Image-BC (CNN) is close to 0 on most tasks, and C2FARM-BC is the most competitive baseline.

方法Method	10 Demos 平均成功率Avg. success rate, 10 demos	100 Demos 平均成功率Avg. success rate, 100 demos	相对 C2FARM-BC 提升Gain over C2FARM-BC
Image-BC (CNN)	~1%	~1%	—
Image-BC (ViT)	~1%	~4%	—
C2FARM-BC	~20%	~26%	基线baseline
PerAct (ours)	~29%	~49%	+2.83× (100 demos)

关键发现Key Findings

PerAct 在 10 个 demonstrations 时平均成功率约 29%，100 个 demonstrations 提升至约 49%，使用更多 demonstrations 时还有进一步提升空间（"likely due to insufficient capacity"，论文原文）。With 10 demonstrations PerAct reaches about 29% average success rate, rising to about 49% with 100 demonstrations, and more demonstrations still leave room for further gains ("likely due to insufficient capacity", in the words of the paper).
对于 stack blocks（3 种变体）、insert peg、place cups 等高精度任务，PerAct 超越所有基线但成功率仍低于 65%，说明这些任务对末端执行器精度要求极高（"a few centimeters or degrees could lead to unrecoverable failures"）。On high-precision tasks such as stack blocks (3 variations), insert peg and place cups, PerAct surpasses all baselines but its success rate still stays below 65%, indicating that these tasks demand extremely high end-effector precision ("a few centimeters or degrees could lead to unrecoverable failures").
PerAct 展现出对语言描述的语义理解：可以泛化到训练中未见过的语言指令变体（如颜色词、位置描述），即 emergent language grounding。PerAct exhibits semantic understanding of language descriptions: it generalizes to instruction variations unseen during training (such as color words and position descriptions), i.e. emergent language grounding.

消融实验（Ablations）Ablations

论文（Appendix G）分析了以下因素：The paper (Appendix G) analyzes the following factors:

体素化分辨率（Voxelization resolution）：100³ 体素（约 1cm 精度）对多数任务足够，高精度任务（如 insert peg）受益于更高分辨率。Voxelization resolution: 100³ voxels (about 1cm precision) are enough for most tasks, while high-precision tasks such as insert peg benefit from a finer resolution.
全局 vs. 局部感受野（Global vs. Local Receptive Fields）：PerceiverIO 的全局感受野显著优于仅使用局部感受野的变体，说明 long-range attention 对多步规划至关重要。Global vs. Local Receptive Fields: the global receptive field of PerceiverIO is markedly better than variants restricted to local receptive fields, showing that long-range attention is crucial for multi-step planning.
Skip connection：去除 Voxel Encoder→Decoder 的 skip connection 后性能下降，说明细粒度空间特征对精确体素预测有帮助。Skip connection: removing the Voxel Encoder→Decoder skip connection degrades performance, showing that fine-grained spatial features help precise voxel prediction.
Latent 向量维度：2048 latent 向量优于更小的 latent 维度，提供足够的容量来编码多任务信息。Latent array size: 2048 latent vectors outperform smaller latent dimensions, providing enough capacity to encode multi-task information.

真实机器人实验Real-Robot Experiments

在 Franka Panda 机械臂上验证了 7 个真实任务（18 种变体），每任务仅使用 5–10 个人工演示，成功率优于从头训练的 C2FARM-BC，且无需任何 domain adaptation，展示了 3D voxel 表示在 sim-to-real 方面的优势。On a Franka Panda arm, 7 real-world tasks (18 variations) were validated with only 5-10 human demonstrations per task; the success rate beats a C2FARM-BC trained from scratch, without any domain adaptation, demonstrating the advantage of the 3D voxel representation for sim-to-real.

04 局限性 LimitationsLimitations

Note：以下局限性部分来自论文正文的明确陈述（标注 stated），部分为从方法设计推断（标注 inferred）。Note: Some of the limitations below are stated explicitly in the paper (marked stated), others are inferred from the method design (marked inferred).

静态 keyframe 动作表示的局限性（stated）Limitations of the static keyframe action representation (stated)

PerAct 使用 keyframe-based behavior cloning，仅在关键帧处预测动作，默认采用线性插值运动规划器（motion planner）连接关键帧。论文明确指出："our observations and actions are colorless and motionless"，不建模非关键帧的连续轨迹，对高度动态或时序依赖的操作任务（如抓取移动物体）不适用。PerAct uses keyframe-based behavior cloning, predicting actions only at keyframes and connecting them by default with a linear-interpolation motion planner. The paper states explicitly: "our observations and actions are colorless and motionless"; continuous trajectories between keyframes are not modeled, so the approach does not apply to highly dynamic or temporally dependent manipulation tasks such as grasping a moving object.

计算量与内存开销（inferred）Compute and memory cost (inferred)

100³ 体素网格在每步推理时需处理约 100 万个体素 token，即便 PerceiverIO 通过 latent 压缩将复杂度降至 O(NM)，仍需显著的计算资源。论文未报告单步推理时延，在实时机器人控制场景下可能存在瓶颈。A 100³ voxel grid requires processing about 1000000 voxel tokens at every inference step; even though PerceiverIO reduces the complexity to O(NM) through latent compression, substantial compute resources are still needed. The paper reports no single-step inference latency, which may be a bottleneck for real-time robot control.

演示数量上限与容量瓶颈（stated）Demonstration ceiling and capacity bottleneck (stated)

论文实验显示 PerAct 在增加演示（从 10 到 100）时持续涨点，但同时指出额外训练 demonstrations 带来的提升"likely due to insufficient capacity"，说明当前模型容量可能仍是瓶颈，在演示数量更大时需要更大的模型或更高效的训练策略。The experiments show PerAct improving steadily as demonstrations increase (from 10 to 100), but the paper also notes that the gains from additional training demonstrations are "likely due to insufficient capacity", suggesting that the current model capacity may still be the bottleneck and that larger models or more efficient training strategies will be needed at larger demonstration counts.

依赖结构化场景与标定相机（inferred）Reliance on structured scenes and calibrated cameras (inferred)

方法依赖多台标定好的 RGB-D 相机重建 3D 体素场景，对相机标定精度和场景光照条件敏感。在无结构化环境（如非实验室场景）或单相机设置下，3D 重建质量下降会直接影响体素空间中的动作预测精度。The method relies on several calibrated RGB-D cameras to reconstruct the 3D voxel scene, and is sensitive to calibration accuracy and scene lighting. In unstructured environments (outside the lab) or with a single-camera setup, degraded 3D reconstruction directly hurts the accuracy of action prediction in voxel space.

高精度任务成功率仍有差距（stated）Success rates on high-precision tasks still lag (stated)

对于 insert peg、place cups、stack blocks 等高精度任务，PerAct 成功率仍显著低于 65%，论文明确指出"a few centimeters or degrees could lead to unrecoverable failures"，这类任务对当前 ~1cm 体素分辨率和 5° 旋转精度仍构成挑战。On high-precision tasks such as insert peg, place cups and stack blocks, PerAct success rates remain markedly below 65%; the paper states explicitly that "a few centimeters or degrees could lead to unrecoverable failures", and such tasks remain a challenge for the current ~1cm voxel resolution and 5-degree rotation precision.