PointACT：多尺度点云-动作交互的视觉-语言-动作模型PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interaction
PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interaction
Shizhe Chen, Paul Pacaud, Cordelia Schmid · Inria, École normale supérieure, CNRS, PSL Research University
PointACT 将点云（point cloud）三维几何表示深度融合进视觉-语言-动作模型（VLA）的动作解码阶段，通过分层 bottleneck 窗口自注意力机制实现多尺度的"点云特征 ↔ 动作 token"细粒度交互，在 RLBench 和 LIBERO 基准上超越了当前最优的 2D 与 3D 方法，并在真实机器人上验证了其有效性。PointACT deeply integrates 3D geometric representations from point clouds into the action-decoding stage of vision-language-action models (VLA), achieving fine-grained multi-scale interaction between point-cloud features and action tokens through a hierarchical bottleneck window self-attention mechanism. It surpasses state-of-the-art 2D and 3D methods on the RLBench and LIBERO benchmarks, and its effectiveness is validated on real robots.
当前最先进的 VLA 模型几乎全部依赖二维视觉表示，这严重限制了其对精细几何与空间关系的推理能力。State-of-the-art VLA models rely almost entirely on 2D visual representations, which severely limits their ability to reason about fine geometry and spatial relations.
"the physical world is inherently three-dimensional, while most state-of-the-art VLAs rely on 2D image representations."
现有的将三维信息引入 VLA 的尝试存在两类明显不足：Existing attempts to bring 3D information into VLAs suffer from two evident shortcomings:
Monolithic 融合：将点云 embedding 直接注入预训练 VLM，会破坏 VLM 在大规模数据上习得的表示，导致性能严重下滑（实验中 RLBench 仅 18.6%）。Monolithic fusion: injecting point-cloud embeddings directly into a pretrained VLM corrupts the representations the VLM has acquired from large-scale data, causing a severe performance drop (only 18.6% on RLBench in the experiments).
粗粒度全局特征：已有的点云增强 VLA 方法"primarily rely on coarse-grained global geometry features"，无法利用点云编码器的预训练优势，忽略了多尺度局部几何线索。Coarse-grained global features: existing point-cloud-augmented VLA methods "primarily rely on coarse-grained global geometry features", cannot exploit the pretraining advantage of point-cloud encoders, and ignore multi-scale local geometric cues.
图 1：三种将 3D 点云整合进 VLA 的策略对比：(a) Monolithic 融合将点云 token 直接注入 VLM 上下文；(b) Dual-system 仅使用 PTv3 最终层粗粒度特征；(c) PointACT 通过 Bottleneck Window Self-Attention 在解码各层与点云特征进行多尺度细粒度交互。Figure 1: Comparison of three strategies for integrating 3D point clouds into VLAs: (a) monolithic fusion injects point tokens directly into the VLM context; (b) dual-system uses only the coarse-grained features of the final PTv3 layer; (c) PointACT performs multi-scale fine-grained interaction with point-cloud features at every decoding layer through Bottleneck Window Self-Attention.
82.3%RLBench 10 任务平均成功率Mean success rate over 10 RLBench tasks
96.0%LIBERO 4 套件平均成功率Mean success rate over 4 LIBERO suites
+10%相比最优基线在 RLBench 上的提升Improvement over the best baseline on RLBench
300MPointACT 可训练参数量PointACT trainable parameters
02方法Method
PointACT 采用双系统架构（dual-system）：冻结的 VLM（Qwen2.5-VL）负责高层语义理解，可训练的 PointACT 动作专家（action expert）通过多尺度 Bottleneck 窗口自注意力机制将 Point Transformer v3（PTv3）各层的分层几何特征与动作 token 进行深度交互，最终预测机器人动作。PointACT adopts a dual-system architecture: a frozen VLM (Qwen2.5-VL) handles high-level semantic understanding, while a trainable PointACT action expert deeply interacts the hierarchical geometric features of each Point Transformer v3 (PTv3) layer with action tokens through multi-scale bottleneck window self-attention, ultimately predicting robot actions.
图 2：PointACT 整体架构（左）与 Bottleneck 窗口自注意力机制（右）。点云按空间分成 K 个窗口，动作 token 广播至各窗口内的点 token，进行自注意力计算后跨窗口平均聚合，再通过交叉注意力融合 VLM 特征。该过程在 PTv3 各分层阶段重复执行，实现多尺度交互。Figure 2: Overall PointACT architecture (left) and the bottleneck window self-attention mechanism (right). The point cloud is spatially partitioned into K windows; action tokens are broadcast to the point tokens inside each window, self-attention is computed, and the outputs are averaged across windows, after which VLM features are fused in by cross-attention. The process is repeated at each hierarchical stage of PTv3 to realize multi-scale interaction.
问题形式化Problem formulation
策略在多模态观测下预测未来 H 步动作：The policy predicts the next H actions from multimodal observations:
A_t = π_θ(I_t, P_t, s_t, L)
其中 I_t 为多视角 RGB 图像，P_t ∈ ℝ^(N_P×6) 为三维点云（XYZ + RGB），s_t 为本体感受状态，L 为自然语言指令。where I_t is the multi-view RGB image, P_t ∈ ℝ^(N_P×6) is the 3D point cloud (XYZ + RGB), s_t is the proprioceptive state, and L is the natural-language instruction.
将点云划分为 K 个空间窗口。对每个窗口 k，动作 token 广播拼接至该窗口内点 token，进行联合自注意力：The point cloud is divided into K spatial windows. For each window k, action tokens are broadcast and concatenated to the point tokens inside that window, and joint self-attention is applied:
动作 token 跨窗口平均聚合局部上下文：Action tokens aggregate local context by averaging across windows:
Ẑ_a^l = (1/K) Σ_k X̂_{k,a}^l
再通过交叉注意力融合 VLM 嵌入：VLM embeddings are then fused in by cross-attention:
Z̄_a^l = Cross-Attn(Ẑ_a^l, Z_vlm)
该流程在 PTv3 各分层阶段重复，从粗到细提取多尺度几何线索。关键设计：动作 token 充当"bottleneck"，既能感知局部几何细节，又保持全局语义一致，避免了将大量点云 token 直接暴露给 VLM 带来的干扰。This procedure is repeated at each hierarchical stage of PTv3, extracting multi-scale geometric cues from coarse to fine. Key design: action tokens act as a "bottleneck", perceiving local geometric detail while maintaining global semantic consistency, which avoids the interference caused by exposing a large number of point tokens directly to the VLM.
动作预测头Action prediction heads
回归头（Regression Head）Regression Head
用于 LIBERO 等连续控制任务，动作 chunk 大小 H=16，使用 L₂ 损失：Used for continuous-control tasks such as LIBERO, with action chunk size H=16 and an L₂ loss:
L_reg = (1/H) Σ ‖a_i − a_i*‖₂²
分类头（Classification Head）Classification Head
用于 RLBench 关键帧预测，工作空间离散化为空间 bin，使用交叉熵损失：Used for RLBench keyframe prediction, with the workspace discretized into spatial bins and a cross-entropy loss:
L_cls = −Σ_k Σ_b y_{k,b} log(ŷ_{k,b})
图 3：PointACT VLA 完整架构图。冻结的 Qwen2.5-VL 处理图像与语言，输出 VLM 特征；Point Transformer v3-Large（PTv3）对点云提取分层特征；PointACT 模块在解码各阶段通过窗口注意力将两路特征与动作 token 深度融合，最终由动作头输出机器人控制指令。Figure 3: Full architecture of the PointACT VLA. The frozen Qwen2.5-VL processes images and language and outputs VLM features; Point Transformer v3-Large (PTv3) extracts hierarchical features from the point cloud; the PointACT module deeply fuses the two feature streams with action tokens through window attention at each decoding stage, and the action head finally outputs robot control commands.
点云编码器：Point Transformer v3-Large（PTv3-Large），在建筑级场景数据上预训练Point-cloud encoder: Point Transformer v3-Large (PTv3-Large), pretrained on building-scale scene data
可训练参数：约 300M（PointACT 模块）Trainable parameters: about 300M (PointACT module)
点云预处理：工作空间 bounding box 裁剪，1cm 体素化，最多 4096 个点Point-cloud preprocessing: workspace bounding-box cropping, 1cm voxelization, at most 4096 points
03实验Experiments
在 RLBench（10 任务）、LIBERO（4 套件）仿真基准及真实机器人平台（SO-100 和 UR5）上与当前最优方法（包括 π₀、GR00T-N1.5、EO1、ACT3D、3DLotus 等）进行全面对比。A comprehensive comparison against state-of-the-art methods (including π₀, GR00T-N1.5, EO1, ACT3D and 3DLotus) on the RLBench (10 tasks) and LIBERO (4 suites) simulation benchmarks and on real robot platforms (SO-100 and UR5).
图 4：RLBench 10 个操作任务与 LIBERO 4 个任务套件示意图。RLBench 使用关键帧预测 + RRT 运动规划；LIBERO 使用 delta 末端执行器控制，动作 chunk 大小 H=16。Figure 4: The 10 RLBench manipulation tasks and the 4 LIBERO task suites. RLBench uses keyframe prediction plus RRT motion planning; LIBERO uses delta end-effector control with action chunk size H=16.
仿真基准结果Simulation benchmark results
方法Method
LIBERO-Spatial
LIBERO-Object
LIBERO-Goal
LIBERO-Long
LIBERO 均值LIBERO mean
RLBench 均值RLBench mean
OpenVLA
84.7
88.4
79.2
53.7
—
41.0
π₀
—
—
—
—
—
55.0
ACT3D
—
—
—
—
—
64.5
GR00T(arch) + Point
92.0
—
—
—
—
69.7
EO1（复现）EO1 (reproduced)
91.8
—
—
85.6
93.1
73.2
PointACT（本文）PointACT (ours)
97.4
—
—
90.6
96.0
82.3
RLBench 各任务成功率（部分）Per-task success rates on RLBench (excerpt)
任务Task
EO1
PointACT
Phone on base
—
99
Umbrella out
—
99
Wine at rack
—
90
Sweep to dustpan
—
59
Water plants
—
40
均值Mean
73.2
82.3
消融实验Ablations
图 5：PointACT 不同规模变体（Small: 59M, Base: 167M, Large: 314M）的性能曲线，以及是否使用 PTv3 预训练权重的对比。更大的模型从预训练中获益更多，表明优化难度随容量增加。Figure 5: Performance curves of PointACT variants at different scales (Small: 59M, Base: 167M, Large: 314M), together with a comparison of using versus not using PTv3 pretrained weights. Larger models benefit more from pretraining, indicating that optimization difficulty grows with capacity.
消融配置Ablation setting
LIBERO-Spatial (%)
RLBench (%)
EO1 baseline
91.8
73.2
EO1 + Point（Monolithic）EO1 + Point (monolithic)
94.0
18.6
GR00T(arch) baseline
87.0
50.8
GR00T(arch) + Point（Dual-system 粗粒度）GR00T(arch) + Point (dual-system, coarse-grained)
92.0
69.7
多尺度直接拼接（K=64, 128 tokens）Multi-scale direct concatenation (K=64, 128 tokens)
Monolithic 融合在 RLBench 上成功率暴跌至 18.6%，表明将点云直接注入预训练 VLM 会严重干扰其已学到的表示。Monolithic fusion collapses to 18.6% success on RLBench, showing that injecting point clouds directly into a pretrained VLM severely disrupts its learned representations.
多尺度直接拼接（不使用窗口注意力）仅达 65.2–65.6%，证明"细粒度交互"是提升的关键，而非单纯引入多尺度特征。Multi-scale direct concatenation (without window attention) reaches only 65.2–65.6%, proving that fine-grained interaction is the key to the gain, rather than merely introducing multi-scale features.
即使移除图像条件，PointACT 也能达 79.8%（RLBench），但加入 2D 图像后进一步提升至 82.3%，说明 2D 语义特征与 3D 几何特征互补。Even with image conditioning removed, PointACT still reaches 79.8% (RLBench), while adding 2D images pushes it further to 82.3%, indicating that 2D semantic features and 3D geometric features are complementary.
真实机器人实验Real-robot experiments
图 6：真实机器人平台设置。左：SO-100（6-DoF 3D 打印机械臂，Intel RealSense D435）；右：UR5（6-DoF UR5 + RG6 夹爪，Orbbec Femto Mega RGB-D 相机）。Figure 6: Real robot platform setups. Left: SO-100 (6-DoF 3D-printed arm, Intel RealSense D435); right: UR5 (6-DoF UR5 + RG6 gripper, Orbbec Femto Mega RGB-D camera).
任务Task
π₀
GR00T-N1.5
PointACT
SO-100 机械臂（每任务 10 次试验）SO-100 arm (10 trials per task)
Put Banana In Plate
10/10
8/10
10/10
Put Sock In Drawer
2/10
5/10
9/10
Open Microwave
7/10
5/10
8/10
任务Task
π₀
GR00T-N1.5
3DLotus
PointACT
UR5 机械臂（每任务 10 次试验）UR5 arm (10 trials per task)
Stack Yellow Cup
0/10
0/10
7/10
7/10
Close Drawer
9/10
9/10
2/10
7/10
Put Fruit in Plates
0/10
0/10
0/10
4/10
PointACT 在"Put Sock In Drawer"任务上将成功率从 π₀ 的 2/10 提升至 9/10，充分体现了精细三维感知对复杂操作任务的价值。在 UR5 实验中，PointACT 在杯子堆叠上与 3DLotus（纯几何方法）持平，但在关抽屉任务上大幅优于 3DLotus（7/10 vs 2/10），体现了 2D 语义与 3D 几何混合的鲁棒性。On the "Put Sock In Drawer" task PointACT raises the success rate from 2/10 for π₀ to 9/10, fully demonstrating the value of fine-grained 3D perception for complex manipulation tasks. In the UR5 experiments PointACT matches 3DLotus (a purely geometric method) on cup stacking, yet substantially outperforms it on closing the drawer (7/10 vs 2/10), reflecting the robustness of mixing 2D semantics with 3D geometry.
04局限性Limitations
说明：论文在失败案例分析（Failure Analysis）和结论部分明确指出了以下局限性，已标注为作者陈述（stated）；设计层面的推断局限另行注明（推断/inferred）。Note: The paper explicitly states the following limitations in its failure analysis and conclusion; these are marked as stated by the authors, while limitations inferred at the design level are marked separately (inferred).
感知遮挡导致的空间推理失败（stated）Spatial-reasoning failures caused by perceptual occlusion (stated)
部分视角的局限性限制了对空间关系的完整理解，尤其在物体被遮挡时模型难以准确估计目标位置。作者指出此类失败需要多视角图像集成（multi-view integration）来解决。对应案例：Water plants（40% 成功率）任务中浇水壶与植物的位置关系判断困难。The limited viewpoints restrict a complete understanding of spatial relations; when objects are occluded the model struggles to estimate target positions accurately. The authors note that such failures require multi-view integration to be resolved. Corresponding case: judging the relative position of the watering can and the plant in Water plants (40% success rate).
缺乏失败恢复能力（stated）Lack of failure recovery (stated)
"Models lack reactive recovery from execution errors or perturbations."——模型无法从执行错误或外界扰动中主动恢复，一旦中间动作出现偏差，后续动作难以纠正。作者将改进失败恢复能力列为未来工作方向。"Models lack reactive recovery from execution errors or perturbations." — the model cannot actively recover from execution errors or external perturbations; once an intermediate action deviates, subsequent actions are hard to correct. The authors list improving failure recovery as a direction for future work.
当操作需要通过工具间接施力时（如扫把→簸箕的接触几何），模型难以建模精确的接触力学，对应 Sweep to dustpan（59% 成功率）任务表现偏弱。When manipulation requires applying force indirectly through a tool (such as the contact geometry from broom to dustpan), the model struggles to model precise contact mechanics, corresponding to the weak performance on Sweep to dustpan (59% success rate).
点云噪声鲁棒性（推断/inferred）Robustness to point-cloud noise (inferred)
作者在结论中提到"improving robustness under noisy point observations"为未来工作方向，暗示当前模型对传感器噪声（如透明/反光物体的深度缺失）较为敏感。UR5 实验中关闭透明抽屉的案例（Close Drawer）也间接体现了这一挑战，但通过 2D 视觉特征部分弥补了点云缺陷。In the conclusion the authors mention "improving robustness under noisy point observations" as a direction for future work, implying that the current model is sensitive to sensor noise (such as missing depth on transparent or reflective objects). The case of closing a transparent drawer in the UR5 experiments (Close Drawer) also indirectly reflects this challenge, although 2D visual features partly compensate for the point-cloud deficiency.
点云预训练数据与机器人场景不匹配（stated + inferred）Mismatch between point-cloud pretraining data and robot scenes (stated + inferred)
当前使用的 PTv3 预训练权重来自建筑级场景数据，与桌面操作任务存在显著域差异（domain gap）。作者明确将"scaling point-based pretraining on robot datasets"列为重要的未来工作，以提升特征迁移效果，尤其对大容量模型优化帮助更大。The PTv3 pretrained weights currently used come from building-scale scene data, which exhibits a significant domain gap from tabletop manipulation tasks. The authors explicitly list "scaling point-based pretraining on robot datasets" as important future work for improving feature transfer, which helps the optimization of high-capacity models in particular.
