Exploring Spatial Representations for Visual-Language-Action Model
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, Xuelong Li
现有 VLA 模型缺乏显式的三维空间理解，导致在精细操作任务上泛化能力不足。SpatialVLA 提出 Ego3D Position Encoding 将深度信息融入视觉 token，并用 Adaptive Action Grids 将连续动作离散化为跨机器人可迁移的空间 token；在 1.1M 真实机器人数据上预训练后，实现强零样本迁移与高效微调。Existing VLA models lack explicit 3D spatial understanding, which limits their generalization on fine-grained manipulation tasks. SpatialVLA proposes Ego3D Position Encoding to fuse depth information into visual tokens, and Adaptive Action Grids to discretize continuous actions into spatial tokens transferable across robots; after pre-training on 1.1M real-robot data, it achieves strong zero-shot transfer and efficient fine-tuning.
Robotics: Science and Systems 2025预训练数据 1.1M episodesPre-training data 1.1M episodes骨干：SigLIP + Qwen2Backbone: SigLIP + Qwen2📄 arXiv:2501.15830PDF
VLA机器人操作robot manipulationEgo3D Position EncodingAdaptive Action Grids空间理解spatial understandingzero-shot generalizationgeneralist policyaction tokenization
01动机Motivation
机器人操作本质上是一个三维空间感知与动作规划问题，然而主流 VLA 模型（OpenVLA、Octo、RT-2-X 等）仅从 2D 图像 token 中学习动作，缺乏对物体位置、深度与空间布局的显式建模，在需要精细空间推理的任务（如堆叠、精确放置）上表现明显弱于专用方法。Robot manipulation is in essence a problem of 3D spatial perception and action planning, yet mainstream VLA models (OpenVLA, Octo, RT-2-X, etc.) learn actions solely from 2D image tokens and lack explicit modeling of object position, depth and spatial layout; on tasks that demand fine-grained spatial reasoning (e.g. stacking, precise placement) they perform markedly worse than specialized methods.
"Spatial understanding is the key to robot manipulation … we propose SpatialVLA, a spatial visual-language-action model that focuses on exploring spatial representations for robot manipulation."
图1：SpatialVLA 总览。 给定图像观测 ot 与任务指令 L，模型通过 Ego3D Position Encoding 处理图像，自回归预测空间动作 token，再反 tokenize 为连续动作 At 执行控制。Figure 1: Overview of SpatialVLA. Given an image observation ot and a task instruction L, the model processes the image with Ego3D Position Encoding, autoregressively predicts spatial action tokens, and de-tokenizes them back into continuous actions At for control.
81.0%SimplerEnv Google Robot 零样本 Visual MatchingSimplerEnv Google Robot zero-shot Visual Matching
34.4%SimplerEnv WidowX 零样本平均成功率SimplerEnv WidowX zero-shot average success rate
21 Hz推理频率（实时控制）Inference frequency (real-time control)
02 方法
SpatialVLA 由两个核心模块构成：（1）Ego3D Position Encoding——将深度估计得到的三维坐标编码叠加到 SigLIP 视觉 token 上；（2）Adaptive Action Grids——根据训练集动作分布自适应离散化连续 7D 动作，并支持跨机器人迁移。SpatialVLA consists of two core modules: (1) Ego3D Position Encoding — the 3D coordinates obtained from depth estimation are encoded and added onto the SigLIP visual tokens; (2) Adaptive Action Grids — the continuous 7D action is adaptively discretized according to the action distribution of the training set, with support for cross-robot transfer.
Ego3D Position Encoding
给定 RGB 图像，首先用 ZoeDepth 估计深度图，再通过相机内参将每个像素反投影为三维坐标 P（相机自身坐标系，无需外参标定）。三维位置用正弦函数 γ(·) 编码后经 MLP 映射，与 SigLIP 提取的 2D 语义特征 X 相加融合：Given an RGB image, a depth map is first estimated with ZoeDepth, then every pixel is back-projected through the camera intrinsics into a 3D coordinate P (in the camera’s own frame, requiring no extrinsic calibration). The 3D position is encoded by a sinusoidal function γ(·), mapped through an MLP, and fused by addition with the 2D semantic features X extracted by SigLIP:
O3d = X + MLP(γ(P))
该设计以 plug-and-play 方式为视觉 token 注入空间感知，无需额外相机标定，适用于任意机器人平台。This design injects spatial awareness into visual tokens in a plug-and-play fashion, needs no extra camera calibration, and applies to arbitrary robot platforms.
图2：Adaptive Action Grids 示意。 (a) 预训练数据集上平移与旋转动作的统计分布；(b) 根据拟合 Gaussian 的概率密度函数将动作空间划分为等概率区间；(c) 最终获得的自适应动作网格（平移 + 旋转空间各维度）。Figure 2: Illustration of Adaptive Action Grids. (a) Statistical distribution of translation and rotation actions over the pre-training datasets; (b) the action space is partitioned into equiprobable intervals according to the probability density function of the fitted Gaussian; (c) the resulting adaptive action grids (each dimension of the translation and rotation spaces).
Adaptive Action Grids
将连续 7D 动作（平移 x,y,z；旋转 roll,pitch,yaw；夹爪）离散化为可学习 token。关键创新在于自适应分箱：先将平移转为极坐标 (φ, θ, r) 解耦方向与距离，再对各维度拟合 Gaussian 分布，按等概率划分 M 个区间，使每个 bin 覆盖相同比例的训练动作，避免传统线性分箱在长尾分布上的浪费。The continuous 7D action (translation x,y,z; rotation roll,pitch,yaw; gripper) is discretized into learnable tokens. The key innovation is adaptive binning: the translation is first converted into polar coordinates (φ, θ, r) to decouple direction from distance, then a Gaussian is fitted per dimension and split into M equiprobable intervals, so that every bin covers the same fraction of the training actions, avoiding the waste of conventional linear binning under long-tailed distributions.
跨机器人迁移：微调至新机器人时，对目标数据集重新拟合 Gaussian，通过三线性插值将预训练 token embedding 对齐到新网格，保留空间先验同时快速适应新动作分布（即 Spatial Embedding Adaptation）。Cross-robot transfer: when fine-tuning to a new robot, the Gaussian is refitted on the target dataset and the pre-trained token embeddings are aligned to the new grid via trilinear interpolation, preserving the spatial prior while quickly adapting to the new action distribution (i.e. Spatial Embedding Adaptation).
图7：空间网格截面特征可视化。 Spatial Embedding Adaptation 将预训练空间网格特征与微调后模型特征对齐，改善初始化并加速收敛。左：无自适应；右：使用自适应后特征分布更一致。Figure 7: Feature visualization of spatial grid cross-sections. Spatial Embedding Adaptation aligns the pre-trained spatial grid features with the features of the fine-tuned model, improving initialization and accelerating convergence. Left: without adaptation; right: with adaptation the feature distribution is more consistent.
预训练设置Pre-training Setup
以 Qwen2 为语言骨干，SigLIP 为视觉编码器。预训练数据为 Open X-Embodiment (OXE) 中 1.1M 真实机器人 episodes 的混合（Google Fractal、BridgeV2 等多机器人数据集）。Action grid 分辨率默认 8194 token，覆盖平移 + 旋转 + 夹爪各维度。Qwen2 serves as the language backbone and SigLIP as the vision encoder. The pre-training data is a mixture of 1.1M real-robot episodes from Open X-Embodiment (OXE) (multi-robot datasets such as Google Fractal and BridgeV2). The action grid resolution defaults to 8194 tokens, covering every dimension of translation, rotation and the gripper.
03实验Experiments
评估涵盖三大维度：零样本控制（SimplerEnv）、适应新机器人（Franka + WidowX 微调）、空间理解能力（空间布局任务）。仿真基准 SimplerEnv 含 Google Robot 和 WidowX 两个平台，LIBERO 提供 4 个子任务集；真实机器人实验覆盖 7 类任务套件、16 个任务。The evaluation spans three dimensions: zero-shot control (SimplerEnv), adaptation to new robots (Franka + WidowX fine-tuning), and spatial understanding (spatial layout tasks). The simulation benchmark SimplerEnv covers the two platforms Google Robot and WidowX, and LIBERO provides 4 task suites; the real-robot experiments cover 7 task suites and 16 tasks.
图3：实验配置。 跨 7 类机器人学习场景、16 个真实机器人任务、48 个仿真配置评估，聚焦零样本控制、新配置适应性与空间理解三个核心问题。Figure 3: Experimental setup. Evaluation across 7 robot-learning scenarios, 16 real-robot tasks and 48 simulation configurations, focusing on three core questions: zero-shot control, adaptability to new setups, and spatial understanding.
SimplerEnv — Google Robot（表 I）SimplerEnv — Google Robot (Table I)
图4：WidowX 机器人零样本控制评估。 跨 7 类任务套件，探测语言 grounding、语义理解与运动感知能力，背景、物体姿态与运动干扰物均有变化。Figure 4: Zero-shot control evaluation on the WidowX robot. Across 7 task suites, probing language grounding, semantic understanding and motion perception, with variations in background, object pose and moving distractors.图6：空间理解能力评估。 得益于 Ego3D Position Encoding，SpatialVLA 在需要理解空间提示词（如"左侧""前方"）及复杂空间布局的任务中表现显著优于基线。Figure 6: Evaluation of spatial understanding. Thanks to Ego3D Position Encoding, SpatialVLA performs significantly better than the baselines on tasks that require understanding spatial prompts (such as “on the left” or “in front”) and complex spatial layouts.
消融实验Ablation Studies
预训练消融（表 IV）显示：将 Adaptive Grids 替换为线性 256-bin 分箱后，Variant Aggregation 指标下降约 36.5%；去除 Ego3D 编码后，Google Robot 零样本性能下降 12.7%–15.2%。Action grid 分辨率从 1026 提升至 8194 持续带来收益。The pre-training ablation (Table IV) shows that replacing Adaptive Grids with linear 256-bin binning lowers the Variant Aggregation metric by about 36.5%, and removing the Ego3D encoding lowers Google Robot zero-shot performance by 12.7%–15.2%. Raising the action grid resolution from 1026 to 8194 keeps bringing gains.
Note:以下局限性均为作者在论文 Discussion & Limitations 节中明确陈述。All limitations below are explicitly stated by the authors in the Discussion & Limitations section of the paper.
Gaussian 分布建模不够最优Modeling the data distribution as a Gaussian is not optimal
"Is modeling data distributions as Gaussian optimal? We argue that Gaussian modeling is suboptimal, as it can lead to grid clustering on specific coordinate axes in extreme robot operation scenarios, such as single-axis motion, resulting in lost motion capabilities on other axes."（单轴运动等极端场景下，Gaussian 拟合可能导致某些轴的网格过度聚集，使其他轴的运动能力退化。）(In extreme scenarios such as single-axis motion, Gaussian fitting may cause the grids to cluster excessively on certain axes, degrading motion capability on the remaining axes.)
仅依赖当前帧，长时序任务表现受限Relies on the current frame only, limiting performance on long-horizon tasks
"As the model relies solely on current frame observations and history tokens for action prediction, it faces challenges in long-horizon tasks." 作者指出未来需要设计高效的历史信息感知机制以增强长序列建模能力。The authors note that future work needs an efficient mechanism for perceiving historical information in order to strengthen long-sequence modeling.
推理速度慢于 diffusion 方法Inference is slower than diffusion-based methods
"SpatialVLA achieves 21Hz inference speed, it is slower than diffusion decoding." 自回归 token 预测的推理开销高于基于扩散的策略网络，在对实时性要求极高的场景下存在瓶颈。The inference overhead of autoregressive token prediction is higher than that of diffusion-based policy networks, which is a bottleneck in scenarios with extremely strict real-time requirements.
预训练数据质量参差Uneven quality of the pre-training data
"The variable quality of OXE data can hinder training. Therefore, future work exploring optimal data composition and distilling high-quality subsets from the heterogeneous robot data collections is vital for boosting model efficiency and generalizability."
