PokeVLA

01 动机 · MotivationMotivation

现有大型 VLA 模型参数量动辄数十亿，难以在算力受限的边缘设备上部署。与此同时，仅靠模仿学习训练的小型模型严重缺乏对空间关系、操作意图和几何结构的理解，在扰动场景下泛化能力尤为不足。如何在保持轻量化的同时，赋予 VLA 模型足够丰富的 world knowledge？Existing large VLA models routinely carry billions of parameters, which makes them hard to deploy on edge devices with limited compute. At the same time, small models trained by imitation learning alone badly lack any understanding of spatial relations, manipulation intent and geometric structure, and generalize especially poorly under perturbation. How can a VLA model be endowed with sufficiently rich world knowledge while staying lightweight?

"We collect and curate a large-scale embodied multimodal dataset of approximately 2.4 million entries to pre-train a tiny-scale embodied vision-language model… introduce a novel method for learning manipulation-relevant representations, featuring multi-view consistent learning of the manipulation targets and geometric alignment."

传统 VLA 模型要么体量巨大（如 OpenVLA 7B），要么缺乏场景理解能力。PokeVLA 瞄准一个关键问题：即便是 0.5B 语言骨干，只要配合充分的 embodied 先验预训练和精心设计的操作相关表征注入，就能在模拟和真实世界中实现可靠、鲁棒的机器人操作。Conventional VLA models are either enormous (e.g. OpenVLA 7B) or lack scene understanding. PokeVLA targets one key question: even a 0.5B language backbone can achieve reliable, robust robot manipulation in simulation and in the real world, as long as it is combined with sufficient embodied prior pre-training and a carefully designed injection of manipulation-relevant representations.

02 方法 · MethodMethod

PokeVLA 采用两阶段训练范式：第一阶段在大规模具身多模态数据集上预训练出 PokeVLM；第二阶段通过目标感知语义分割（Goal-Aware Segmentation）与几何对齐（Geometry Alignment）将操作相关的表征注入到动作头中。PokeVLA follows a two-stage training paradigm: the first stage pre-trains PokeVLM on a large-scale embodied multimodal dataset; the second stage injects manipulation-relevant representations into the action head through Goal-Aware Segmentation and Geometry Alignment.

第一阶段：PokeVLM 预训练Stage 1: PokeVLM Pre-training

以 Prismatic-VLM 为骨架，语言模型选用 Qwen2.5-0.5B，视觉编码器为 SigLIP（语义）+ DinoV2（空间感知）双组件，通过 MLP projector 桥接视觉与语言空间。预训练数据集共约 2.4M 条，覆盖四类具身任务：The backbone is Prismatic-VLM, the language model is Qwen2.5-0.5B, and the vision encoder combines SigLIP (semantics) with DinoV2 (spatial perception), bridged to the language space by an MLP projector. The pre-training dataset holds roughly 2.4M entries and covers four categories of embodied tasks:

• General Understanding：665K 条，来自 LLaVA-Instruct-665KGeneral Understanding: 665K entries, from LLaVA-Instruct-665K
• Spatial Grounding：694K 条，来自 RoboPoint、RefSpatial、RoboRefit、RoboSpatialSpatial Grounding: 694K entries, from RoboPoint, RefSpatial, RoboRefit and RoboSpatial
• Affordance Learning：553K 条，来自 HOVA（Ego4D、Epic100）和 MolmoActAffordance Learning: 553K entries, from HOVA (Ego4D, Epic100) and MolmoAct
• Embodied Reasoning：511K 条，来自 RefSpatial 和 Cosmos-Reason1-SFTEmbodied Reasoning: 511K entries, from RefSpatial and Cosmos-Reason1-SFT

训练配置：AdamW 优化器，学习率 2e-5，batch size 128（8 卡 × 4 per-GPU × 4 梯度累积），线性 warmup（3% steps）+ cosine decay，共训练 2 个 epoch。Training setup: AdamW optimizer, learning rate 2e-5, batch size 128 (8 GPUs × 4 per-GPU × 4 gradient accumulation steps), linear warmup (3% of steps) + cosine decay, trained for 2 epochs.

目标感知语义分割（Goal-Aware Segmentation）Goal-Aware Segmentation

借鉴 LISA 的设计，在语言模型中引入特殊 token <SEG>，其 embedding 整合了场景上下文与目标物体细节。解码采用 coarse-to-fine 两步范式：Following the design of LISA, a special token <SEG> is introduced into the language model, whose embedding integrates the scene context with target-object details. Decoding follows a coarse-to-fine two-step paradigm:

1. Coarse Decoding：将 <SEG> embedding 通过 SAM prompt encoder，获得 sparse/dense embeddings，经粗粒度 mask decoder 得到初步掩码；Coarse Decoding: the <SEG> embedding is passed through the SAM prompt encoder to obtain sparse/dense embeddings, and a coarse-grained mask decoder yields a preliminary mask;
2. Fine-Grained Decoding：以粗掩码 logits 作为 mask prompt 输入，结合 <SEG> embedding 进一步细化预测。Fine-Grained Decoding: the coarse mask logits are fed in as a mask prompt and combined with the <SEG> embedding to further refine the prediction.

分割损失为 focal loss 与 KL divergence 之和：L_seg = λ_focal FOCAL(M̂,M) + λ_KLD KLD(M̂,M)，其中 λ_focal = λ_KLD = 1。多视角一致性通过跨视角共享 <SEG> embedding 保证，使模型对操作目标形成稳定的跨视角表征。The segmentation loss is the sum of a focal loss and a KL divergence: L_seg = λ_focal FOCAL(M̂,M) + λ_KLD KLD(M̂,M), with λ_focal = λ_KLD = 1. Multi-view consistency is enforced by sharing the <SEG> embedding across views, so that the model forms a stable cross-view representation of the manipulation target.

几何对齐（Geometry Alignment）Geometry Alignment

利用 VGGT（视觉几何基础模型）在训练阶段提取多视角图像的几何特征，通过余弦相似度损失将语言模型视觉 token 的隐状态对齐到 VGGT 特征空间：VGGT (a visual geometry foundation model) extracts geometric features from multi-view images during training, and a cosine-similarity loss aligns the hidden states of the visual tokens of the language model to the VGGT feature space:

L_geo = (1/VN) ∑[1 − cos(P(h_v), f_geo)]

几何对齐仅在训练阶段使用 VGGT，推理时无额外开销。轻量级 projector 负责维度对齐。整体训练目标为：Geometry alignment uses VGGT during training only, so there is no extra cost at inference. A lightweight projector takes care of the dimension alignment. The overall training objective is:

L = L_action + λ_seg L_seg + λ_geo L_geo

其中 λ_seg = 0.2，λ_geo = 0.4。where λ_seg = 0.2 and λ_geo = 0.4.

动作头（Action Head）Action Head

采用含 L 个 transformer 层的 cross-attention 动作头，依次对动作 latent 进行：自注意力 → 与 query embedding 及机器人状态做 cross-attention → 与视觉隐状态做 cross-attention → 与 <SEG> embedding 做 cross-attention，最终通过 LayerNorm + MLP 输出动作 chunk：The cross-attention action head contains L transformer layers, which process the action latents in turn: self-attention → cross-attention with the query embedding and the robot state → cross-attention with the visual hidden states → cross-attention with the <SEG> embedding, and finally emit an action chunk through LayerNorm + MLP:

"att^l = [SA(at^l), CA(at^l,[h_q,MLP(s_t)]), CA(at^l,h_v), CA(at^l,h_seg)]"

03 实验 · ExperimentsExperiments

在模拟平台（LIBERO、LIBERO-Plus）和真实机器人（xArm7 + 双 Realsense 相机）上对 PokeVLA 进行全面评估，对比基线为 OpenVLA-OFT 和 VLA-Adapter。PokeVLA is evaluated comprehensively both in simulation (LIBERO, LIBERO-Plus) and on a real robot (xArm7 with two Realsense cameras), with OpenVLA-OFT and VLA-Adapter as baselines.

VLM 能力评估（PokeVLM）VLM Capability Evaluation (PokeVLM)

在空间感知、空间推理等具身 VLM 基准上，PokeVLM 大幅超越 Prismatic-VLM 基线：On embodied VLM benchmarks for spatial perception and spatial reasoning, PokeVLM outperforms the Prismatic-VLM baseline by a wide margin:

LIBERO 基准（表 IV）LIBERO Benchmark (Table IV)

LIBERO-Plus 鲁棒性基准（表 V）LIBERO-Plus Robustness Benchmark (Table V)

在迁移设置下（在 LIBERO 上训练，在 LIBERO-Plus 上测试），PokeVLA 超越 OpenVLA-OFT +9.7%，超越 VLA-Adapter +20.2%。在各类扰动下的性能对比：Under the transfer setting (trained on LIBERO, tested on LIBERO-Plus), PokeVLA beats OpenVLA-OFT by +9.7% and VLA-Adapter by +20.2%. Performance under the individual perturbation types:

真实机器人实验Real-Robot Experiments

在真实场景扰动测试（Table X）中，PokeVLA 综合超越 VLA-Adapter +20%（63.0% vs 43.0%），在末端执行器初始位姿扰动（70.0%）、目标物体扰动（60.0%）、光照扰动（75.0%）三类测试中均保持较高成功率。In the real-world perturbation tests (Table X), PokeVLA outperforms VLA-Adapter overall by +20% (63.0% vs 43.0%), keeping a high success rate in all three test categories: end-effector initial-pose perturbation (70.0%), target-object perturbation (60.0%) and lighting perturbation (75.0%).

消融实验（LIBERO-Plus Transfer 设置）Ablation Study (LIBERO-Plus Transfer setting)

消融实验表明，预训练、几何对齐和目标感知分割三个模块各自独立带来明显提升，三者结合后整体达到最佳 85.3% 的迁移成功率。The ablations show that pre-training, geometry alignment and goal-aware segmentation each bring a clear gain on their own, and that combining all three reaches the best overall transfer success rate of 85.3%.

04 局限性 · LimitationsLimitations