Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration
Jiahui Zhang, Yurui Chen, Yueming Xu, Ze Huang, Yanpeng Zhou, Yu-Jie Yuan, Xinyue Cai, Guowei Huang, Xingyue Quan, Hang Xu, Li Zhang† · 复旦大学数据科学学院 & 华为诺亚方舟实验室School of Data Science, Fudan University & Huawei Noah's Ark Lab
4D-VLA 将 RGB-D 深度信息与历史时序帧融入 VLA 预训练，通过空间对齐 token 和记忆库采样（Memory Bank Sampling）解决多机器人数据集联合训练时出现的"坐标系混乱"与"状态混乱"问题，在 LIBERO 仿真平台和真实操作实验中均大幅超越 OpenVLA 基线。4D-VLA incorporates RGB-D depth information and historical temporal frames into VLA pretraining. Through spatially aligned tokens and Memory Bank Sampling, it resolves the "coordinate system chaos" and "state chaos" that arise when jointly training on multiple robotic datasets, substantially outperforming the OpenVLA baseline on both the LIBERO simulation platform and real-world manipulation experiments.
2025 · 复旦大学2025 · Fudan UniversityLIBERO Avg +12.1pp vs OpenVLA真实场景 Avg 85.63% vs 27.70%Real-world Avg 85.63% vs 27.70%📄 arXiv:2506.22242GitHub 代码GitHub code
VLA机器人预训练robot pretrainingRGB-D4D表征4D representationspatiotemporal reasoningmemory bank sampling多视角泛化multi-view generalizationcoordinate alignmentimitation learningMV-Bench
01动机Motivation
大规模机器人预训练的关键障碍：当把多个异构机器人数据集混合训练时，单帧 RGB 图像缺乏充足的空间与时序上下文，导致 action 分布极度分散，模型难以收敛。The key obstacle to large-scale robot pretraining: when multiple heterogeneous robotic datasets are mixed for training, a single RGB frame lacks sufficient spatial and temporal context, so the action distribution becomes extremely dispersed and the model struggles to converge.
坐标系混乱（Coordinate System Chaos）Coordinate System Chaos
动作定义在机器人坐标系中，而视觉输入缺乏足够的空间上下文。论文指出："if the image does not fully capture the robot's body, it becomes challenging to infer the robot's exact position and orientation." 不同相机外参与内参使得同一动作在不同数据集中的视觉表征截然不同。Actions are defined in the robot coordinate frame, whereas the visual input lacks sufficient spatial context. The paper notes: "if the image does not fully capture the robot's body, it becomes challenging to infer the robot's exact position and orientation." Different camera extrinsics and intrinsics make the visual representation of one and the same action look entirely different across datasets.
状态混乱（State Chaos）State Chaos
单帧图像缺少必要的时序与上下文线索以消除动作歧义。论文指出这包括"symmetric trajectories——where it is difficult to infer the direction of motion"，以及"visually similar observations correspond to entirely different actions"的情形，使得模型对当前运动方向无法判断。A single frame lacks the temporal and contextual cues needed to disambiguate actions. The paper notes that this covers "symmetric trajectories——where it is difficult to infer the direction of motion", as well as cases where "visually similar observations correspond to entirely different actions", leaving the model unable to tell the current direction of motion.
"We identify two primary factors contributing to incomplete input: coordinate system chaos and state chaos, both of which severely limit the training efficiency achievable with diverse robotic datasets."
图 1：左侧展示坐标系混乱问题——不同数据集的相机视角与机器人摆放导致同一动作在视觉上呈现截然不同的分布；右侧对比 OpenVLA 与 4D-VLA 的整体性能，后者在 LIBERO 和真实场景均取得显著提升。Figure 1: The left panel illustrates coordinate system chaos — differing camera viewpoints and robot placements across datasets make the same action appear as radically different visual distributions; the right panel compares the overall performance of OpenVLA and 4D-VLA, the latter achieving marked gains on both LIBERO and real-world scenes.
88.6%LIBERO Avg（±0.3）LIBERO Avg (±0.3)
+12.1ppvs OpenVLA (76.5%)
85.63%真实操作 Full ModelReal-world manipulation, Full Model
81.0%MV-Bench In-View Avg
02方法Method
4D-VLA 以顺序 RGB-D 输入为基础，构建两类核心机制：① 空间感知视觉 token（Spatial Vision Token）将深度信息提升至世界坐标，实现跨场景坐标系对齐；② 记忆库采样（Memory Bank Sampling, MBS）从时序窗口中自适应挑选 k=5 帧历史信息，既捕捉关键状态变化，又避免冗余。Built on sequential RGB-D input, 4D-VLA introduces two core mechanisms: ① the Spatial Vision Token lifts depth information into world coordinates, achieving cross-scene coordinate-frame alignment; ② Memory Bank Sampling (MBS) adaptively picks k=5 historical frames from a temporal window, capturing key state changes while avoiding redundancy.
图 2：4D-VLA 整体 pipeline。视觉编码器提取 RGB 特征图；深度图经由相机内外参反投影至世界坐标，通过可学习位置嵌入 ℰₛ 编码后与视觉特征做逐元素加法，形成 spatial vision token eˢᵀ；记忆库从 n=20 帧的时序窗口中选取 k=5 帧历史 token；拼接文本 token 后输入 LLM 骨干，最终接 MLP action head 输出预测动作。Figure 2: The overall 4D-VLA pipeline. The visual encoder extracts an RGB feature map; the depth map is back-projected into world coordinates via the camera intrinsics and extrinsics, encoded by the learnable positional embedding ℰₛ and added element-wise to the visual features, forming the spatial vision token eˢᵀ; the memory bank selects k=5 historical frames of tokens from a temporal window of n=20 frames; after the text tokens are concatenated, the sequence is fed to the LLM backbone, and an MLP action head finally outputs the predicted action.
给定 RGB 图像 I ∈ ℝ³ˣʰˣʷ，视觉编码器 ℰ 提取特征图 fᵥ = ℰ(I)。深度图 D 通过相机外参 [R|T] 和内参 K 反投影得到世界坐标点云 Pₘ，再经可学习位置嵌入 ℰₛ 编码并与视觉特征做逐元素加法：Given an RGB image I ∈ ℝ³ˣʰˣʷ, the visual encoder ℰ extracts the feature map fᵥ = ℰ(I). The depth map D is back-projected through the camera extrinsics [R|T] and intrinsics K into the world-coordinate point cloud Pₘ, which is then encoded by the learnable positional embedding ℰₛ and added element-wise to the visual features:
eˢᵀ = P(ℰ(I) + ℰₛ(Pₘ))
此设计使 token 同时携带外观与精确三维位置信息，消除坐标系混乱。This design makes each token carry appearance and precise 3D position at the same time, removing coordinate system chaos.
记忆库采样（Memory Bank Sampling, MBS）Memory Bank Sampling (MBS)
从长度为 n=20 的历史帧序列中自适应选取 k=5 帧：算法顺序遍历帧组，维护一个相似度队列，保证每帧与已选帧的相似度低于当前队列最大值，从而避免冗余、保留关键状态转变。历史 token 与时序偏移通过可学习编码 ℰₜ(t-j) 注入相对位置信息，最终与当前帧合并：From a history sequence of length n=20, k=5 frames are adaptively selected: the algorithm walks through the frame groups in order and maintains a similarity queue, ensuring that each frame's similarity to the already selected frames stays below the current queue maximum, thereby avoiding redundancy while preserving key state transitions. Relative positional information for the historical tokens and their temporal offsets is injected through the learnable encoding ℰₜ(t-j), and they are finally merged with the current frame:
𝒳 = ⋃ᵢ∈ℋ [eᵢᵀ | eᵢˢᵀ] ∪ {eᵗᵉˣᵗ}
Action head 采用 MLP，在延迟（160.0 ms）与成功率（86.6%）之间取得最优平衡，相比 Autoregressive（0.6 FPS）和 Diffusion head 均有优势。The action head is an MLP, striking the best balance between latency (160.0 ms) and success rate (86.6%), and outperforming both the autoregressive head (0.6 FPS) and the diffusion head.
MV-Bench 多视角评测基准MV-Bench: a multi-view evaluation benchmark
论文提出 MV-Bench——一个覆盖 270° 前向视角范围内 6 个训练视角的仿真评测集，同时评估 In-View（已见视角）与 Cross-View（偏移 Δ15°/Δ30°）的泛化能力，填补了现有 VLA 多视角评测的空白。The paper proposes MV-Bench — a simulated evaluation suite with 6 training viewpoints spanning a 270° frontal range — which measures generalization under both In-View (seen viewpoints) and Cross-View (Δ15°/Δ30° offsets) settings, filling a gap in existing multi-view evaluation of VLAs.
图 3：MV-Bench 6 个训练相机视角的设置（覆盖 0°、60°、120°、270°、300°、330°），用于评估 VLA 的空间感知与新视角泛化能力。Figure 3: The layout of the 6 training camera viewpoints in MV-Bench (covering 0°, 60°, 120°, 270°, 300°, 330°), used to assess a VLA's spatial perception and novel-view generalization.
03实验Experiments
评测平台：LIBERO 仿真基准（4 个子任务）、MV-Bench 多视角仿真评测、真实 Franka 机器臂操作（4 项任务 × 多视角泛化）。基线方法：OpenVLA。所有数字均源自论文原文。Evaluation platforms: the LIBERO simulation benchmark (4 sub-task suites), the MV-Bench multi-view simulation evaluation, and real Franka arm manipulation (4 tasks × multi-view generalization). Baseline: OpenVLA. All numbers are taken from the paper.
Table 3 · 真实机器臂操作Table 3 · Real robot arm manipulation
配置Configuration
Task 1
Task 2
Task 3
Task 4
Avg
OpenVLA
45.00
22.50
30.00
13.33
27.70
Base VLA
35.00
20.00
5.00
2.67
15.67
+Pretraining
60.00
60.00
40.00
28.00
47.00
+Pretraining+Coord
75.00
60.00
85.00
34.67
63.67
+Pretraining+Hist
80.00
77.50
70.00
36.00
65.88
Full Model（Ours）Full Model (Ours)
90.00
82.50
90.00
80.00
85.63
图 5：真实场景多视角泛化测试——标准视角（In-View）与新视角（Δ15°、Δ25°、Δ45° 偏移）下的成功率对比。4D-VLA 在所有偏移角度均显著优于 OpenVLA（平均 55% vs 18%），体现出空间感知预训练带来的跨视角泛化能力。Figure 5: Real-world multi-view generalization test — success rates under the standard viewpoint (In-View) and novel viewpoints (Δ15°, Δ25°, Δ45° offsets). 4D-VLA is markedly better than OpenVLA at every offset angle (55% vs 18% on average), reflecting the cross-view generalization brought by spatially aware pretraining.
消融实验（Ablations）Ablations
Table 5 对时序编码方式进行系统消融：可学习相对位置编码 + Concat 融合达到最高成功率 75.6%；使用绝对位置编码则完全失效（0.0%）；去掉所有历史编码降至 63.0%。Table 5 systematically ablates the temporal encoding scheme: learnable relative positional encoding with Concat fusion reaches the highest success rate, 75.6%; absolute positional encoding fails completely (0.0%); dropping all history encoding falls to 63.0%.
Table 8 对帧采样方法进行对比（LIBERO-Spatial）：MBS（0.866）优于单帧（0.738）、Adaptive Pooling（0.604）、Grid Pooling（0.620）和 Q-Former（0.556），同时延迟 160.0 ms 与显存占用 8682.9 MB 均处于合理水平。Table 8 compares frame sampling methods (LIBERO-Spatial): MBS (0.866) beats single-frame (0.738), Adaptive Pooling (0.604), Grid Pooling (0.620) and Q-Former (0.556), while its latency of 160.0 ms and memory footprint of 8682.9 MB both stay at reasonable levels.
图 6：历史帧窗口大小与帧采样效率的对比分析。随时序窗口 n 增大，成功率先升后趋于稳定；MBS 在 n=20、k=5 时取得最佳权衡。Figure 6: Comparative analysis of history window size and frame sampling efficiency. As the temporal window n grows, the success rate first rises and then levels off; MBS attains the best trade-off at n=20, k=5.
04局限性Limitations
说明：以下第 1 条为论文作者在正文中明确陈述的局限；第 2、3 条为基于方法设计推断（inferred）的潜在局限，未在论文中直接点明。Note: Item 1 below is a limitation explicitly stated by the authors in the paper; items 2 and 3 are potential limitations inferred from the method design and are not pointed out directly in the paper.
依赖 RGB-D 硬件（stated）Reliance on RGB-D hardware (stated)
论文明确指出："A limitation of our approach is its reliance on RGB-D input, which introduces hardware restriction." 相比纯 RGB 方法，4D-VLA 需要深度传感器，限制了在无深度摄像头场景下的直接部署。The paper states explicitly: "A limitation of our approach is its reliance on RGB-D input, which introduces hardware restriction." Compared with pure-RGB methods, 4D-VLA needs a depth sensor, which limits direct deployment in settings without a depth camera.
计算与延迟开销（inferred）Compute and latency overhead (inferred)
Memory Bank Sampling 引入 k=5 历史帧的 spatial token，相比单帧推理延迟从 76.5 ms 增加至 160.0 ms（约 2.1×）。在对实时性要求极高的场景（如高速操作）下可能存在瓶颈。Memory Bank Sampling brings in spatial tokens for k=5 historical frames, raising inference latency from 76.5 ms for a single frame to 160.0 ms (about 2.1×). This may become a bottleneck in scenarios with extremely strict real-time requirements, such as high-speed manipulation.
相机标定精度依赖（inferred）Dependence on camera calibration accuracy (inferred)
空间感知 token 的构造依赖精确的相机内参 K 和外参 [R|T] 将深度图反投影至世界坐标。若标定存在误差或外参在运行中漂移，空间对齐效果会受损，可能影响在标定精度受限场景下的稳定性。Constructing the spatially aware tokens relies on accurate camera intrinsics K and extrinsics [R|T] to back-project the depth map into world coordinates. If the calibration carries error or the extrinsics drift at run time, spatial alignment degrades, which may affect stability in settings where calibration accuracy is limited.
