RLDX-1 Technical Report

01 动机Motivation

VLA 模型通过继承预训练 Vision-Language Model 的通用理解与语言条件泛化能力，在机器人操作领域取得了显著进步——但这种"通用智能"并不等于"灵巧操作所需的全部能力"。在动态环境（如传送带上的移动物体）、需要记住历史交互状态的任务、以及依赖触觉/力矩反馈的接触丰富场景中，现有 VLA 普遍失效。By inheriting the general understanding and language-conditioned generalization of pre-trained Vision-Language Models, VLA models have made striking progress in robotic manipulation — but this "versatile intelligence" is not the same as "everything dexterous manipulation requires". In dynamic environments (e.g. objects moving on a conveyor belt), in tasks that require remembering the state of past interactions, and in contact-rich settings that rely on tactile/torque feedback, existing VLAs generally fail.

"While Vision-Language-Action models (VLAs) have shown remarkable progress toward human-like generalist robotic policies through the versatile intelligence … they still struggle with complex real-world tasks requiring broader functional capabilities (e.g. motion awareness, long-term memory, and physical sensing)."

RLDX-1 overview — 图 1：RLDX-1 系统概览。该模型整合三类核心功能能力：运动感知（Motion Awareness）用于动态环境下的物体轨迹追踪；长时记忆（Long-Term Memory）用于基于历史状态的决策；物理感知（Physical Sensing）用于接触丰富任务中的力/触觉推断。（来源：论文 Figure 1）Figure 1: Overview of the RLDX-1 system. The model integrates three core functional capabilities: Motion Awareness, for tracking object trajectories in dynamic environments; Long-Term Memory, for decision-making conditioned on past states; and Physical Sensing, for inferring force/tactile signals in contact-rich tasks. (Source: paper Figure 1)

97.8%LIBERO 平均成功率
（超越 GR00T N1.6 的 96.7%）LIBERO average success rate
(above the 96.7% of GR00T N1.6)

91.7%ALLEX 长时记忆任务
（Object-in-Box Selection）ALLEX long-term-memory task
(Object-in-Box Selection)

87.5%ALLEX 运动感知任务
（Conveyor Pick-and-Place）ALLEX motion-awareness task
(Conveyor Pick-and-Place)

1.63×推理加速比
（71.2ms → 43.7ms）Inference speedup
(71.2ms → 43.7ms)

02 方法Method

RLDX-1 包含四大核心组件：(1) Multi-Stream Action Transformer（MSAT）神经网络架构；(2) 基于运动一致性过滤的合成数据生成管线；(3) 预训练 → 中训练 → 后训练的三阶段训练流程；(4) 静态图转换与算子融合推理优化。RLDX-1 comprises four core components: (1) the Multi-Stream Action Transformer (MSAT) neural network architecture; (2) a synthetic data generation pipeline built on motion-consistency filtering; (3) a three-stage training procedure of pre-training → mid-training → post-training; and (4) inference optimization via static graph conversion and kernel fusion.

图 2：RLDX-1 架构（MSAT）。模型由 Vision-Language Model（VLM）和动作模型两部分组成。VLM 接收多帧视频观测，通过 Motion Module 捕获时序动态，通过 Memory Module 维护历史认知特征队列；物理信号（触觉/力矩）通过独立的 Physics（P）流输入。各模态流通过跨模态联合自注意力（joint self-attention）交互。（来源：论文 Figure 3）Figure 2: The RLDX-1 architecture (MSAT). The model has two parts, a Vision-Language Model (VLM) and an action model. The VLM takes multi-frame video observations, captures temporal dynamics through the Motion Module and maintains a queue of past cognition features through the Memory Module; physical signals (tactile/torque) enter through a separate Physics (P) stream. The modality streams interact through cross-modal joint self-attention. (Source: paper Figure 3)

Multi-Stream Action Transformer（MSAT）Multi-Stream Action Transformer (MSAT)

MSAT 在标准 flow-matching VLA 架构（π₀）基础上，为每种模态分配专用流（dedicated stream），通过 joint self-attention 实现跨模态交互，同时保留各模态专属参数。具体地：Building on the standard flow-matching VLA architecture (π₀), MSAT assigns a dedicated stream to each modality, realizes cross-modal interaction through joint self-attention, and at the same time keeps modality-specific parameters. Specifically:

Motion Awareness（运动感知）：在视觉编码器各层中插入 Motion Module，利用 space-time self-similarity（STSS）捕获帧间时序动态，通过残差连接集成，无需额外视频帧存储。Motion Awareness: Motion Modules are inserted into every layer of the vision encoder and use space-time self-similarity (STSS) to capture inter-frame temporal dynamics; they are integrated through residual connections and need no extra video-frame storage.
Long-Term Memory（长时记忆）：Memory Module 维护最近 n_mem 个以 H+1 时步为间隔采样的"认知特征"（cognition feature）队列，由轻量级 Transformer 配合因果注意力处理历史观测。Long-Term Memory: the Memory Module maintains a queue of the most recent n_mem "cognition features", sampled at intervals of H+1 timesteps, and processes the past observations with a lightweight Transformer using causal attention.
Physical Sensing（物理感知）：解耦的 Physics（P）流独立处理触觉传感器与力矩信号，训练时辅助预测未来物理信号作为额外监督目标。Physical Sensing: a decoupled Physics (P) stream processes tactile-sensor and torque signals independently and, during training, auxiliarily predicts future physical signals as an additional supervision target.

合成数据生成与过滤Synthetic Data Generation and Filtering

Synthetic data pipeline — 图 3：合成数据生成框架。(1) 数据生成：对源演示进行场景/任务增广，并用逆动力学模型（IDM）标注动作标签；(2) 两阶段过滤：VLM 评估"指令跟随"与"轨迹合理性"（视频质量过滤），再通过轻量级 attentive probe 对比生成视频与仿真回放（运动一致性过滤）。共生成 150K 条合成演示数据。（来源：论文 Figure 4）Figure 3: The synthetic data generation framework. (1) Data generation: source demonstrations are augmented over scenes/tasks and action labels are annotated with an inverse dynamics model (IDM); (2) Two-stage filtering: a VLM rates "instruction following" and "trajectory plausibility" (video quality filtering), then a lightweight attentive probe compares the generated video against a simulation replay (motion consistency filtering). 150K synthetic demonstrations are produced in total. (Source: paper Figure 4)

合成数据管线共生成 150K 条人形机器人演示数据，用于补充真实数据中稀缺的灵巧操作场景。两阶段过滤机制有效剔除不合格样本，在 GR-1 Tabletop 基准上，加入合成数据后成功率相对仅用真实数据提升了 9.1%（41.0% → 50.1%）。The synthetic data pipeline produces 150K humanoid-robot demonstrations in total, supplementing the dexterous manipulation scenarios that are scarce in real data. The two-stage filter effectively removes substandard samples: on the GR-1 Tabletop benchmark, adding synthetic data raises the success rate by 9.1% over real data alone (41.0% → 50.1%).

三阶段训练流程Three-Stage Training Procedure

预训练（Pre-training）：在 1.5M 条多具身数据集上训练 100K 步（batch size 8192），涵盖单臂 gripper、双臂 gripper 和人形机器人，共 64 块 NVIDIA H200 GPU 训练约 195 小时。Pre-training: 100K steps on a 1.5M-episode cross-embodiment dataset (batch size 8192), covering single-arm grippers, bimanual grippers and humanoid robots; about 195 hours on 64 NVIDIA H200 GPUs in total.
中训练（Mid-training）：在 ALLEX 和 FR3 的专属数据（含合成数据）上训练 25K 步，引入 Motion、Memory、Physical Sensing 三大模块（dropout 0.3，2K 步热身对齐），共 15 小时。Mid-training: 25K steps on ALLEX- and FR3-specific data (including synthetic data), introducing the three modules Motion, Memory and Physical Sensing (dropout 0.3, with a 2K-step warm-up for alignment); 15 hours in total.
后训练（Post-training）：基于改进版 RECAP 框架的强化学习，采用"基于文本的 VLM critic"（text-based VLM critic）自回归预测价值函数，配合自适应数据收集。Post-training: reinforcement learning based on an improved RECAP framework, using a text-based VLM critic to autoregressively predict the value function, together with adaptive data collection.

推理优化Inference Optimization

通过两项关键技术实现 1.63× 端到端推理加速（71.2ms → 43.7ms）：Two key techniques deliver a 1.63× end-to-end inference speedup (71.2ms → 43.7ms):

Static Graph Conversion：将前向传播转换为单一 CUDA Graph，消除重复 kernel 启动开销，并预计算 rotary position embeddings 和 attention masks。Static Graph Conversion: the forward pass is converted into a single CUDA Graph, eliminating repeated kernel-launch overhead, and rotary position embeddings and attention masks are precomputed.
Kernel Optimization（算子融合）：自定义 kernel 融合关键算子组，协调数据移动与计算，减少短序列推理中的内存往返开销。Kernel Optimization: custom kernels fuse critical operator groups and coordinate data movement with computation, reducing memory round-trip overhead in short-sequence inference.

03 实验Experiments

实验在仿真基准（LIBERO、SIMPLER、RoboCasa、GR-1 Tabletop）和三类真实机器人平台（OpenArm 28-DoF 人形、ALLEX 48-DoF 人形、Franka Research 3 单臂）上进行，主要对比 π₀-FAST、π₀、π₀.₅、GR00T N1.5、GR00T N1.6。Experiments are run on simulation benchmarks (LIBERO, SIMPLER, RoboCasa, GR-1 Tabletop) and on three classes of real robot platform (the OpenArm 28-DoF humanoid, the ALLEX 48-DoF humanoid and the single-arm Franka Research 3), compared mainly against π₀-FAST, π₀, π₀.₅, GR00T N1.5 and GR00T N1.6.

仿真基准（Table 1a：经典基准）Simulation Benchmarks (Table 1a: Classic Benchmarks)

方法Method	LIBERO Short	LIBERO Long	LIBERO Avg	LIBERO-Plus	SIMPLER Google-VM	SIMPLER WidowX
π₀-FAST	93.9	60.2	85.5	64.2	61.9	48.3
π₀	97.1	85.2	94.1	54.6	58.8	27.1
π₀.₅	98.0	92.0	96.9	86.5	72.7	46.9
GR00T N1.6	97.4	94.4	96.7	72.6	76.1	57.1
RLDX-1（本文）RLDX-1 (ours)	98.6	95.3	97.8	86.7	81.5	71.9

仿真基准（Table 1b：复合挑战基准）Simulation Benchmarks (Table 1b: Composite Challenge Benchmarks)

方法Method	RoboCasa Kitchen	GR-1 Tabletop	RoboCasa365 Comp.-S	RoboCasa365 Comp.-U	RoboCasa365 Avg
π₀	62.5	13.6	6.1	1.1	14.8
π₀.₅	62.1	15.4	7.1	1.2	16.9
GR00T N1.5	65.7	48.0	9.6	4.4	20.0
GR00T N1.6	66.2	47.6	12.6	2.6	26.9
RLDX-1（本文）RLDX-1 (ours)	70.6	58.7	19.0	5.6	32.1

真实机器人：ALLEX 人形基准Real Robots: the ALLEX Humanoid Benchmark

ALLEX benchmark results — 图 4：ALLEX 人形基准结果。RLDX-1 在所有任务类别上大幅超越基线：运动感知（Conveyor Pick-and-Place）、长时记忆（Object-in-Box Selection）、物理感知（Card Slide-and-Pick, Pot-to-Cup-Pouring）。ALLEX 是一款 48-DoF 上半身人形机器人，配备立体自我中心相机。（来源：论文 Figure 16）Figure 4: Results on the ALLEX humanoid benchmark. RLDX-1 surpasses the baselines by a wide margin in every task category: motion awareness (Conveyor Pick-and-Place), long-term memory (Object-in-Box Selection) and physical sensing (Card Slide-and-Pick, Pot-to-Cup-Pouring). ALLEX is a 48-DoF upper-body humanoid robot equipped with a stereo egocentric camera. (Source: paper Figure 16)

在 ALLEX 人形基准上，关键结果如下（作者原文引用）：On the ALLEX humanoid benchmark the key results are as follows (quoted from the authors):

长时记忆任务（Object-in-Box Selection）："both GR00T N1.6 and π₀.₅ achieve success rates in the 30% range, whereas RLDX-1 achieves a substantially higher success rate of 91.7%"Long-term memory task (Object-in-Box Selection): "both GR00T N1.6 and π₀.₅ achieve success rates in the 30% range, whereas RLDX-1 achieves a substantially higher success rate of 91.7%"
运动感知任务（Conveyor Pick-and-Place）："RLDX-1 reaches a success rate of over 87.5% while π₀.₅ remains below 29.2%"Motion awareness task (Conveyor Pick-and-Place): "RLDX-1 reaches a success rate of over 87.5% while π₀.₅ remains below 29.2%"
整体 ALLEX 成功率：约 90%，而前沿 VLA 维持在约 40%。Overall ALLEX success rate: about 90%, while frontier VLAs stay at about 40%.

消融实验（Ablation Studies）Ablation Studies

论文对多个设计维度进行消融验证：The paper ablates several design dimensions:

合成数据比例：随合成数据加入比例（0% → 25% → 50% → 100%），GR-1 Tabletop 成功率从 41.0% 依次提升至 45.6% → 46.6% → 50.1%，提升 9.1%。Synthetic data ratio: as the share of added synthetic data grows (0% → 25% → 50% → 100%), the GR-1 Tabletop success rate rises from 41.0% through 45.6% → 46.6% → 50.1%, a gain of 9.1%.
VLM 层选择：在第 18 层插入接口效果最佳（60.9%），高于第 8 层（51.1%）和第 28 层（56.3%）。VLM layer choice: inserting the interface at layer 18 works best (60.9%), ahead of layer 8 (51.1%) and layer 28 (56.3%).
推理优化逐步叠加：PyTorch Eager（71.2ms）→ CUDA Graph + Torch.Compile（59.6ms，1.19×）→ Static Graph（48.9ms，1.46×）→ Kernel Optimization（43.7ms，1.63×）。Cumulative inference optimizations: PyTorch Eager (71.2ms) → CUDA Graph + Torch.Compile (59.6ms, 1.19×) → Static Graph (48.9ms, 1.46×) → Kernel Optimization (43.7ms, 1.63×).
强化学习 critic 对比：text-based VLM critic 产生更单调的值函数曲线，有助于 test-time sampling 中选择最优动作块。Reinforcement-learning critic comparison: the text-based VLM critic yields a more monotonic value-function curve, which helps select the best action chunk during test-time sampling.

04 局限性Limitations

Note：论文原文（Section 8 Conclusion）未设独立 Limitations 章节。以下各条均为从设计与实验中推断（inferred from design and experiments），如有明确陈述会标注。Note: The paper itself (Section 8 Conclusion) has no dedicated Limitations section. Every item below is inferred from design and experiments; anything explicitly stated is marked as such.

仅针对特定具身平台专项训练，泛化性有待验证Trained specifically for particular embodiments; generalization remains to be verified

RLDX-1 的中训练阶段专门针对 ALLEX 人形和 Franka Research 3 两类平台进行具身特化。这意味着应用于其他机器人形态时，需重新采集数据并重复多阶段训练流程，跨具身零样本迁移能力未被系统评估。（inferred）The mid-training stage of RLDX-1 specializes the model for two embodiments in particular, the ALLEX humanoid and the Franka Research 3. Applying it to other robot morphologies therefore requires collecting new data and repeating the multi-stage training procedure, and zero-shot cross-embodiment transfer is not systematically evaluated. (inferred)

合成数据质量依赖视频扩散模型与过滤流程的可靠性Synthetic data quality depends on the video diffusion model and the reliability of the filtering procedure

合成数据管线基于视频扩散模型生成多样化演示，再通过两阶段过滤（VLM 视频质量过滤 + 运动一致性过滤）筛选有效样本。若扩散模型对特定场景分布外泛化能力不足，或过滤器产生漏判，合成数据的有效性将受限。当前消融仅验证了 GR-1 Tabletop 基准上的效果。（inferred）The synthetic data pipeline generates diverse demonstrations with a video diffusion model and then selects valid samples through two-stage filtering (VLM video quality filtering plus motion consistency filtering). If the diffusion model generalizes poorly out of distribution for particular scenes, or if the filters miss bad samples, the usefulness of the synthetic data is limited. The current ablation only verifies the effect on the GR-1 Tabletop benchmark. (inferred)

推理优化依赖特定硬件环境（NVIDIA GPU + CUDA Graph）Inference optimization depends on a specific hardware environment (NVIDIA GPU + CUDA Graph)

静态图转换与 kernel 融合优化高度依赖 NVIDIA GPU 运行环境。在边缘设备或不同硬件栈（如 ARM CPU、其他加速器）上，所报告的 1.63× 加速收益可能无法复现。（inferred）Static graph conversion and kernel-fusion optimization depend heavily on an NVIDIA GPU runtime. On edge devices or different hardware stacks (e.g. ARM CPUs or other accelerators), the reported 1.63× speedup may not be reproducible. (inferred)

长时记忆模块的记忆容量与时序粒度固定The long-term memory module has a fixed capacity and temporal granularity

Memory Module 以固定间隔（H+1 时步）采样并维护固定数量（n_mem）的历史认知特征。对于需要跨越更长时间跨度或更细粒度历史状态的任务，该设计的扩展性未被论文深入讨论。（inferred）The Memory Module samples at a fixed interval (H+1 timesteps) and maintains a fixed number (n_mem) of past cognition features. For tasks that span longer horizons or need finer-grained historical states, the scalability of this design is not discussed in depth by the paper. (inferred)