Training-Time Action Conditioning for Efficient Real-Time Chunking
把 RTC 的推理时 inpainting 移入训练阶段，消除部署时的计算开销Moving RTC's inference-time inpainting into the training stage, eliminating the compute overhead at deployment
Kevin Black · Allen Z. Ren · Michael Equi · Sergey Levine | Physical Intelligence
Real-Time Chunking（RTC）通过在推理时对 action chunk 做 inpainting 来实现异步、连续的机器人控制，但 inpainting 带来了额外的推理延迟。本文提出 training-time RTC：在训练中随机模拟推理延迟 d，并将模型条件化在已确定的 action prefix 上生成后缀动作，彻底消除推理时的 inpainting 计算开销。该方法无需修改模型架构或机器人运行时，仅需几行代码即可实现。Real-Time Chunking (RTC) achieves asynchronous, continuous robot control by inpainting the action chunk at inference time, but the inpainting introduces additional inference latency. This paper proposes training-time RTC: randomly simulating the inference delay d during training and conditioning the model on the already-committed action prefix to generate the postfix actions, thereby completely eliminating the inpainting compute overhead at inference time. The method requires no change to the model architecture or the robot runtime, and takes only a few lines of code to implement.
实时动作块执行real-time action chunk executionaction chunkingreal-time chunkingVLAflow matchinginference accelerationtraining-time conditioning机器人控制robot control
01动机Motivation
大规模 VLA（Vision-Language-Action）模型的推理延迟可达数十至数百毫秒，而机器人需要在实时约束下连续执行动作。
RTC（Real-Time Chunking）通过异步预测 action chunk 并在推理时用 inpainting 保证连续性来缓解这一问题，
但其 inpainting 本身引入了额外的计算开销，反而加剧了延迟。The inference latency of large-scale VLA (Vision-Language-Action) models can reach tens to hundreds of milliseconds, while a robot must execute actions continuously under real-time constraints. RTC (Real-Time Chunking) alleviates this by predicting action chunks asynchronously and using inpainting at inference time to guarantee continuity, but the inpainting itself introduces extra computation and thus aggravates the latency instead.
"Unlike chatbots or search engines, embodied agents must operate in real time. The feedback loop between an agent's actions and its environment necessitates reactivity — like a human athlete, an agent cannot simply 'stop and think' while the outside world changes."
Figure 1：两个重叠 action chunk 的示意图。时刻 t 到 t+d 之间的 d 个动作来自上一个 chunk，称为 action prefix（红色）；从 t+d 开始到 chunk 末尾的动作为 action postfix（需要新生成）。Inference-time RTC 利用所有 H−s 个重叠动作（红+黄）做 guidance；training-time RTC 只使用前 d 个动作（红色）作为条件。Figure 1: Illustration of two overlapping action chunks. The d actions between time t and t+d come from the previous chunk and are called the action prefix (red); the actions from t+d to the end of the chunk are the action postfix (to be newly generated). Inference-time RTC uses all H−s overlapping actions (red + yellow) for guidance; training-time RTC uses only the first d actions (red) as conditioning.
135msInference-time RTC 推理延迟（d≈7）Inference latency of inference-time RTC (d≈7)
50Hz机器人控制频率，支持最大 200ms 延迟Robot control frequency; supports up to 200ms of latency
几行代码A few lines of code实现所需的额外修改量（无架构改动）Extra changes needed to implement it (no architectural modification)
02方法Method
Training-time RTC 的核心思想：在训练时随机采样延迟 d，将 ground-truth 的 prefix 动作作为已知条件（flow matching timestep τ=1.0，无噪声），只对 postfix 动作做去噪学习（p(At+d:H | ot, At:t+d)）。训练时就学会了在任意延迟下根据已执行动作预测后续动作，部署时只需一次标准 forward pass，无需任何 inpainting 计算。The core idea of training-time RTC: during training, randomly sample a delay d and treat the ground-truth prefix actions as known conditioning (flow matching timestep τ=1.0, noise-free), learning to denoise only the postfix actions (p(At+d:H | ot, At:t+d)). The model therefore learns during training to predict subsequent actions from the already-executed ones under an arbitrary delay, so deployment needs only a single standard forward pass and no inpainting computation whatsoever.
Figure 2：Conditioning 架构示意（以 π₀.₆ action expert 中的 diffusion transformer 为例）。Prefix 动作 token 始终输入 ground-truth、无噪声的动作值，其 flow matching timestep τ 固定为 1.0；postfix 动作 token 则使用正常的 τ 进行去噪。不同 token 之间 τ 不同，向模型传达了当前的推理延迟信息。Figure 2: Illustration of the conditioning architecture (taking the diffusion transformer inside the π₀.₆ action expert as an example). Prefix action tokens always receive the ground-truth, noise-free action values, and their flow matching timestep τ is fixed at 1.0; postfix action tokens are denoised with a normal τ. The differing τ across tokens conveys the current inference delay to the model.
三处最小改动（minimal changes）Three minimal changes
每个 token 独立的 flow matching timestep：对 adaLN-zero conditioning 稍作修改，允许不同 action token 使用不同的 τ，无额外参数。A per-token flow matching timestep: a slight modification to the adaLN-zero conditioning allows different action tokens to use different τ, with no extra parameters.
Prefix 动作输入无噪声 ground-truth：将 prefix 对应 token 的 τ 设为 1.0，输入原始动作值而非加噪版本。Prefix actions are fed as noise-free ground-truth: the τ of the corresponding prefix tokens is set to 1.0, and the raw action values are fed in instead of noised ones.
只对 postfix 计算 loss：训练时 mask 掉 prefix token 的 loss，只监督后缀动作的生成。Loss is computed on the postfix only: the loss on prefix tokens is masked out during training, supervising only the generation of the postfix actions.
训练细节Training details
基于 π₀.₆ VLA base model 进行 fine-tuning，共 8,000 gradient steps，batch size 512。Fine-tuned from the π₀.₆ VLA base model for 8,000 gradient steps in total, with batch size 512.
延迟 d 在训练中均匀采样于 0 到 10 之间，支持 50Hz 机器人上最大 200ms 的推理延迟。During training the delay d is sampled uniformly between 0 and 10, supporting up to 200ms of inference latency on a 50Hz robot.
从第 24 个 epoch 的 checkpoint 继续训练 8 个额外 epoch（仿真实验设置）。Training continues from the checkpoint of epoch 24 for 8 additional epochs (the simulation experiment setting).
03实验Experiments
实验分两部分：仿真环境使用 dynamic Kinetix benchmark；真实环境使用 π₀.₆ VLA，在 box building（组装纸箱）和 espresso making（咖啡制作）两个精细操作任务上评估。对比基线为同步推理（synchronous）和 inference-time RTC。The experiments come in two parts: simulation uses the dynamic Kinetix benchmark; the real-world setting uses the π₀.₆ VLA and is evaluated on two fine-grained manipulation tasks, box building and espresso making. The baselines for comparison are synchronous inference and inference-time RTC.
4 层 MLP-Mixer 架构，预测长度 H=8，测试延迟 0–4，每个数据点对应 2048 次 rollout，误差区间为 95% Wilson score interval。
结果显示：training-time RTC 在推理延迟 ≥ 2 时表现优于 inference-time RTC，且差距随延迟增大而显著扩大；延迟为 0–1 时因训练监督减少而略低于 inference-time RTC。A 4-layer MLP-Mixer architecture, prediction horizon H=8, test delays 0–4, 2048 rollouts per data point, and error bands given as 95% Wilson score intervals. The results show that training-time RTC outperforms inference-time RTC once the inference delay is ≥ 2, with the gap widening markedly as the delay grows; at delays of 0–1 it is slightly worse than inference-time RTC because of the reduced training supervision.
Figure 3：仿真实验结果——推理延迟 vs. solve rate（固定执行长度 s=max(d,1)）。Training-time RTC 在延迟 ≥ 2 时超越 inference-time RTC，且随延迟增大差距持续扩大。每个数据点代表 2048 次试验，阴影为 95% Wilson score interval。Figure 3: Simulation results — inference delay vs. solve rate (with the execution horizon fixed at s=max(d,1)). Training-time RTC surpasses inference-time RTC at delays ≥ 2, and the gap keeps widening as the delay grows. Each data point represents 2048 trials; the shaded band is the 95% Wilson score interval.
真实世界实验Real-world experiments
使用 π₀.₆ base model，在远程 H100 服务器上以 5 个去噪步完成推理。Training-time RTC 平均端到端延迟 108ms（d≈5），inference-time RTC 平均延迟 135ms（d≈7）。Using the π₀.₆ base model, inference is run on a remote H100 server with 5 denoising steps. Training-time RTC has an average end-to-end latency of 108ms (d≈5), while inference-time RTC averages 135ms (d≈7).
Figure 4(a)：Box building 任务——将纸板折叠组装成完整纸箱，要求精确操作。Figure 4(a): The box building task — folding cardboard into a complete box, which demands precise manipulation.Figure 4(b)：Espresso making 任务——包含研磨、填压、萃取、倒杯四个子步骤，任务链较长。Figure 4(b): The espresso making task — comprising four sub-steps (grinding, tamping, extraction and pouring), giving a relatively long task chain.
Figure 5：真实世界结果——两项任务的 success rate 和 duration。Training-time RTC 与 inference-time RTC 表现相当，两者均明显快于同步推理基线（synchronous baseline 在 chunk 之间有明显停顿）。Success rate 误差棒为 68% Wilson score interval，duration 误差棒为 ±1 SEM。Figure 5: Real-world results — success rate and duration on the two tasks. Training-time RTC performs on par with inference-time RTC, and both are clearly faster than the synchronous inference baseline (the synchronous baseline pauses noticeably between chunks). Success rate error bars are 68% Wilson score intervals; duration error bars are ±1 SEM.
见 Figure 5（与 RTC 相当）See Figure 5 (on par with RTC)
见 Figure 5（与 RTC 相当）See Figure 5 (on par with RTC)
108ms（d≈5）108ms (d≈5)
无None
注：论文图表（Figure 5）以柱状图形式呈现具体成功率和时长，原文未在正文以数字明确列出精确百分比。实验结论为 "training-time RTC maintains both task performance and speed parity with inference-time RTC while being computationally cheaper"。Note: Figure 5 of the paper presents the concrete success rates and durations as bar charts; the text never lists the exact percentages numerically. The experimental conclusion is "training-time RTC maintains both task performance and speed parity with inference-time RTC while being computationally cheaper".
关键结论Key takeaways
Training-time RTC 与 inference-time RTC 在两个真实任务上性能持平，同时减少了推理延迟（108ms vs. 135ms）。Training-time RTC matches inference-time RTC on both real-world tasks while reducing inference latency (108ms vs. 135ms).
两种 RTC 变体均明显优于同步推理基线（后者在 chunk 之间存在明显停顿）。Both RTC variants clearly outperform the synchronous inference baseline, which pauses noticeably between chunks.
仿真结果表明：推理延迟越大，training-time RTC 相对 inference-time RTC 的优势越显著（延迟 ≥ 2 时 training-time RTC 更优）。Simulation shows that the larger the inference delay, the more pronounced the advantage of training-time RTC over inference-time RTC (training-time RTC is better once the delay is ≥ 2).
04局限性Limitations
Note:以下局限性均为论文作者在 Discussion 节中明确陈述（stated）。All limitations below are explicitly stated by the authors of the paper in the Discussion section.
灵活性不及 inference-time RTC（Flexibility）Less flexible than inference-time RTC (Flexibility)
如原文所述："training-time RTC is fundamentally less flexible than inference-time RTC; it only supports conditioning on a 'hard' action prefix corresponding to the inference delay, whereas inference-time RTC can 'softly' incorporate additional actions beyond the prefix."
即 training-time RTC 只能处理固定 d 步的 hard prefix，无法像 inference-time RTC 那样对 prefix 之外的重叠动作进行软性加权引导。As the paper puts it: "training-time RTC is fundamentally less flexible than inference-time RTC; it only supports conditioning on a 'hard' action prefix corresponding to the inference delay, whereas inference-time RTC can 'softly' incorporate additional actions beyond the prefix." That is, training-time RTC can only handle a hard prefix of a fixed d steps, and cannot apply the soft weighted guidance over overlapping actions beyond the prefix the way inference-time RTC does.
需要预先估计推理延迟分布（Delay Distribution Estimation）Requires estimating the inference delay distribution in advance (Delay Distribution Estimation)
Training-time RTC 需要在训练时选择合适的延迟分布（本文为 Uniform[0, 10]），该分布必须与实际部署时的推理延迟匹配。若实际延迟超出训练分布范围，性能可能下降。Training-time RTC must choose an appropriate delay distribution at training time (Uniform[0, 10] in this paper), and that distribution has to match the inference delay seen at deployment. If the actual delay falls outside the training distribution, performance may degrade.
不支持 prefix 之外重叠动作的软掩码（Soft Masking）No soft masking of overlapping actions beyond the prefix (Soft Masking)
Inference-time RTC 可以利用来自前一个 chunk 的所有 H−s 个重叠动作（不只是前 d 个）进行 inpainting，而 training-time RTC 仅使用前 d 个 action 作为 hard prefix，丢弃了额外的重叠信息。Inference-time RTC can use all H−s overlapping actions from the previous chunk (not just the first d) for inpainting, whereas training-time RTC uses only the first d actions as a hard prefix and discards the extra overlap information.
