Playable Video Generation

01 动机 MotivationMotivation

人类从很小就能识别关键物体及其与环境的交互方式——我们能够在从未被明确告知可能动作的情况下，自发地理解和预判视频中的行为。然而，现有视频生成方法要么需要帧级动作标注（限制于游戏或机器人场景），要么仅能用单个标签控制整段视频（无法逐帧实时交互）。From an early age humans can identify the key objects in a scene and how they interact with the environment: we spontaneously understand and anticipate the behaviour in a video without ever being explicitly told which actions are possible. Existing video generation methods, however, either require frame-level action annotations (which restricts them to games or robotic settings) or can only control a whole video with a single label (making frame-by-frame real-time interaction impossible).

"We aim at allowing a user to control the generated video by selecting a discrete action at every time step as when playing a video game."

问题的难点What Makes It Hard

PVG 的核心挑战有两个：（1）动作的语义一致性——在无监督前提下学习到真正有意义的离散动作，而非任意划分；（2）真实视频生成——根据用户提供的离散动作，生成高质量且连贯的视频帧。现实视频中的随机性（如摄像机抖动、光照变化等）进一步增加了用离散标签完整描述帧间过渡的难度。PVG poses two core challenges: (1) semantic consistency of the actions — learning discrete actions that are genuinely meaningful rather than an arbitrary partition, and doing so without supervision; (2) realistic video generation — producing high-quality, coherent frames conditioned on the discrete action supplied by the user. The stochasticity of real-world videos (camera shake, illumination changes, etc.) further increases the difficulty of fully describing frame-to-frame transitions with a discrete label.

02 方法 MethodMethod

CADDY（Clustering for Action Decomposition and DiscoverY）是一个端到端自监督框架，由四个主模块组成：编码器 E、动作网络 A、循环动力学网络 R、解码器 D。整体以视频帧重建损失为主要驱动，迫使动作网络学到语义一致的离散表示。CADDY (Clustering for Action Decomposition and DiscoverY) is an end-to-end self-supervised framework made of four main modules: the encoder E, the action network A, the recurrent dynamics network R and the decoder D. It is driven mainly by a video-frame reconstruction loss, which forces the action network to learn semantically consistent discrete representations.

概率动作网络（Probabilistic Action Network）Probabilistic Action Network

动作网络 A 的目标是将帧间过渡分解为：（1）离散动作标签 a_t（高层语义）；（2）连续变分嵌入 v_t（捕捉每个动作执行方式的细节差异）。具体做法是：先用动作状态子网络 A_s 预测当前帧特征 f_t 的动作嵌入 e_t 服从高斯分布；再将前后两帧嵌入之差 d_t = e_{t+1} − e_t 送入分类层；分类层采用 Gumbel-Softmax 实现端到端可微分的离散化，输出动作概率 p_t 及离散标签 a_t。v_t 定义为 d_t 与其对应动作簇质心 c_k 的差，强制 d_t 无法完全从 v_t 中恢复，从而迫使网络学习真正的离散动作。The action network A decomposes a frame-to-frame transition into (1) a discrete action label a_t (high-level semantics) and (2) a continuous variational embedding v_t (capturing the fine-grained differences in how each action is carried out). Concretely, an action-state subnetwork A_s first predicts a Gaussian-distributed action embedding e_t for the current frame feature f_t; the difference between the embeddings of two consecutive frames, d_t = e_{t+1} − e_t, is then fed to a classification layer that uses Gumbel-Softmax for end-to-end differentiable discretisation, yielding the action probabilities p_t and the discrete label a_t. v_t is defined as the difference between d_t and the centroid c_k of its action cluster, so that d_t cannot be fully recovered from v_t and the network is forced to learn genuinely discrete actions.

训练目标Training Objective

总损失函数由五项组成：The total loss consists of five terms:

• 帧重建损失 L_rec^x：基于 VGG-19 感知损失（perceptual loss），在多分辨率金字塔上计算，辅以 L1 损失；Frame reconstruction loss L_rec^x: a VGG-19 perceptual loss computed over a multi-resolution pyramid, complemented by an L1 loss;
• 特征重建损失 L_rec^f：最小化输入帧特征与重建帧特征的欧氏距离，保证语义信息保真；Feature reconstruction loss L_rec^f: minimises the Euclidean distance between the features of the input frame and those of the reconstructed frame, so that semantic information is preserved;
• 动作匹配损失 L_act：最大化原始序列与重建序列所预测动作概率之间的互信息（mutual information），同时防止动作空间退化（所有帧预测同一动作）；Action matching loss L_act: maximises the mutual information between the action probabilities predicted for the original and for the reconstructed sequence, while preventing the action space from degenerating (all frames predicting the same action);
• 动作变分损失 L_rec^a：用 KL 散度约束 v_t 分布的一致性；Action variational loss L_rec^a: uses a KL divergence to enforce the consistency of the distribution of v_t;
• KL 正则化 L_KL：将 d_t 的后验对齐到标准高斯先验。KL regularisation L_KL: aligns the posterior of d_t with a standard Gaussian prior.

03 实验 ExperimentsExperiments

在三个数据集上评测：BAIR（机器人推物，约44K段30帧视频，256×256）、Atari Breakout（Rainbow DQN采集，1407段约32帧，160×210）、Tennis（YouTube网球比赛，约900段，256×96）。基线方法包括 MoCoGAN、MoCoGAN+、SAVP、SAVP+（增容量版）、SRVP。评估指标涵盖视频质量（LPIPS、FID、FVD）、动作空间质量（Δ-MSE、Δ-Acc）和动作条件生成质量（ADD、MDR）。Evaluation is carried out on three datasets: BAIR (robot pushing, about 44K sequences of 30 frames, 256×256), Atari Breakout (collected with a Rainbow DQN agent, 1407 sequences of about 32 frames, 160×210) and Tennis (YouTube tennis matches, about 900 sequences, 256×96). The baselines are MoCoGAN, MoCoGAN+, SAVP, SAVP+ (increased-capacity version) and SRVP. The metrics cover video quality (LPIPS, FID, FVD), action-space quality (Δ-MSE, Δ-Acc) and action-conditioned generation quality (ADD, MDR).

BAIR 数据集对比（Table 2）Comparison on BAIR (Table 2)

CADDY 相比最佳基线在 Δ-MSE 上提升 26.1%，在 Δ-Acc 上提升 24.2%，展示出更强的离散动作学习能力。SRVP 因预测整个测试集只有单一动作类别（退化行为），Δ-MSE 和 Δ-Acc 均失去意义。Against the best baseline, CADDY improves Δ-MSE by 26.1% and Δ-Acc by 24.2%, showing a much stronger ability to learn discrete actions. Since SRVP predicts a single action class over the whole test set (degenerate behaviour), its Δ-MSE and Δ-Acc are meaningless.

Atari Breakout 数据集对比（Table 3）Comparison on Atari Breakout (Table 3)

CADDY 在 Atari Breakout 上取得最优性能：平均 ADD 仅 7.29 像素，MDR 仅 2.70%，说明学到的动作空间与用户控制平台的运动高度一致。CADDY attains the best performance on Atari Breakout: the average ADD is only 7.29 pixels and the MDR only 2.70%, which shows that the learned action space is highly consistent with the motion of the user-controlled platform.

Tennis 数据集对比（Table 4）Comparison on Tennis (Table 4)

Tennis 数据集上，CADDY 在绝大多数指标取得最优，MDR 仅 1.01%，ADD 仅 8.85 像素，显著优于基线，说明能准确生成并追踪球员位置。On the Tennis dataset CADDY is best on the vast majority of metrics, with an MDR of only 1.01% and an ADD of only 8.85 pixels, clearly outperforming the baselines and indicating that the player position is generated and tracked accurately.

消融实验（Table 1，BAIR 数据集）Ablation Study (Table 1, BAIR)

消融实验证明三个核心组件（Gumbel-Softmax、动作变分嵌入 v_t、互信息损失 L_act）缺一不可：去掉 G.S. 则学到连续而非离散动作；单独使用 L_act 优化会与重建目标冲突导致质量下降；v_t 和 L_act 必须配合使用，才能避免网络将所有信息编码进连续变量而忽略离散动作。The ablation shows that the three core components (Gumbel-Softmax, the action variational embedding v_t and the mutual-information loss L_act) are all indispensable: without G.S. the learned actions are continuous rather than discrete; optimising L_act on its own conflicts with the reconstruction objective and degrades quality; v_t and L_act must be used together, otherwise the network encodes all the information in the continuous variable and ignores the discrete actions.

用户研究（Tennis 数据集）User Study (Tennis)

作者进行了用户研究，要求23名用户从生成序列中辨认出执行的动作（左/右/前进/后退/击球/静止）。CADDY 获得最高的 Fleiss' kappa 一致性（0.469），而基线方法（MoCoGAN: −3.15×10⁻³，MoCoGAN+: −2.84×10⁻³，SAVP: 0.0718，SAVP+: −1.97×10⁻³）均无法达到有意义的一致性，说明 CADDY 学到的动作具有稳定的语义含义。In a user study, 23 users were asked to recognise the action performed in a generated sequence (left / right / forward / backward / hit / stay). CADDY obtains the highest Fleiss’ kappa agreement (0.469), whereas the baselines (MoCoGAN: −3.15×10⁻³, MoCoGAN+: −2.84×10⁻³, SAVP: 0.0718, SAVP+: −1.97×10⁻³) reach no meaningful agreement at all, showing that the actions learned by CADDY carry stable semantics.

04 局限性 LimitationsLimitations