Playable Environments:
Video Manipulation in Space and Time

01 动机Motivation

视频记录的是静态的事件副本。人们希望能像观看网球比赛时那样，随意改变球员的动作、调整相机角度、切换场地风格——这需要在三维空间中理解并重建场景，同时提供直观的交互界面，如同操控电子游戏。现有方法各有局限：A video is a static copy of an event. One would like to freely change a player's actions, adjust the camera angle and switch the court style while watching a tennis match—which requires understanding and reconstructing the scene in 3D space while offering an intuitive interaction interface, as in a video game. Existing methods each have their limitations:

• Novel View Synthesis（如 NeRF）：支持静态场景的自由视点渲染，但难以处理动态对象、不支持用户交互。Novel View Synthesis (e.g., NeRF): supports free-viewpoint rendering of static scenes, but struggles with dynamic objects and offers no user interaction.
• 视频生成方法（如 MoCoGAN、SAVP）：可预测未来帧或动画化对象，但通常缺乏相机控制和多对象支持，不考虑三维本质。Video generation methods (e.g., MoCoGAN, SAVP): can predict future frames or animate objects, but usually lack camera control and multi-object support, and disregard the 3D nature of the scene.
• Playable Video Generation（PVG，如 CADDY）：可学习离散动作控制视频，但仅限于二维、单对象、有限相机运动场景。Playable Video Generation (PVG, e.g., CADDY): learns discrete actions that control video, but is restricted to 2D, single-object scenes with limited camera motion.

"We call these representations Playable Environments (PE)... [they] encapsulate and extend representations built by several prior image or video manipulation methods."

论文定义了可玩环境需满足的六项核心特性：The paper defines six core properties that a playable environment must satisfy:

02 方法Method

框架采用编码器-解码器结构，核心由两个模块构成：Synthesis Module（合成模块）负责从场景状态渲染图像；Action Module（动作模块）负责在状态空间中学习并预测用户动作。训练分两阶段进行，全程仅依赖重建损失，无需人工动作标注。The framework follows an encoder-decoder design built from two modules: the Synthesis Module renders images from the scene state, while the Action Module learns and predicts user actions in the state space. Training proceeds in two stages and relies on reconstruction losses alone, requiring no manual action labels.

Synthesis Module：组合式非刚性 NeRFSynthesis Module: Compositional Non-Rigid NeRF

合成模块以 NeRF 为基础架构，实现相机控制 ⑵。每个对象由独立的 MLP V 参数化的 feature field 建模，各 field 以对象位置为中心、受 bounding volume 约束，从而支持多对象建模 ⑶。The synthesis module builds on NeRF as its backbone, which delivers camera control ⑵. Each object is modeled by a feature field parameterized by a separate MLP V; every field is centered on the object position and constrained by a bounding volume, which enables multi-object modeling ⑶.

为处理可变形对象（如人体）⑷，引入 ray bending network B：给定姿态描述子 π 与采样点 x_p，通过 B 将其映射到正则空间坐标 x̃_p = x_p + B(x_p, π_t)，从而在正则空间中编码对象几何。To handle deformable objects such as human bodies ⑷, a ray bending network B is introduced: given a pose descriptor π and a sample point x_p, B maps it to the canonical-space coordinate x̃_p = x_p + B(x_p, π_t), thereby encoding object geometry in canonical space.

为建模外观多样性 ⑸，受 AdaIN 启发，在 V 的 feature prediction branch 中嵌入风格调制层：h̃_t = γ(w_t) h_t + β(w_t)，其中 γ, β 为可训练线性层，风格码 w_t 只调制颜色特征而不影响几何。To model appearance diversity ⑸, and inspired by AdaIN, style modulation layers are embedded in the feature prediction branch of V: h̃_t = γ(w_t) h_t + β(w_t), where γ, β are trainable linear layers and the style code w_t modulates only color features without affecting geometry.

Action Module：状态空间离散动作学习Action Module: Discrete Action Learning in State Space

动作模块由 Action Network A 和 Dynamics Network R 组成。A 给定相邻状态 (s_t, s_{t+1})，输出离散动作 a_t ∈ {1,...,K} 与 action variability embedding v_t；R（LSTM）以 (s_t, a_t, v_t) 为输入自回归预测下一状态。The action module consists of an Action Network A and a Dynamics Network R. Given adjacent states (s_t, s_{t+1}), A outputs a discrete action a_t ∈ {1,...,K} together with an action variability embedding v_t; R (an LSTM) takes (s_t, a_t, v_t) as input and autoregressively predicts the next state.

为使动作与相机朝向一致（符合游戏直觉），Dynamics Network 预测相机坐标系下的位移 Δ，再通过旋转矩阵 M 转换：x̂_{t+1} = x_t + MΔ。To keep actions consistent with the camera orientation (matching game intuition), the Dynamics Network predicts the displacement Δ in the camera coordinate system and converts it through a rotation matrix M: x̂_{t+1} = x_t + MΔ.

训练策略Training Strategy

阶段一（合成模块）：用感知损失（VGG perceptual loss）+ L2 像素重建损失训练编码器与合成模块。为避免风格 w 和姿态 π 解耦失败，在每个序列的时间维度上打乱 w 的顺序再送入合成模块。Stage 1 (synthesis module): the encoder and the synthesis module are trained with a perceptual loss (VGG perceptual loss) plus an L2 pixel reconstruction loss. To prevent the style w and the pose π from failing to disentangle, the order of w is shuffled along the temporal dimension of each sequence before it is fed to the synthesis module.

阶段二（动作模块）：联合优化四项损失——重建损失 L_rec、信息论动作学习损失 L_act（最大化互信息）、Δ-MSE 软损失 L_Δ（同类动作应对应相似位移）、以及对抗 Temporal Discriminator D（判别真实/重建状态序列，促使动作生成真实肢体动作）。Stage 2 (action module): four losses are optimized jointly—the reconstruction loss L_rec, the information-theoretic action learning loss L_act (maximizing mutual information), the soft Δ-MSE loss L_Δ (identical actions should correspond to similar displacements), and an adversarial Temporal Discriminator D (which tells real from reconstructed state sequences, pushing actions to generate realistic limb motion).

03 实验Experiments

在三个数据集上评估：Tennis（43场网球赛，12h真实视频）、Minecraft（1h合成视频，宽视角运动）、Minecraft Camera（相机运动序列，提供 novel view 真值）；另采用 Static Tennis（PVG基准）做方法对比。Evaluation is carried out on three datasets: Tennis (43 tennis matches, 12h of real video), Minecraft (1h of synthetic video with wide-viewpoint motion) and Minecraft Camera (camera motion sequences providing novel view ground truth); Static Tennis (the PVG benchmark) is additionally used for method comparison.

与 PVG 方法对比（Static Tennis 数据集）Comparison with PVG Methods (Static Tennis Dataset)

在 playable video generation 设定（无显式相机控制）下，与 MoCoGAN、SAVP、CADDY 等方法对比：Under the playable video generation setting (no explicit camera control), the method is compared against MoCoGAN, SAVP and CADDY:

本方法在动作质量指标（Δ-MSE、Δ-Acc）和视频重建质量（LPIPS、MDR）上均显著优于 CADDY，动作与球员运动的一致性大幅提升。The proposed method clearly outperforms CADDY on both the action quality metrics (Δ-MSE, Δ-Acc) and the video reconstruction quality (LPIPS, MDR), with a large gain in the consistency between actions and player motion.

与 Baselines 对比（Tennis + Minecraft Camera）Comparison with Baselines (Tennis + Minecraft Camera)

在完整 PE 设定下（多对象、真实相机运动），与基于 CADDY 的多种变体比较。本方法在 Tennis 上 LPIPS=0.181、FVD=485、Δ-MSE=0.293%、Δ-Acc=95.7%、MDR=4.84%，全面超越所有基线；在 Minecraft Camera（相机控制）上 LPIPS=0.242、FID=29.2，而最佳基线 FID≥244——本方法能真正从新视角合成场景，而 CADDY 各变体由于缺乏显式相机模型，无法处理 novel view 生成。Under the full PE setting (multiple objects, real camera motion), the method is compared with several CADDY-based variants. It reaches LPIPS=0.181, FVD=485, Δ-MSE=0.293%, Δ-Acc=95.7% and MDR=4.84% on Tennis, surpassing every baseline across the board; on Minecraft Camera (camera control) it reaches LPIPS=0.242 and FID=29.2, whereas the best baseline has FID≥244—the proposed method genuinely synthesizes the scene from novel viewpoints, while the CADDY variants cannot handle novel view generation because they lack an explicit camera model.

用户研究User Study

在 Tennis 数据集上的用户研究中，采用 Fleiss' kappa 衡量动作一致性：本方法 kappa=0.444，最佳基线 kappa=0.353，说明本方法学到的动作空间更清晰、更易于用户一致识别。In a user study on the Tennis dataset, action consistency is measured with Fleiss' kappa: the proposed method reaches kappa=0.444 against kappa=0.353 for the best baseline, indicating that the learned action space is clearer and easier for users to recognize consistently.

消融实验Ablation Studies

合成模块消融（Minecraft）：逐步加入 Multi-object（⑶）、非刚性变形 π（⑷）、风格调制 w（⑸）、Feature Renderer F（⑹）。结果表明：deformation 和 style modulation 对精确合成均不可缺少（⑷⑸），但单独使用会因定标噪声导致模糊；引入 Feature Renderer 后，LPIPS 从 0.350 降至 0.193，FID 从 61.0 降至 16.5，FVD 从 465 降至 289，显著恢复清晰度。Synthesis module ablation (Minecraft): Multi-object (⑶), non-rigid deformation π (⑷), style modulation w (⑸) and the Feature Renderer F (⑹) are added step by step. The results show that deformation and style modulation are both indispensable for accurate synthesis (⑷⑸), yet on their own they yield blur caused by calibration noise; adding the Feature Renderer lowers LPIPS from 0.350 to 0.193, FID from 61.0 to 16.5 and FVD from 465 to 289, restoring sharpness substantially.

动作模块消融：去掉 Temporal Discriminator D 会导致 FVD 显著上升（动作序列中肢体运动不真实）；Δ-MSE 损失 L_Δ 对动作空间质量有正向影响；相机相对坐标预测（Rel.）对于生成视角一致的动作效果至关重要。Action module ablation: removing the Temporal Discriminator D raises FVD markedly (limb motion within the action sequences becomes unrealistic); the Δ-MSE loss L_Δ has a positive effect on the quality of the action space; and predicting in camera-relative coordinates (Rel.) is crucial for producing viewpoint-consistent actions.

04 局限性Limitations