GameFactory: Creating New Games with Generative Interactive Videos

01 动机 MotivationMotivation

游戏开发是一项极度耗费人力的过程，生成式视频模型具备自主创作游戏内容的潜力。然而现有方法普遍面临两大核心挑战：动作可控性（精准响应键盘与鼠标输入）和场景泛化性（不局限于固定的游戏风格与场景）。Game development is an extremely labor-intensive process, and generative video models have the potential to create game content autonomously. Existing methods, however, generally face two core challenges: action controllability (responding precisely to keyboard and mouse input) and scene generalization (not being confined to a fixed game style and scene).

"Generative videos have the potential to revolutionize game development by autonomously creating new content… GameFactory tackles the critical challenge of scene-generalizable action control, which most existing methods fail to address."

现有方法的不足Shortcomings of existing methods

以往的游戏视频生成工作（如 GameNGen、DIAMOND、Genie 等）大多在固定游戏环境下训练，动作控制能力与特定游戏风格深度耦合，无法泛化到新场景。此外，人类采集数据（如 VPT 数据集）存在严重的行为偏差：前进键（W）出现频率高达 50.11%，而后退键仅占 0.32%，导致模型难以学到均匀分布的动作控制能力。Prior work on game video generation (GameNGen, DIAMOND, Genie, etc.) is mostly trained in a fixed game environment, so its action control is deeply coupled with a specific game style and cannot generalize to new scenes. Moreover, human-collected data (such as the VPT dataset) suffers from severe behavioral bias: the forward key (W) occurs with a frequency as high as 50.11%, while the backward key accounts for only 0.32%, making it hard for the model to learn uniformly distributed action control.

02 方法 MethodMethod

GameFactory 的核心创新在于三个相互配合的模块：动作控制模块（精准处理键盘与鼠标输入）、自回归长视频生成（支持无限长度交互视频）、以及style-action 解耦策略（通过多阶段训练使动作控制脱离游戏风格绑定）。The core innovation of GameFactory lies in three mutually supporting modules: an action control module (handling keyboard and mouse input precisely), autoregressive long-video generation (supporting interactive video of unlimited length), and a style-action decoupling strategy (multi-phase training that frees action control from being bound to the game style).

动作控制模块（Action Control Module）Action Control Module

将动作控制集成到基于 transformer 的 latent video diffusion 模型中：Action control is integrated into a transformer-based latent video diffusion model:

• 时序对齐（Temporal Alignment）：采用滑动窗口（大小 w=3）对动作进行分组，处理视频帧与动作输入之间的粒度不匹配，同时捕捉动作效果的延迟性。Temporal Alignment: a sliding window (size w=3) groups the actions, handling the granularity mismatch between video frames and action inputs while capturing the delayed nature of action effects.
• 鼠标控制：连续运动信号经 MLP 处理后与特征拼接，再通过 temporal self-attention 建模时序依赖。消融实验显示 concatenation 优于 cross-attention（Cam: 0.0685 vs. 更高误差）。Mouse control: the continuous motion signal is processed by an MLP and concatenated with the features, after which temporal self-attention models the temporal dependency. Ablations show that concatenation outperforms cross-attention (Cam: 0.0685 vs. a higher error).
• 键盘控制：离散输入通过 cross-attention 与视频特征交互（类似文本条件化），消融实验显示 cross-attention 优于 concatenation（Cam: 0.0439 vs. 更高误差）。Keyboard control: the discrete input interacts with the video features through cross-attention (akin to text conditioning); ablations show that cross-attention outperforms concatenation (Cam: 0.0439 vs. a higher error).

自回归长视频生成（Autoregressive Long-Video Generation）Autoregressive Long-Video Generation

标准扩散模型在长序列生成上存在局限。GameFactory 采用变噪声水平策略：靠后帧携带更多噪声，靠前帧噪声较少作为条件帧。在训练时，仅对预测帧计算损失（不含条件帧），避免梯度泄漏。在推理时，系统迭代选取最近的 k+1 帧作为条件，生成 N−k 个新帧，从而支持无限长度的交互视频生成。Standard diffusion models are limited in long-sequence generation. GameFactory adopts a varying noise level strategy: later frames carry more noise, while earlier frames carry less noise and serve as condition frames. During training the loss is computed only on the predicted frames (excluding the condition frames), avoiding gradient leakage. At inference the system iteratively takes the most recent k+1 frames as conditions and generates N−k new frames, thereby supporting interactive video generation of unlimited length.

Style-Action 解耦策略（四阶段训练）Style-Action Decoupling Strategy (Four-Phase Training)

关键洞察：若在 Phase #2 同时学习风格和动作控制，动作控制能力将与特定游戏风格深度绑定，无法泛化。通过先用 LoRA 隔离风格学习（Phase #1），再单独训练动作控制（Phase #2），最后推理时丢弃 LoRA（Phase #3），动作控制模块得以保留开放域生成先验，从而实现跨场景泛化。Key insight: learning style and action control together in Phase #2 would tie action control tightly to the specific game style and prevent generalization. By first isolating style learning with LoRA (Phase #1), then training action control on its own (Phase #2), and finally discarding LoRA at inference (Phase #3), the action control module retains the open-domain generative prior and thus generalizes across scenes.

GF-Minecraft 数据集The GF-Minecraft dataset

为解决人类行为偏差问题，作者构建了 GF-Minecraft 数据集：70 小时的 Minecraft 游戏录像，采用无偏采样策略收集动作（前进/后退/左/右/跳跃等各键出现频率均衡，约 13.56%），涵盖多样化环境（森林、沙漠、雪地等），并附有文本描述。To address human behavioral bias, the authors build the GF-Minecraft dataset: 70 hours of Minecraft gameplay recordings whose actions are collected under an unbiased sampling strategy (forward/backward/left/right/jump and the other keys occur at balanced frequencies, about 13.56%), covering diverse environments (forest, desert, snowfield, etc.) and accompanied by text descriptions.

03 实验 ExperimentsExperiments

实验在 GF-Minecraft 测试集上评估动作可控性（Cam 相机姿态误差↓、Flow 光流误差↓）和生成质量（FID↓、FVD↓），并在开放域视频上评估场景泛化能力。Experiments evaluate action controllability (Cam, camera pose error↓; Flow, optical flow error↓) and generation quality (FID↓, FVD↓) on the GF-Minecraft test set, and assess scene generalization on open-domain videos.

动作控制机制消融（Table 2）Ablation on the action control mechanism (Table 2)

消融实验表明：离散键盘输入适合用 cross-attention 建模（类似文本条件化），而连续鼠标信号适合用 concatenation 方式注入特征。鼠标运动对视觉的影响强于键盘输入。The ablations show that discrete keyboard input is best modeled with cross-attention (akin to text conditioning), whereas the continuous mouse signal is best injected into the features by concatenation. Mouse motion affects the visuals more strongly than keyboard input does.

场景泛化对比（Table 3）Scene-generalization comparison (Table 3)

多阶段训练在所有指标上均显著优于单阶段基线，验证了 style-action 解耦策略的有效性。Multi-phase training significantly outperforms the one-phase baseline on every metric, confirming the effectiveness of the style-action decoupling strategy.

数据集对比：GF-Minecraft vs. VPT（Table 4）Dataset comparison: GF-Minecraft vs. VPT (Table 4)

VPT 数据集中前进键（W）出现频率为 50.11%，后退键仅 0.32%；GF-Minecraft 中各键频率均衡（约 13.56%）。使用 VPT 训练的模型无法执行跳跃、后退等罕见动作，而 GF-Minecraft 训练的模型能成功完成这些动作。In the VPT dataset the forward key (W) occurs with a frequency of 50.11% and the backward key with only 0.32%; in GF-Minecraft the key frequencies are balanced (about 13.56%). A model trained on VPT cannot perform rare actions such as jumping or moving backward, whereas the model trained on GF-Minecraft completes them successfully.

长视频生成消融（Table 6）Ablation on long-video generation (Table 6)

拓展实验：跨游戏迁移Extended experiment: cross-game transfer

GameFactory 不仅限于 Minecraft 风格。实验展示了将学到的动作控制能力迁移至赛车游戏场景的能力，验证了该框架的跨游戏类型泛化潜力。此外，论文还展示了碰撞检测行为（在 Minecraft 中遇到墙壁时停止前进）的自动涌现，以及超过 100 帧的长序列生成效果。GameFactory is not restricted to the Minecraft style. Experiments demonstrate transferring the learned action control to a racing-game scene, verifying the framework's potential for generalization across game genres. The paper further shows the automatic emergence of collision-detection behavior (stopping when hitting a wall in Minecraft) as well as long-sequence generation of more than 100 frames.

04 局限性 LimitationsLimitations