GameNGen 是史上第一个完全由神经模型驱动的游戏引擎，以单块 TPU 实现 20 FPS 的实时互动游玩。
该系统以经典射击游戏 DOOM 为实验平台：先用强化学习智能体收集游戏轨迹数据，再训练扩散模型进行下一帧预测。
在人类评估中，短片段的真假辨别率仅略高于随机水平，而经历 5–10 分钟游玩后更下降至随机猜测水平（50%）。GameNGen is the first game engine in history driven entirely by a neural model, delivering real-time interactive play at 20 FPS on a single TPU. The system uses the classic shooter DOOM as its testbed: a reinforcement learning agent first collects gameplay trajectory data, and a diffusion model is then trained for next-frame prediction. In human evaluation, raters tell real from generated clips only slightly above chance on short clips, and after 5–10 minutes of play their accuracy drops to chance level (50%).
2024 年 8 月 · arXivAugust 2024 · arXivICLR 2025Google Research / DeepMind / Tel Aviv University📄 arXiv:2408.14837Project Page
扩散模型Diffusion Models游戏引擎Game EngineGameNGenDOOM神经渲染Neural Rendering强化学习数据收集RL Data Collection自回归生成Autoregressive Generationreal-time interactionnoise augmentationnext-frame prediction
01动机Motivation
传统游戏引擎依赖大量手工编写的规则和渲染管线，开发成本极高。
随着生成模型能力的飞速提升，一个自然的问题随之浮现：
神经网络能否取代传统游戏引擎，直接从数据中学会模拟复杂的交互式环境？
此前的探索（如 World Models、GameGAN）已展现出可能性，但均无法支持实时互动游玩，也难以保持长时序的视觉一致性。Traditional game engines rely on large amounts of hand-written rules and rendering pipelines, making development extremely costly. As generative models grow rapidly more capable, a natural question arises: can a neural network replace the traditional game engine and learn to simulate complex interactive environments directly from data? Earlier efforts (such as World Models and GameGAN) showed this might be possible, but none supported real-time interactive play, and all struggled to keep long-horizon visual consistency.
"We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality."
图 1：GameNGen 实时游玩演示。人类玩家在 GameNGen 上以 20 FPS 游玩 DOOM。
模型在单块 TPU-v5 上运行，根据玩家输入的动作序列实时生成每一帧画面，无需任何传统游戏引擎代码。Figure 1: GameNGen real-time gameplay demo. A human player plays DOOM on GameNGen at 20 FPS. The model runs on a single TPU-v5 and generates every frame in real time from the player's action sequence, without any traditional game engine code.
20 FPS单块 TPU-v5 实时推理速度Real-time inference speed on a single TPU-v5
29.4next-frame PSNR（相当于 JPEG 有损压缩）next-frame PSNR (on par with lossy JPEG compression)
50%5–10 分钟游玩后人类辨别真假的准确率Human accuracy at telling real from generated after 5–10 minutes of play
4步4 stepsDDIM 去噪步数（兼顾质量与速度）DDIM denoising steps (balancing quality and speed)
02方法Method
GameNGen 分两个阶段训练：第一阶段，训练一个强化学习（PPO）智能体玩 DOOM 并录制全程轨迹作为数据集；
第二阶段，在 Stable Diffusion v1.4 基础上微调扩散模型，以历史帧序列和动作序列为条件生成下一帧，
配合 noise augmentation（噪声增广）和解码器微调来保证长时序稳定性。GameNGen is trained in two stages. Stage 1 trains a reinforcement learning (PPO) agent to play DOOM and records its full trajectories as the dataset. Stage 2 fine-tunes a diffusion model on top of Stable Diffusion v1.4, conditioning on a sequence of history frames and actions to generate the next frame, with noise augmentation and decoder fine-tuning to ensure long-horizon stability.
图 2：GameNGen 训练流程总览。
左侧：阶段一，PPO 智能体在真实 DOOM 引擎中游玩 5000 万步，收集动作—帧对作为训练数据。
右侧：阶段二，基于 Stable Diffusion v1.4 的扩散模型，以 64 帧历史画面（约 3.2 秒）和对应动作
嵌入为条件，预测下一帧的潜在表示；通过 DDIM 4步采样实现实时推理。Figure 2: Overview of the GameNGen training pipeline. Left: Stage 1, a PPO agent plays 50000000 steps in the real DOOM engine, collecting action-frame pairs as training data. Right: Stage 2, a diffusion model built on Stable Diffusion v1.4 conditions on 64 frames of visual history (about 3.2 seconds) and the corresponding action embeddings to predict the latent representation of the next frame; real-time inference is achieved with 4-step DDIM sampling.
阶段一：数据收集（RL Agent Training）Stage 1: Data Collection (RL Agent Training)
采用 PPO 算法训练智能体，共执行 5000 万个环境步骤。
智能体在训练和评估过程中的全部动作与观测均被保存，形成包含 7000 万条轨迹的训练集。
使用智能体（而非随机策略）收集数据的好处在于：智能体会主动探索游戏中的难关和过场，使训练数据分布更接近人类玩法。
消融实验显示，在中等难度关卡中，智能体数据训练的模型在 3 秒自回归生成后 PSNR 为 20.21，
而随机策略仅为 16.50。The agent is trained with PPO for a total of 50000000 environment steps. All actions and observations recorded during training and evaluation are saved, forming a training set of 70000000 trajectories. Collecting data with an agent rather than a random policy helps because the agent actively explores the game's hard sections and transitions, bringing the training distribution closer to human play. Ablations show that on medium-difficulty levels the model trained on agent data reaches a PSNR of 20.21 after 3 seconds of autoregressive generation, versus only 16.50 for the random policy.
阶段二：扩散模型微调Stage 2: Diffusion Model Fine-tuning
以 Stable Diffusion v1.4 为主干，将文本条件替换为动作嵌入（用 cross-attention 注入），
并将编码后的历史帧拼接至去噪 latent 的通道维度。具体设计要点如下：Stable Diffusion v1.4 serves as the backbone: the text condition is replaced by action embeddings injected through cross-attention, and the encoded history frames are concatenated along the channel dimension of the denoising latent. The key design choices are as follows:
Context length = 64 帧（约 3.2 秒历史），实验表明更长的历史收益递减。Context length = 64 frames (about 3.2 seconds of history); experiments show diminishing returns from a longer history.
Noise Augmentation（噪声增广）：训练时向历史帧加入随机高斯噪声（最大幅度 0.7），并用可学习的嵌入向量表示噪声级别，共 10 个 bucket；这迫使模型学会"纠正"上下文帧中积累的误差，从根本上抑制自回归漂移。Noise Augmentation: During training, random Gaussian noise (maximum magnitude 0.7) is added to the history frames, and the noise level is carried by a learnable embedding over 10 buckets; this forces the model to learn to "correct" the errors accumulated in its context frames, suppressing autoregressive drift at its root.
解码器微调（Latent Decoder Fine-tuning）：在冻结 U-Net 的情况下，用 MSE 损失单独微调 VAE 解码器，以提升像素级保真度。Latent Decoder Fine-tuning: With the U-Net frozen, the VAE decoder alone is fine-tuned with an MSE loss to improve pixel-level fidelity.
推理：DDIM 采样 4 步，配合 classifier-free guidance（权重 1.5，仅对历史帧侧），在 TPU-v5 上实现 20 FPS。Inference: DDIM sampling with 4 steps, together with classifier-free guidance (weight 1.5, applied only to the history-frame branch), reaches 20 FPS on a TPU-v5.
图 3：噪声增广对自回归漂移的抑制效果。
在玩家静止不动的场景下，连续生成 50 帧。
上行（无噪声增广）：约 20–30 步后画面迅速退化、出现明显视觉伪影；
下行（有噪声增广）：画面质量始终稳定。Figure 3: How noise augmentation suppresses autoregressive drift. 50 frames are generated in a row in a scene where the player stands still. Top row (no noise augmentation): the image degrades rapidly after roughly 20–30 steps, with obvious visual artifacts. Bottom row (with noise augmentation): image quality stays stable throughout.
训练细节Training Details
模型在 128 块 TPU-v5e 上训练 70 万步，batch size 128（U-Net）和 2048（解码器），
使用 Adafactor 优化器，学习率 2e-5，常数调度。
输入分辨率为 320×240，padding 至 320×256，损失函数采用 velocity parameterization。The model is trained for 700000 steps on 128 TPU-v5e chips, with batch size 128 (U-Net) and 2048 (decoder), using the Adafactor optimizer at a constant learning rate of 2e-5. Inputs are 320×240, padded to 320×256, and the loss uses velocity parameterization.
03实验Experiments
实验在 DOOM（ViZDoom 框架）上展开，评估指标涵盖图像质量（PSNR、LPIPS）、视频质量（FVD）以及人类辨别测试。
基线包括 World Models 和 GameGAN（但数据集不同，不可直接对比）。
推理硬件为单块 TPU-v5。Experiments are run on DOOM (the ViZDoom framework), with metrics covering image quality (PSNR, LPIPS), video quality (FVD) and a human discrimination test. Baselines include World Models and GameGAN, although they use different datasets and are not directly comparable. Inference hardware is a single TPU-v5.
人类辨别测试（Human Evaluation）Human Discrimination Test (Human Evaluation)
使用 10 名评估者，共 130 个随机片段，对不同时长的真实游戏与 GameNGen 生成画面进行辨别。10 raters judged a total of 130 random clips, distinguishing real gameplay from GameNGen-generated footage across different clip lengths.
片段时长Clip length
人类选择真实游戏的比例Share picking the real game
1.6 秒1.6 s
58%
3.2 秒3.2 s
60%
5–10 分钟游玩后After 5–10 minutes of play
50%（随机水平）50% (chance level)
即便面对数分钟的完整游玩录像，人类仍无法可靠地区分真实与生成内容，
充分证明 GameNGen 在视觉保真度方面已达到令人信服的水平。Even when shown several minutes of complete gameplay recordings, humans still cannot reliably separate real from generated content, strong evidence that GameNGen has reached a convincing level of visual fidelity.
图 4：真实帧（Ground Truth）与 GameNGen 预测帧对比。
左侧为玩家的历史上下文帧，右侧对比展示真实下一帧与模型生成帧。
在场景几何、光照和敌人位置等关键细节上，生成帧与真实帧高度一致。Figure 4: Ground-truth frames compared with GameNGen predictions. The left shows the player's past context frames; the right compares the true next frame with the model's generated frame. On key details such as scene geometry, lighting and enemy positions, the generated frames match the real ones closely.
数据策略消融（Agent vs. Random Policy）Data Strategy Ablation (Agent vs. Random Policy)
难度Difficulty
策略Policy
PSNR ↑
LPIPS ↓
Easy
Agent
20.94±0.76
0.48±0.01
Easy
Random
20.20±0.83
0.48±0.01
Medium
Agent
20.21±0.36
0.50±0.01
Medium
Random
16.50±0.41
0.59±0.01
Hard
Agent
17.51±0.35
0.60±0.01
Hard
Random
15.39±0.43
0.61±0.00
在中等和高难度关卡中，以强化学习智能体数据训练的模型显著优于随机策略数据，
说明高质量、覆盖全面的数据对模型泛化至关重要。On medium- and hard-difficulty levels, models trained on reinforcement learning agent data clearly outperform those trained on random-policy data, showing that high-quality, broadly covering data is critical for generalization.
图 5：自回归轨迹中的 PSNR 变化曲线。
对比强化学习智能体数据与随机策略数据训练的模型，在人类实际游玩的 3 秒自回归生成过程中，
PSNR 随时间的变化趋势。智能体数据训练的模型能在整个轨迹上保持更高且更稳定的质量。Figure 5: PSNR along an autoregressive trajectory. Models trained on reinforcement learning agent data and on random-policy data are compared over 3 seconds of autoregressive generation during real human play, showing how PSNR evolves over time. The model trained on agent data keeps higher and more stable quality across the whole trajectory.
04局限性Limitations
说明：以下三点均为作者在论文中明确陈述的局限性（stated by the authors）。Note: All three points below are limitations explicitly stated by the authors in the paper.
有限的历史记忆（Limited Memory）Limited Memory
模型仅能访问约 3.2 秒的历史上下文（64 帧），而真实游戏状态可能需要更长的记忆才能维持一致性。
作者指出："The model only has access to a little over 3 seconds of history"，
模型极可能依赖强启发式来弥补，但在需要长期记忆的场景（如打开过的门、拾起的物品）中可能出现错误。The model can only access about 3.2 seconds of history context (64 frames), whereas the true game state may require far longer memory to stay consistent. The authors note: "The model only has access to a little over 3 seconds of history"; the model most likely leans on strong heuristics to compensate, but errors can appear in situations that demand long-term memory, such as doors already opened or items already picked up.
智能体在训练中并未探索游戏的所有关卡、位置和交互，导致模型在这些未见过的场景中可能产生错误行为。
作者明确表示："the trained agent does not explore all of the game's locations and interactions, leading to erroneous behavior in those cases."During training the agent does not explore all of the game's levels, locations and interactions, so the model may behave incorrectly in those unseen situations. The authors state explicitly: "the trained agent does not explore all of the game's locations and interactions, leading to erroneous behavior in those cases."
无法用于创作新游戏（Game Creation Limitations）Cannot Be Used to Create New Games (Game Creation Limitations)
现阶段 GameNGen 是针对特定游戏（DOOM）的模拟器，无法像传统游戏引擎那样用于开发全新游戏内容。
作者坦言："We are not able to easily produce new games with GameNGen"，
这意味着其应用场景目前仍局限于已有游戏的模拟与研究。At this stage GameNGen is a simulator for one specific game (DOOM) and cannot be used, the way a traditional game engine is, to develop entirely new game content. The authors concede: "We are not able to easily produce new games with GameNGen"; its applications remain limited to simulating and studying existing games.
