Haoxuan Che · Xuanhua He · Quande Liu · Cheng Jin · Hao Chen | HKUST / USTC / CUHK
GameGen-X 是首个专为开放世界游戏视频生成与交互控制设计的 Diffusion Transformer 模型。它通过两阶段训练——基础模型预训练与 InstructNet 指令微调——既能从文本描述中生成新颖游戏场景，又能响应键盘信号和结构化文本指令实时调整游戏内容，首次将角色交互控制与场景内容控制统一到一个视频生成框架中。GameGen-X is the first Diffusion Transformer model designed specifically for open-world game video generation and interactive control. Through two-stage training — base model pre-training plus InstructNet instruction tuning — it can both generate novel game scenes from text descriptions and adjust game content in real time in response to keyboard signals and structured text instructions, for the first time unifying character interactive control and scene content control within a single video generation framework.
game video generationdiffusion transformeropen-world gameinteractive controlInstructNetOGameData游戏视频生成game video generation指令微调instruction tuning
01动机Motivation
开放世界游戏开发耗费大量人力与时间：即便是早期原型，也需要小型团队数月的密集工作。现有神经网络方法（如 GameGAN、Genie、GameNGen）主要针对 2D 简单游戏（Pac-Man、DOOM），在复杂的次世代开放世界游戏内容生成上面临本质挑战——不仅要生成动态环境与多样角色，还要支持用户实时交互控制。Open-world game development consumes an enormous amount of manpower and time: even an early prototype requires months of intensive work by a small team. Existing neural network approaches (such as GameGAN, Genie and GameNGen) mainly target simple 2D games (Pac-Man, DOOM) and face fundamental challenges in generating content for complex next-generation open-world games — they must generate dynamic environments and diverse characters, and also support real-time interactive control by the user.
"Can a diffusion model generate and control high-quality, complex open-world video game content?"
图 1 · Teaser：GameGen-X 可以生成全新开放世界游戏视频片段，并支持玩家通过键盘和文本指令进行实时交互控制，模拟游戏游玩体验（点击图片可查看交互演示视频）。Figure 1 · Teaser:GameGen-X can generate entirely new open-world game video clips and lets players perform real-time interactive control through keyboard and text instructions, simulating the experience of playing a game (click the image to view the interactive demo video).
首个First开放世界游戏视频生成与控制模型open-world game video generation and control model
02方法Method
GameGen-X 采用两阶段训练策略：第一阶段在 OGameData-GEN 上预训练基础模型，学习游戏视频的文本-视频生成与视频续写；第二阶段引入 InstructNet，在基础模型参数冻结的前提下，通过 OGameData-INS 进行指令微调，赋予模型多模态交互控制能力。GameGen-X adopts a two-stage training strategy: the first stage pre-trains the base model on OGameData-GEN, learning text-to-video generation and video continuation for game videos; the second stage introduces InstructNet and, with the base model parameters frozen, performs instruction tuning on OGameData-INS, endowing the model with multi-modal interactive control ability.
OGameData：大规模游戏领域数据集OGameData: A Large-Scale Game-Domain Dataset
图 2 · OGameData 构建流程（human-in-the-loop）：从网络和游戏引擎采集视频，经 TransNetV2 场景切分、CLIP-AVA 美观评分、UniMatch 运动过滤、VideoCLIP 内容相似度筛选、CoTrackerV2 摄像机运动标注，最终由 GPT-4o 生成结构化标注。OGameData-GEN 提供详细场景描述，OGameData-INS 提供面向交互控制的差异化指令描述。Figure 2 · OGameData construction pipeline (human-in-the-loop):Videos are collected from the web and from game engines, then processed by TransNetV2 scene segmentation, CLIP-AVA aesthetic scoring, UniMatch motion filtering, VideoCLIP content-similarity screening and CoTrackerV2 camera-motion annotation, with GPT-4o finally producing structured captions. OGameData-GEN provides detailed scene descriptions, while OGameData-INS provides differential instruction descriptions oriented toward interactive control.
OGameData 是首个专为开放世界游戏视频生成与控制构建的大规模数据集，包含 100 万高分辨率（720p–4K）视频-文本对，来自 150+ 次世代游戏，总时长约 4000 小时，标注密度达 607 words/min——是对比数据集 MiraData（264 words/min）的 2.3 倍。数据集由两个子集组成：OGameData-GEN（用于生成预训练）和 OGameData-INS（用于指令微调与交互控制）。OGameData is the first large-scale dataset built specifically for open-world game video generation and control. It contains 1000000 high-resolution (720p–4K) video-text pairs from 150+ next-generation games, totalling about 4000 hours, with an annotation density of 607 words/min — 2.3 times that of the comparison dataset MiraData (264 words/min). The dataset consists of two subsets: OGameData-GEN (for generative pre-training) and OGameData-INS (for instruction tuning and interactive control).
两阶段训练框架Two-Stage Training Framework
图 3 · 两阶段训练框架：第一阶段（上）在 OGameData-GEN 上训练基础模型，支持文本到视频生成与视频续写；第二阶段（下）冻结基础模型，仅训练 InstructNet，在 OGameData-INS 上学习多模态交互控制（键盘输入 + 结构化文本指令 + 视频 prompt）。Figure 3 · Two-stage training framework:In the first stage (top) the base model is trained on OGameData-GEN, supporting text-to-video generation and video continuation; in the second stage (bottom) the base model is frozen and only InstructNet is trained, learning multi-modal interactive control on OGameData-INS (keyboard input + structured text instructions + video prompt).
基础模型使用 3D Spatio-Temporal VAE 将视频片段压缩至潜在表示，再由 MSDiT 进行去噪生成。MSDiT 叠加 Spatial Attention（帧内关系）、Temporal Attention（帧间相干性）和 Cross-Attention（与 T5 文本嵌入对齐）三种机制。为统一文本到视频生成与视频续写，引入 掩码机制（Masking Mechanism）：对文本到视频任务，所有帧均加噪（x=0）；对视频续写任务，前 x 帧作为上下文保持不变，仅对后续帧去噪。训练还结合了 bucket training、classifier-free guidance 和 rectified flow。The base model uses a 3D Spatio-Temporal VAE to compress video clips into latent representations, which MSDiT then denoises to generate video. MSDiT stacks three mechanisms: Spatial Attention (intra-frame relations), Temporal Attention (inter-frame coherence) and Cross-Attention (alignment with T5 text embeddings). To unify text-to-video generation and video continuation, a masking mechanism is introduced: for the text-to-video task all frames are noised (x=0); for the video continuation task the leading x frames are kept unchanged as context and only the subsequent frames are denoised. Training further combines bucket training, classifier-free guidance and rectified flow.
InstructNet：多模态交互控制InstructNet: Multi-Modal Interactive Control
图 4 · GameGen-X 架构：InstructNet 在冻结的基础模型旁并联运行，包含 N 个 InstructNet Block，每个 Block 由 Operation Fusion Expert Layer（处理键盘输入，通过 feature scaling/shifting 调制）和 Instruction Fusion Expert Layer（处理结构化文本，通过 cross-attention 融合）组成。InstructNet 输出注入基础模型的潜在表示，引导视频内容响应用户输入。Figure 4 · GameGen-X architecture:InstructNet runs in parallel beside the frozen base model and contains N InstructNet Blocks, each composed of an Operation Fusion Expert Layer (handling keyboard input, modulating through feature scaling/shifting) and an Instruction Fusion Expert Layer (handling structured text, fusing it through cross-attention). The InstructNet output is injected into the latent representation of the base model, steering the video content to respond to user input.
InstructNet 通过多模态专家（Multi-modal Experts）处理三类控制信号：InstructNet handles three kinds of control signal through multi-modal experts:
键盘输入（Keyboard Bindings）：通过 Operation Fusion Expert 以 feature scale & shift 方式调制潜在表示，主要控制角色运动方向，计算开销轻量。Keyboard bindings: modulate the latent representation through the Operation Fusion Expert in a feature scale & shift manner, mainly controlling the direction of character motion at light computational cost.
结构化文本指令（Instruction Text）：通过 Instruction Fusion Expert 以 cross-attention 方式融合，控制环境变化、光照调整等复杂场景元素。Instruction text: fused through the Instruction Fusion Expert in a cross-attention manner, controlling complex scene elements such as environment changes and lighting adjustments.
视频 Prompt（Canny 边缘、运动向量、姿态序列等）：经 3D-VAE 编码后加到潜在特征上，提供辅助引导，支持复杂动作生成或视频编辑（推理时可选）。Video prompts (Canny edges, motion vectors, pose sequences, etc.): encoded by the 3D-VAE and added to the latent features, providing auxiliary guidance that supports complex action generation or video editing (optional at inference time).
交互控制通过自回归生成实现：基于历史帧 v₁:ₓ，在控制信号 c 的条件下自回归预测后续帧 vₓ₊₁:N，形成类游戏的连续交互体验。Interactive control is realized through auto-regressive generation: given the historical frames v₁:ₓ and conditioned on the control signal c, the subsequent frames vₓ₊₁:N are predicted auto-regressively, forming a continuous, game-like interactive experience.
03实验Experiments
在自建游戏视频测试集上与 Mira、OpenSora-Plan 1.2、CogVideoX-5B、OpenSora 1.2 等开源模型对比，评估生成质量（FID、FVD、TVA、UP、MS、DD、SC、IQ）和交互控制能力（SR-C：角色动作成功率；SR-E：环境事件成功率）。On a self-built game video test set, GameGen-X is compared with open-source models such as Mira, OpenSora-Plan 1.2, CogVideoX-5B and OpenSora 1.2, evaluating generation quality (FID, FVD, TVA, UP, MS, DD, SC, IQ) and interactive control ability (SR-C: success rate of character actions; SR-E: success rate of environment events).
交互控制性能对比（Table 3）Interactive Control Performance Comparison (Table 3)
模型Model
SR-C↑
SR-E↑
UP↑
MS↑
SC↑
OpenSora-Plan 1.2
26.6%
31.7%
0.46
0.99
0.90
CogVideoX-5B
23.0%
30.3%
0.45
0.98
0.85
OpenSora 1.2
21.6%
14.2%
0.17
0.99
0.84
GameGen-X（Ours）GameGen-X (Ours)
63.0%
56.8%
0.71
0.99
0.88
GameGen-X 在角色动作成功率（SR-C：63.0% vs. 次优 26.6%）和环境事件成功率（SR-E：56.8% vs. 次优 31.7%）上大幅领先，同时 FID（252.1）和 FVD（759.8）显著优于所有对比模型。IQ 指标上稍弱，论文指出这是因为 IQ 倾向于偏好在自然场景数据集上训练的模型。GameGen-X leads by a wide margin on the success rate of character actions (SR-C: 63.0% vs. 26.6% for the runner-up) and on the success rate of environment events (SR-E: 56.8% vs. 31.7% for the runner-up), while its FID (252.1) and FVD (759.8) are markedly better than those of all compared models. It is slightly weaker on IQ; the paper notes that this is because IQ tends to favour models trained on natural-scene datasets.
定性结果Qualitative Results
图 5 · 生成展示：模型可生成多样化游戏内容——角色（刺客、法师）、环境（樱花森林、热带雨林）、动作（飞行、驾马车）和天气事件（暴风雪、大雨）。Figure 5 · Generation showcase:The model can generate diverse game content — characters (assassin, mage), environments (cherry-blossom forest, tropical rainforest), actions (flying, driving a carriage) and weather events (blizzard, heavy rain).图 7–8 · 对比：（上）与 OpenSora、OpenSora-Plan、CogVideoX 的开放域生成对比；（下）与 KLing 1.5、Luma、TongYi、Pika、RunWay 等商业产品的交互控制对比。在给定提示"从山洞深处走出并靠近水边"的场景中，仅 Pika、KLing 1.5 和 GameGen-X 正确跟随文本描述，GameGen-X 进一步保持了游戏式的摄像机逻辑与场景一致性。Figures 7–8 · Comparison:(Top) open-domain generation compared with OpenSora, OpenSora-Plan and CogVideoX; (bottom) interactive control compared with commercial products such as KLing 1.5, Luma, TongYi, Pika and RunWay. In the scene given by the prompt "walking out from deep inside a cave and approaching the water", only Pika, KLing 1.5 and GameGen-X follow the text description correctly, and GameGen-X further preserves game-style camera logic and scene consistency.
消融实验（Tables 4 & 5）Ablation Studies (Tables 4 & 5)
数据策略消融（Table 4）：使用 GameGen-X 完整数据策略（FID 252.1 / FVD 759.8 / TVA 0.87 / UP 0.82）显著优于使用 MiraData（303.7 / 1423.6 / 0.70 / 0.48）、短标注（303.8 / 1167.7 / 0.53 / 0.49）或渐进训练（294.2 / 1169.8 / 0.68 / 0.53）的变体。Data strategy ablation (Table 4):Using the full GameGen-X data strategy (FID 252.1 / FVD 759.8 / TVA 0.87 / UP 0.82) is clearly better than the variants using MiraData (303.7 / 1423.6 / 0.70 / 0.48), short captions (303.8 / 1167.7 / 0.53 / 0.49) or progressive training (294.2 / 1169.8 / 0.68 / 0.53).
InstructNet 组件消融（Table 5）：移除 InstructNet 使 SR-C 从 45.6% 降至 12.3%、SR-E 从 45.0% 降至 17.5%，用户偏好 UP 从 0.50 降至 0.16，证明 InstructNet 是实现用户偏好交互控制的核心组件。移除 Instruct Captions 或 Decomposition 也显著影响控制指标。InstructNet component ablation (Table 5):Removing InstructNet drops SR-C from 45.6% to 12.3% and SR-E from 45.0% to 17.5%, and lowers user preference UP from 0.50 to 0.16, showing that InstructNet is the core component for achieving user-preferred interactive control. Removing the instruct captions or the decomposition also markedly affects the control metrics.
04局限性Limitations
Note：Note:以下局限性均为作者在论文 Appendix E 中明确陈述（stated）。All the limitations below are explicitly stated by the authors in Appendix E of the paper.
实时生成与交互（Real-Time Generation）Real-Time Generation
扩散模型的采样过程和空间-时间自注意力机制计算代价高昂，目前无法实现游戏所需的实时交互响应。The sampling process of diffusion models and the spatial-temporal self-attention mechanism are computationally expensive, so the real-time interactive response required by games is not yet achievable.
自回归生成中的一致性漂移（Consistency in Auto-Regressive Generation）Consistency in Auto-Regressive Generation
自回归生成中错误会累积，导致长序列中的角色外观和场景连贯性下降，重新进入已生成场景时尤为明显。Errors accumulate during auto-regressive generation, degrading character appearance and scene coherence over long sequences, which is especially noticeable when re-entering an already generated scene.
模型难以独立生成快速、复杂的动作（如战斗序列），需要借助视频 prompt（Canny 边缘、运动向量等）辅助引导，限制了模型的自主生成能力。The model struggles to generate fast, complex actions (such as combat sequences) on its own and needs video prompts (Canny edges, motion vectors, etc.) as auxiliary guidance, which limits its autonomous generation ability.
受内存与计算约束，GameGen-X 尚不支持超高分辨率内容生成（如 2K/4K），限制了其在 AAA 级次世代游戏开发中的实际应用。Constrained by memory and computation, GameGen-X does not yet support ultra-high-resolution content generation (such as 2K/4K), which limits its practical use in AAA-level next-generation game development.
模型短期记忆窗口仅为 1–108 帧，当玩家返回已生成场景时，场景可能发生显著改变，无法维持长时间的环境连贯性。The short-term memory window of the model is only 1–108 frames, so when a player returns to an already generated scene the scene may change significantly, and long-term environmental coherence cannot be maintained.
生成内容在光照反射、角色-环境交互的物理准确性方面仍有不足；当前训练数据分布也限制了多角色协同交互场景的生成质量。The generated content is still deficient in the physical accuracy of light reflection and of character-environment interaction; the current training data distribution also limits the generation quality of scenes with multiple characters interacting.
与现有游戏引擎不兼容（Engine Integration）Engine Integration
模型输出的视频目前无法直接兼容现有游戏引擎，需要额外的转换步骤（如视频转 3D 模型）才能融入游戏开发工作流。The videos output by the model cannot currently be used directly with existing game engines; extra conversion steps (such as video-to-3D-model conversion) are needed before they fit into the game development workflow.
