The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control

01 动机Motivation

现有世界模型面临三个核心瓶颈：（1）仅聚焦于低保真非 AAA 游戏；（2）生成长度受限（通常不超过约一分钟）；（3）无法实现实时渲染。这些问题使世界模型难以作为真正可交互的游戏引擎或仿真平台。Existing world models face three core bottlenecks: (1) they focus only on low-fidelity, non-AAA games; (2) generation length is limited, typically to no more than about one minute; (3) real-time rendering is out of reach. These issues make world models hard to use as genuinely interactive game engines or simulation platforms.

"The first scalable, high-fidelity 1280×720 pixel world model in real time" — combining AAA game realism, infinite-length generation, and frame-level control in a single system.

teaser — The Matrix 生成的多场景画面 — The Matrix 在 Forza Horizon 5、Cyberpunk 2077 及真实世界场景中的生成效果展示，分别对应驾驶、步行与机器人操控三种控制模式。Generation results of The Matrix on Forza Horizon 5, Cyberpunk 2077 and real-world scenes, corresponding respectively to the driving, walking and robot-manipulation control modes.

720p生成分辨率 (1280×720)Output resolution (1280×720)

8–16 FPS实时渲染速度Real-time rendering speed

2.7B模型参数量Model parameters

∞理论生成长度（无限时域）Theoretical generation length (infinite horizon)

overview — 系统与基线对比 — 与现有方法的对比：The Matrix 是唯一同时实现无限视频长度、720p 分辨率、帧级实时控制（8–16 FPS）及控制泛化能力的世界模型，而 Genie（1 FPS、无泛化）、GameNGen（4秒上限）与 GameGen-X（仅视频级控制）均在一个或多个维度存在明显不足。Comparison with existing methods: The Matrix is the only world model that simultaneously achieves unlimited video length, 720p resolution, frame-level real-time control (8–16 FPS) and control generalization, whereas Genie (1 FPS, no generalization), GameNGen (a 4-second cap) and GameGen-X (video-level control only) each fall clearly short on one or more of these dimensions.

02 方法Method

The Matrix 由三个核心模块构成：Interactive Module（将键盘输入转化为语言描述后注入扩散过程）、Shift-Window Denoising Process Model（Swin-DPM）（滑动窗口无限长生成）以及 Stream Consistency Model（SCM）（蒸馏加速实现实时推理）。骨干网络为 Video DiT，共 32 个注意力块，基础参数量 2.3B，加上 Interactive Module 后总计 2.7B。The Matrix comprises three core modules: Interactive Module (which converts keyboard input into language descriptions and injects them into the diffusion process), Shift-Window Denoising Process Model (Swin-DPM) (sliding-window infinite-length generation) and Stream Consistency Model (SCM) (distillation-based acceleration for real-time inference). The backbone is a Video DiT with 32 attention blocks and 2.3B base parameters, reaching 2.7B in total once the Interactive Module is added.

architecture — 四阶段训练流程 — 四阶段训练流程：Stage 1 在无标注 Source 数据上进行 warm-up（20k 步，LoRA rank 128）；Stage 2 训练 Interactive Module（20k 步，冻结 DiT）；Stage 3 微调 Swin-DPM（60k 步，全参数）；Stage 4 蒸馏 SCM（10k 步，以 Swin-DPM 为教师模型）。所有阶段：batch size 32，学习率 1×10⁻⁵，bfloat16 精度。Four-stage training pipeline: Stage 1 warms up on unlabeled Source data (20k steps, LoRA rank 128); Stage 2 trains the Interactive Module (20k steps, DiT frozen); Stage 3 fine-tunes Swin-DPM (60k steps, all parameters); Stage 4 distills SCM (10k steps, with Swin-DPM as teacher). All stages: batch size 32, learning rate 1×10⁻⁵, bfloat16 precision.

modules — Interactive Module 与 Swin-DPM 示意图 — （a）Interactive Module：将键盘输入翻译为自然语言描述（如 "The car is driving forward"），经 T5 encoder 编码后通过 causal cross-attention 层（每两个 DiT block 后插入一层，causal relation window ω=4）注入生成过程。（b）Swin-DPM：滑动时域窗口去噪，窗口大小 w=T；每完成 k 步去噪后，最左侧 token 出队并缓存解码，下一窗口通过重新追加 noise level 为 0 的缓存 token 维持时序连续性。(a) Interactive Module: keyboard input is translated into a natural-language description (e.g. "The car is driving forward"), encoded by a T5 encoder and injected into generation through causal cross-attention layers (one inserted after every two DiT blocks, causal relation window ω=4). (b) Swin-DPM: denoising over a sliding temporal window of size w=T; after every k denoising steps the leftmost token is dequeued, cached and decoded, and the next window re-appends the cached token at noise level 0 so that temporal continuity is maintained.

训练数据：GameData 平台Training Data: the GameData Platform

自主构建的游戏数据采集系统，利用 Cheat Engine、Reshade 插件和 OBS 录制，从 CPU 内存状态中同步捕获游戏视频帧与控制信号：A self-built game-data collection system that uses Cheat Engine, Reshade plugins and OBS recording to capture game video frames and control signals synchronously from CPU memory state:

Forza Horizon 5：约 1.2M 视频-控制对Forza Horizon 5: about 1.2M video-control pairs
Cyberpunk 2077：约 1M 视频-控制对Cyberpunk 2077: about 1M video-control pairs
共 750k 标注样本 + 1.2M 无标注样本（6 秒片段，60 FPS）750k labeled samples in total + 1.2M unlabeled samples (6-second clips, 60 FPS)
DROID 机器人数据集：564 个场景中筛选出 50k 条 6 秒片段（含 7-DOF 关节角度标签）DROID robot dataset: 50k six-second clips selected from 564 scenes (with 7-DOF joint-angle labels)

Forza 数据过滤：平衡控制信号、碰撞检测、卡顿角色去除、运动-控制不匹配检测、伪影去除，共五项策略。Forza data filtering: balancing control signals, collision detection, removal of stuck characters, motion-control mismatch detection and artifact removal — five strategies in all.

Stream Consistency Model（SCM）Stream Consistency Model (SCM)

SCM 将 Swin-DPM 蒸馏为 4 步一致性模型，推理速度提升 10–20×，最终实现 8–16 FPS 的实时渲染。SCM distills Swin-DPM into a 4-step consistency model, speeding up inference by 10–20× and ultimately achieving 8–16 FPS real-time rendering.

代价是视觉质量有所下降（FVD 从 1211.30 上升到 1936.79），但控制精度（Move-LPIPS）从 0.125 进一步提升至 0.109，体现了速度-质量之间的权衡关系。The cost is some loss of visual quality (FVD rises from 1211.30 to 1936.79), while control accuracy (Move-LPIPS) improves further from 0.125 to 0.109, reflecting the trade-off between speed and quality.

03 实验Experiments

实验在三个场景下评估：Forza Horizon 5（赛车驾驶）、Cyberpunk 2077（角色步行）以及 DROID 机器人操控。评估指标包含视觉质量（FID、FVD、CLIP Score，在 2048 秒随机生成视频上计算）与控制精度（Move-PSNR、Move-LPIPS，在 2048 秒固定测试集上计算）。Evaluation covers three settings: Forza Horizon 5 (racing and driving), Cyberpunk 2077 (character walking) and DROID robot manipulation. The metrics comprise visual quality (FID, FVD, CLIP Score, computed over 2048 seconds of randomly generated video) and control accuracy (Move-PSNR, Move-LPIPS, computed over a fixed 2048-second test set).

帧级控制精度（4.1 节）Frame-Level Control Accuracy (Section 4.1)

场景Scenario	Move-LPIPS ↓	Move-PSNR ↑
Cyberpunk 2077	0.129	28.24
Forza Horizon 5	0.125	28.98
DROID 机器人DROID robot	0.180	27.90

各模块消融（速度-质量权衡，Table 2）Per-Module Ablation (Speed-Quality Trade-off, Table 2)

组件Component	推理速度Inference speed	FVD ↓	Move-LPIPS ↓
+ Interactive Module（基线）+ Interactive Module (baseline)	55 sec / 48 frames	1211.30	0.125
+ Swin-DPM	0.8 FPS	1651.50	0.113
+ SCM（完整系统）+ SCM (full system)	8–16 FPS	1936.79	0.109

注：Swin-DPM 的引入使 FVD 上升（视觉质量有所牺牲），但保持了强控制精度；SCM 进一步提升推理速度约 10–20×，并进一步降低 Move-LPIPS 至 0.109。Note: introducing Swin-DPM raises FVD (some visual quality is sacrificed) but preserves strong control accuracy; SCM further speeds up inference by roughly 10–20× and lowers Move-LPIPS further to 0.109.

infinite_gen — 无限长视频生成示例 — 无限时域生成示例：一分钟连续生成视频横跨沙漠、河流、草地、雪地并伴随昼夜切换；另有 2.5 分钟视频通过 DiT 文字提示跨越多个多样化场景。补充材料中包含长达半小时的连续生成示例。Infinite-horizon generation examples: a one-minute continuous video sweeps across desert, river, grassland and snowfield with day-night transitions; another 2.5-minute video traverses multiple diverse scenes via DiT text prompts. The supplementary material includes continuous generations lasting up to half an hour.

generalization — 对未见场景与对象的泛化 — Zero-shot 泛化能力：（左）汽车在从未出现于训练数据的室内办公室场景中行驶；（右）通过文字提示将运动控制迁移到人类角色。关闭 Interactive Module 后，系统可作为无条件长视频生成器使用。Zero-shot generalization: (left) a car drives through an indoor office scene that never appears in the training data; (right) motion control is transferred to a human character via a text prompt. With the Interactive Module switched off, the system can serve as an unconditional long-video generator.

04 局限性Limitations

说明：论文中未设专门的 Limitations 节。以下内容部分为作者在实验分析中直接指出（标注"论文指出"），部分为从设计与实验结果中推断（标注"推断"）。Note: The paper has no dedicated Limitations section. Some of the points below are stated directly by the authors in their experimental analysis (marked "stated in the paper"), while others are inferred from the design and the results (marked "inferred").

速度与视觉质量之间存在明显权衡（论文指出）A clear trade-off between speed and visual quality (stated in the paper)

引入 Swin-DPM 后 FVD 从 1211.30 上升至 1651.50，进一步加入 SCM 后升至 1936.79。实现实时推理（8–16 FPS）是以视觉保真度为代价的，现阶段难以同时达到最高画质与最快速度。After Swin-DPM is introduced, FVD rises from 1211.30 to 1651.50, and adding SCM pushes it further to 1936.79. Real-time inference (8–16 FPS) comes at the cost of visual fidelity; the best image quality and the highest speed cannot yet be attained at the same time.

泛化能力仅作定性验证（推断）Generalization is validated only qualitatively (inferred)

对未见场景（如室内办公室）和未见对象（如人类角色）的泛化结果仅以定性图示呈现，未提供系统性的量化评估，泛化边界尚不明确。Generalization to unseen scenes (such as an indoor office) and unseen objects (such as human characters) is presented only through qualitative figures, with no systematic quantitative evaluation, so the boundaries of generalization remain unclear.

训练数据依赖特定游戏，评估领域有限（推断）The training data depends on specific games and the evaluation domain is limited (inferred)

训练数据主要来自 Forza Horizon 5 与 Cyberpunk 2077 两款 AAA 游戏，以及 DROID 机器人数据集；评估域相对集中，系统在其他类型游戏或真实世界场景中的表现尚未系统验证。The training data comes mainly from two AAA games, Forza Horizon 5 and Cyberpunk 2077, together with the DROID robot dataset; the evaluation domain is relatively concentrated, and performance on other genres of game or on real-world scenes has not been systematically verified.

控制粒度受限于键盘指令抽象（推断）Control granularity is limited by the keyboard-command abstraction (inferred)

Interactive Module 将键盘输入翻译为自然语言描述（如 "The car is driving forward"），控制信号经过一层语言抽象，精细的运动轨迹或连续动作空间的控制能力尚未得到充分探索。The Interactive Module translates keyboard input into a natural-language description (e.g. "The car is driving forward"), so the control signal passes through a layer of linguistic abstraction; fine-grained motion trajectories and control over continuous action spaces remain insufficiently explored.