170 亿参数交互式游戏世界基础模型，精确响应键盘与鼠标控制An interactive game-world foundation model with 17000000000 parameters, responding precisely to keyboard and mouse control
Matrix-Game 是一个面向可交互游戏世界生成的基础模型，通过两阶段训练——先在 2,700 小时无标注 Minecraft 视频上进行大规模预训练，再用 1,200 小时精标动作数据微调——实现了对角色动作（键盘）和摄像机（鼠标）的精确帧级控制，同时保持高视觉质量与时序一致性。Matrix-Game is a foundation model for interactive game-world generation. A two-stage training recipe — large-scale pre-training on 2,700 hours of unlabeled Minecraft video, followed by fine-tuning on 1,200 hours of precisely annotated action data — delivers accurate frame-level control over character actions (keyboard) and camera (mouse) while preserving high visual quality and temporal consistency.
world modelinteractive generationdiffusion transformeraction controllabilityautoregressive generationMinecraftGameWorld Score可交互世界模型interactive world model视频生成video generation3D Causal VAE
01动机 MotivationMotivation
世界模型是智能体感知、模拟和推理环境动态的核心。然而，现有可交互视频生成方法面临三重瓶颈：数据匮乏（精标动作视频采集昂贵）、物理动态难以建模（细粒度时序可控性不足）、以及评测标准缺失（缺乏客观比较的统一基准）。World models are the core of how an agent perceives, simulates and reasons about environment dynamics. Existing interactive video generation methods, however, face three bottlenecks: data scarcity (precisely annotated action video is expensive to collect), physical dynamics that are hard to model (insufficient fine-grained temporal controllability), and missing evaluation standards (no unified benchmark for objective comparison).
"Interactive video datasets with rich annotations (e.g., precise actions, camera movement) are scarce and expensive to collect, especially at scale."
Figure 1(a)：Matrix-Game 在 Minecraft Desert 场景中的可控生成示例。模型能在沙漠、海滩、森林、山地、冰原、平原、河流、蘑菇岛等 8 种生物群系中精确响应用户输入，同时保持视觉真实感。Figure 1(a): controllable generation with Matrix-Game in a Minecraft Desert scene. The model responds precisely to user input across 8 biomes — desert, beach, forest, hills, icy, plains, river and mushroom island — while preserving visual realism.
与 Oasis 和 MineWorld 等已有开源世界模型相比，Matrix-Game 在可控性（keyboard / mouse accuracy）和物理一致性（object consistency）上取得显著提升，同时维持高视觉质量与时序流畅度。Compared with existing open-source world models such as Oasis and MineWorld, Matrix-Game achieves marked gains in controllability (keyboard / mouse accuracy) and physical consistency (object consistency), while maintaining high visual quality and temporal smoothness.
02方法 MethodMethod
Matrix-Game 采用 image-to-world 范式：以单帧图像为条件，通过 3D Causal VAE 压缩时空信息，再由 Multi-Modal Diffusion Transformer (MMDiT) 生成动作条件视频，并借助自回归策略实现长时程连续生成。Matrix-Game adopts an image-to-world paradigm: conditioned on a single frame, a 3D Causal VAE compresses spatiotemporal information, a Multi-Modal Diffusion Transformer (MMDiT) then generates action-conditioned video, and an autoregressive strategy enables long-horizon continuous generation.
Figure 4：Matrix-Game 整体架构——3D Causal VAE（空间 8× / 时序 4× 压缩）将输入帧编码为 latent，MMDiT 在此空间进行去噪生成；动作控制信号（键盘离散指令 + 鼠标连续偏角）作为额外条件注入。Figure 4: overall architecture of Matrix-Game — a 3D Causal VAE (8× spatial / 4× temporal compression) encodes input frames into latents, on which MMDiT performs denoising generation; action control signals (discrete keyboard commands + continuous mouse angles) are injected as additional conditions.
两阶段训练流程Two-Stage Training Pipeline
Stage 1 — 无标注视频预训练：在 2,700 小时 720p Minecraft 原始视频上进行大规模预训练，帮助模型习得场景外观、物理规律与时序动态。原始数据经三级分层过滤（视频质量评估 → 菜单/字幕/人脸清除 → 运动模糊过滤），最终保留 870 小时高质量子集。支持变长帧数（17 / 33 / 65 帧）与多种宽高比（16:9 / 4:3 / 21:9）。Stage 1 — pre-training on unlabeled video: large-scale pre-training on 2,700 hours of raw 720p Minecraft video helps the model acquire scene appearance, physical regularities and temporal dynamics. The raw data passes through three-level hierarchical filtering (video quality assessment → removal of menus / subtitles / faces → motion-blur filtering), leaving a high-quality subset of 870 hours. Variable frame counts (17 / 33 / 65 frames) and multiple aspect ratios (16:9 / 4:3 / 21:9) are supported.
Stage 2 — 精标动作微调：在 1,200 小时带精确键盘 + 鼠标标注的视频上进行监督训练，使模型能精确响应帧级动作控制信号。标注数据来源于 MineRL 智能体自主探索（VPT 模型，16Hz 采样）和 Unreal 程序化仿真两条管线，覆盖 14 种生物群系，每类占比 4–7%，保证语义多样性。Stage 2 — fine-tuning on precisely annotated actions: supervised training on 1,200 hours of video with accurate keyboard + mouse annotations lets the model respond precisely to frame-level action control signals. The annotated data comes from two pipelines, autonomous exploration by MineRL agents (VPT model, 16Hz sampling) and procedural Unreal simulation, covering 14 biomes with 4–7% each to ensure semantic diversity.
动作控制模块Action Control Module
Matrix-Game 支持两类控制信号：Matrix-Game supports two kinds of control signals:
键盘离散动作（前/后/左/右/跳跃/攻击）：通过 cross-attention 注入 MMDiT。Discrete keyboard actions (forward / back / left / right / jump / attack): injected into MMDiT via cross-attention.
鼠标连续输入（俯仰角变化量，摄像机控制）：经 MLP 编码后通过 temporal self-attention 融合。Continuous mouse input (pitch-angle increments, camera control): encoded by an MLP and fused through temporal self-attention.
Figure 6：Matrix-Game diffusion transformer block 细节。MMDiT 架构将视频 token 与图像 token 在 token 维度拼接，动作控制信号作为独立 condition 注入，键盘经 cross-attention、鼠标经 MLP + temporal self-attention 集成。Figure 6: details of the Matrix-Game diffusion transformer block. The MMDiT architecture concatenates video tokens and image tokens along the token dimension; action control signals are injected as separate conditions, keyboard through cross-attention and mouse through MLP + temporal self-attention.
自回归长视频生成Autoregressive Long-Video Generation
为突破单次生成固定帧长的限制，模型采用自回归策略：每个生成片段末尾的 k=5 帧 latent 与下一段 noisy latent 在 channel 维度拼接，并附 binary mask 指示有效运动帧。同时以 0.2 概率向 motion frame 添加 Gaussian noise、以 0.25 概率将其替换为零 latent（classifier-free guidance），从而抑制误差累积并提升长时稳定性。To break the fixed frame length of a single generation pass, the model adopts an autoregressive strategy: the last k=5 latent frames of each generated clip are concatenated with the noisy latent of the next segment along the channel dimension, with a binary mask marking valid motion frames. Gaussian noise is added to the motion frames with probability 0.2 and they are replaced by zero latents with probability 0.25 (classifier-free guidance), which suppresses error accumulation and improves long-horizon stability.
Figure 3：Matrix-Game-MC 无标注数据三阶段分层过滤流程——Stage 1 过滤低质量画面，Stage 2 清除游戏菜单、字幕与人脸，Stage 3 过滤运动模糊与不合理摄像机移动，最终保留高质量训练数据。Figure 3: the three-stage hierarchical filtering pipeline for unlabeled Matrix-Game-MC data — Stage 1 filters low-quality footage, Stage 2 removes game menus, subtitles and faces, Stage 3 filters motion blur and implausible camera movement, leaving high-quality training data.
03实验 ExperimentsExperiments
作者提出 GameWorld Score——一个覆盖 4 大支柱、8 个维度的统一评测基准，并在该基准上与 Oasis 和 MineWorld 进行全面对比，同时进行人类双盲评测。The authors propose GameWorld Score, a unified benchmark covering 4 pillars and 8 dimensions, and compare comprehensively against Oasis and MineWorld on it, together with a double-blind human evaluation.
Matrix-Game 在全部 8 个维度均超过 Oasis 和 MineWorld，尤其在可控性（Keyboard Accuracy: 0.95 vs 0.77 / 0.86；Mouse Accuracy: 0.95 vs 0.56 / 0.64）和物理一致性（Object Consistency: 0.76 vs 0.56 / 0.51）方面优势显著。Matrix-Game surpasses Oasis and MineWorld on all 8 dimensions, with especially clear margins in controllability (Keyboard Accuracy: 0.95 vs 0.77 / 0.86; Mouse Accuracy: 0.95 vs 0.56 / 0.64) and physical consistency (Object Consistency: 0.76 vs 0.56 / 0.51).
细粒度键盘动作准确率（Table 3 节选）Fine-Grained Keyboard Action Accuracy (excerpt from Table 3)
动作 ActionAction
Oasis
MineWorld
Matrix-Game
Forward
0.85
0.86
0.99
Left
0.80
0.87
0.92
Right
0.79
0.88
0.96
Jump
0.77
0.82
0.88
Figure 2：GameWorld Score 雷达图对比。Matrix-Game（蓝色）在全部 8 个维度上均领先 Oasis 和 MineWorld，特别在 Keyboard Accuracy 和 Mouse Accuracy 两个可控性维度提升幅度最大。Aesthetic 维度已 ×2 缩放以便可视化。Figure 2: GameWorld Score radar chart. Matrix-Game (blue) leads Oasis and MineWorld on all 8 dimensions, with the largest gains on the two controllability dimensions, Keyboard Accuracy and Mouse Accuracy. The Aesthetic dimension is scaled by ×2 for visualization.
人类双盲评测Double-Blind Human Evaluation
Figure 8：人类双盲评测结果（win rate vs Oasis + MineWorld）。Matrix-Game 在 Overall Quality（96.30%）、Visual Quality（98.23%）、Controllability（93.76%）和 Temporal Consistency（89.56%）四个维度均显著领先。Figure 8: double-blind human evaluation results (win rate vs Oasis + MineWorld). Matrix-Game leads clearly on all four dimensions: Overall Quality (96.30%), Visual Quality (98.23%), Controllability (93.76%) and Temporal Consistency (89.56%).
消融与 Scenario 泛化Ablations and Scenario Generalization
模型在 8 种 Minecraft 生物群系（beach、desert、forest、hills、icy、mushroom、plains、river）上进行测试。论文指出："Our model consistently outperforms existing open-source baselines...across all eight scenarios."。键盘动作 IDM（Inverse Dynamics Model）分类准确率为 90.6%，鼠标运动回归 R² 达 0.97，验证了评测指标的可靠性。The model is tested on 8 Minecraft biomes (beach, desert, forest, hills, icy, mushroom, plains, river). The paper states: "Our model consistently outperforms existing open-source baselines...across all eight scenarios.". The keyboard-action IDM (Inverse Dynamics Model) reaches a classification accuracy of 90.6% and mouse-motion regression reaches R² = 0.97, confirming the reliability of the evaluation metrics.
04局限性 LimitationsLimitations
Note：以下局限性由作者在论文 Failure Cases 及 Conclusion 节明确陈述，并附有可视化示例（Figure 11）。Note: the limitations below are stated explicitly by the authors in the Failure Cases and Conclusion sections of the paper, with visual examples (Figure 11).
边缘场景泛化不足Limited generalization to edge scenarios
"The model may occasionally struggle with precise controllability or spatial consistency in rare biomes or edge cases, typically due to insufficient data coverage."——在数据覆盖不足的罕见生物群系或极端场景下，模型偶尔出现可控性下降或空间一致性问题。"The model may occasionally struggle with precise controllability or spatial consistency in rare biomes or edge cases, typically due to insufficient data coverage." — in rare biomes or extreme scenarios where data coverage is insufficient, the model occasionally shows degraded controllability or spatial-consistency problems.
物理理解存在盲区Blind spots in physical understanding
"There is still room to further enhance its understanding of physical dynamics, particularly in interactions such as object collisions or terrain traversal"——对于物体碰撞、穿越特定地形（如树叶）等涉及精细物理交互的场景，模型仍可能出现穿模或物理错误（如角色走过树叶时不符合物理规律）。"There is still room to further enhance its understanding of physical dynamics, particularly in interactions such as object collisions or terrain traversal" — in scenes involving fine-grained physical interaction, such as object collisions or traversing certain terrain (e.g. leaves), the model can still show clipping or physically wrong behaviour (for example a character walking through leaves in a way that violates physics).
动作空间与游戏场景受限Limited action space and game scenarios
当前鼠标控制范围有限（每帧最大 15° 偏转），键盘动作空间也未覆盖 Minecraft 全部操作。模型目前仅在 Minecraft 场景训练，作者将扩展至 Black Myth: Wukong、赛车模拟器、CS:GO 等复杂游戏作为未来工作。The current mouse control range is limited (at most 15° of deflection per frame), and the keyboard action space does not cover all Minecraft operations. The model is so far trained only on Minecraft scenes; the authors leave extension to complex games such as Black Myth: Wukong, racing simulators and CS:GO as future work.
长时序误差累积Error accumulation over long horizons
自回归生成策略虽通过 Gaussian noise augmentation 和 CFG 缓解了时序漂移，但长时程视频（多段自回归）仍面临误差累积风险。论文将引入更长 motion context 或基于记忆机制的方案列为未来工作。Although the autoregressive generation strategy mitigates temporal drift through Gaussian noise augmentation and CFG, long-horizon video (multi-segment autoregression) still risks error accumulation. The paper lists longer motion context or memory-based mechanisms as future work.
