Genie: Generative Interactive Environments

01 动机Motivation

生成式 AI 已能生成高质量的文本、图像和视频，但真正可交互的虚拟环境仍高度依赖人工设计或带标注的数据。能否从海量无标注的网络视频中，自动学习出一个可供智能体"玩耍"的交互世界？Generative AI can already produce high-quality text, images and video, yet genuinely interactive virtual environments still rely heavily on manual design or annotated data. Could an interactive world for an agent to "play" in be learned automatically from massive amounts of unlabelled web video?

"We introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos."

Genie 生成的多样轨迹 — 图 1：Genie 从文字提示、手绘草图或真实照片生成 2D 平台游戏场景，并通过学到的潜在动作逐帧控制游戏进程。同一个提示下可产生风格各异的可玩环境。Figure 1: From a text prompt, a hand-drawn sketch or a real photograph, Genie generates 2D platformer scenes and controls the progress of the game frame by frame through the latent actions it has learned. The same prompt can yield playable environments of widely differing styles.

11B总参数量Total parameters

6.8M训练视频（30k 小时，2D 游戏）Training videos (30k hours, 2D games)

8离散潜在动作数（无人工标注）Discrete latent actions (no human annotation)

~1 FPS当前推理速度Current inference speed

现有方法要么依赖真实动作标注（如环境模拟器日志），要么只能生成被动视频，无法响应用户交互。Genie 的核心洞见在于：动作信息隐含在视频帧的时序变化之中——只要能从相邻帧中推断出潜在动作，便可在完全无标注的条件下学习一个可控的动态模型。Existing approaches either depend on ground-truth action labels (such as environment simulator logs) or can only generate passive video that cannot respond to user interaction. The core insight of Genie is that action information is implicit in the temporal changes between video frames — as long as latent actions can be inferred from adjacent frames, a controllable dynamics model can be learned entirely without annotation.

02 方法Method

Genie 由三个协同训练的模块构成：时空视频 tokenizer（将视频帧压缩为离散 token）、潜在动作模型（LAM）（无监督推断帧间动作）、以及自回归动态模型（依据动作预测下一帧）。推理时，LAM 被丢弃，用户直接输入 codebook 索引来控制生成。Genie comprises three jointly trained modules: a spatiotemporal video tokenizer (compressing video frames into discrete tokens), a latent action model (LAM) (inferring inter-frame actions without supervision), and an autoregressive dynamics model (predicting the next frame conditioned on the action). At inference time the LAM is discarded and the user directly enters codebook indices to control generation.

Genie 整体训练流程 — 图 2：Genie 训练流程。视频帧先经 ST-ViViT tokenizer 编码为离散 token；LAM 从像素级帧对中推断潜在动作；动态模型（MaskGIT）以 token + 动作 embedding 为输入，预测下一帧 token；最终用 VQ-VAE decoder 重建像素图像。Figure 2: The Genie training pipeline. Video frames are first encoded into discrete tokens by the ST-ViViT tokenizer; the LAM infers latent actions from pixel-level frame pairs; the dynamics model (MaskGIT) takes tokens plus action embeddings as input and predicts the tokens of the next frame; finally a VQ-VAE decoder reconstructs the pixel image.

① 时空视频 Tokenizer（ST-ViViT）① Spatiotemporal Video Tokenizer (ST-ViViT)

tokenizer 采用改进的 VQ-VAE 架构，将视频建模为二维 token 网格（空间 × 时间）。空间 attention 在每一帧内处理 O(H×W) 个 token，时间 attention 则跨帧处理 O(T) 个 token，相比 C-ViViT 的全时空 attention 大幅降低显存。因果遮罩（causal masking）保证编码时不泄露未来信息，使 tokenizer 可在流式场景中使用。The tokenizer adopts an improved VQ-VAE architecture that models video as a two-dimensional token grid (space × time). Spatial attention processes O(H×W) tokens within each frame while temporal attention processes O(T) tokens across frames, cutting memory substantially compared with the full spatiotemporal attention of C-ViViT. Causal masking ensures that encoding never leaks future information, making the tokenizer usable in streaming settings.

ST-Transformer 结构 — 图 3：ST-Transformer 交错执行空间 attention 与时间 attention，同时使用因果遮罩。相比 C-ViViT，在同等性能下显存仅需 0.9 GB（C-ViViT 需 1.6 GB）。Figure 3: The ST-Transformer interleaves spatial attention with temporal attention while applying causal masking. Compared with C-ViViT, it needs only 0.9 GB of memory at equal performance (C-ViViT needs 1.6 GB).

② 潜在动作模型（Latent Action Model, LAM）② Latent Action Model (LAM)

LAM 以像素级别的前若干帧和当前帧为输入，通过 VQ-VAE 编码器推断出离散潜在动作（codebook 大小为 8），再由解码器以前几帧 + 潜在动作重建当前帧。VQ-VAE 的信息瓶颈迫使 codebook 只保留决定帧间差异的最关键信号——即动作。论文特别指出，像素输入比 token 输入更好地保留了运动信息，在可控性指标 ΔtPSNR 上提升明显（1.91 vs 1.33）。The LAM takes the preceding frames and the current frame at the pixel level as input, infers a discrete latent action through a VQ-VAE encoder (codebook size 8), and then reconstructs the current frame from the preceding frames plus the latent action via the decoder. The information bottleneck of the VQ-VAE forces the codebook to retain only the most critical signal determining the difference between frames — namely the action. The paper points out in particular that pixel input preserves motion information better than token input, yielding a clear gain on the controllability metric ΔtPSNR (1.91 vs 1.33).

LAM 架构 — 图 4：Latent Action Model 结构。编码器输入前帧 + 当前帧，解码器输入前帧 + 潜在动作，VQ 瓶颈约束 codebook 仅存储动作相关信息。推理时 LAM 被丢弃，用户直接操控 codebook 索引。Figure 4: Structure of the Latent Action Model. The encoder takes the previous frames plus the current frame, the decoder takes the previous frames plus the latent action, and the VQ bottleneck constrains the codebook to store action-related information only. At inference the LAM is discarded and the user manipulates codebook indices directly.

③ 自回归动态模型（MaskGIT Dynamics Model）③ Autoregressive Dynamics Model (MaskGIT Dynamics Model)

动态模型基于 MaskGIT 架构，输入为历史帧的视频 token 序列加上对应的潜在动作 embedding（加性注入，效果优于拼接），预测下一帧的 token 分布。训练时随机遮罩 50%–100% 的目标 token 并用交叉熵损失优化；推理时经 25 步迭代采样还原完整帧。动态模型占整体参数量的 10.1B（LAM 300M + tokenizer 200M）。The dynamics model builds on the MaskGIT architecture: it takes the video token sequence of past frames together with the corresponding latent action embeddings (injected additively, which works better than concatenation) and predicts the token distribution of the next frame. During training 50%–100% of the target tokens are randomly masked and optimised with a cross-entropy loss; at inference the complete frame is recovered through 25 iterative sampling steps. The dynamics model accounts for 10.1B of the total parameter count (LAM 300M + tokenizer 200M).

03 实验Experiments

主要实验在两个域进行：Platformers（6.8M 视频，约 30k 小时，2D 平台游戏）和Robotics（~130k 机器人演示 + 209k 真实场景片段）。评估指标包括视频质量 FVD（Fréchet Video Distance，越低越好）和可控性 ΔtPSNR（越高越好）。The main experiments are run on two domains: Platformers (6.8M videos, about 30k hours of 2D platform games) and Robotics (~130k robot demonstrations plus 209k real-world clips). The evaluation metrics are video quality FVD (Fréchet Video Distance, lower is better) and controllability ΔtPSNR (higher is better).

Tokenizer 架构消融Tokenizer Architecture Ablation

架构Architecture	显存Memory	FVD ↓	ΔtPSNR ↑
ViT（仅空间 attention）ViT (spatial attention only)	0.3 GB	114.5	1.39
C-ViViT	1.6 GB	272.7	1.37
ST-ViViT（本文）ST-ViViT (ours)	0.9 GB	81.4	1.66

LAM 输入方式消融LAM Input Ablation

LAM 输入LAM input	FVD ↓	ΔtPSNR ↑
Token 输入Token input	38.8	1.33
像素输入（本文）Pixel input (ours)	40.1	1.91

像素输入在 FVD 上略差，但显著提升了可控性（ΔtPSNR +0.58），实验选择像素输入以优先保证可交互性。Pixel input is slightly worse on FVD but markedly improves controllability (ΔtPSNR +0.58); the experiments choose pixel input in order to put interactivity first.

缩放实验Scaling Experiments

模型大小与 batch size 缩放曲线 — 图 8：从 40M 到 2.7B 参数规模，token 预测 loss 持续下降，验证 Genie 具备良好的缩放性质（scaling law）。增大 batch size（128→448）同样带来性能提升。Figure 8: From 40M to 2.7B parameters, the token prediction loss keeps decreasing, confirming that Genie enjoys good scaling behaviour (scaling law). Increasing the batch size (128→448) brings a further gain as well.

泛化到分布外提示（OOD Prompts）Generalisation to Out-of-Distribution Prompts (OOD Prompts)

OOD 提示图 — 图 9：Genie 能够接受 Imagen 2 生成图、手绘草图、真实照片作为提示，生成风格迥异但可交互的场景，展示了跨域泛化能力。Figure 9: Genie can accept images generated by Imagen 2, hand-drawn sketches and real photographs as prompts, producing scenes of very different styles that remain interactive, which demonstrates cross-domain generalisation.

智能体训练（Agent Training）Agent Training

智能体训练结果 — 图 14：在 Genie 生成的 CoinRun 环境中，仅用行为克隆（BC）训练的智能体在 200 个专家样本下即可匹配"oracle BC"基线（即在真实环境中训练），验证了 Genie 作为训练环境的可行性。Figure 14: In the CoinRun environment generated by Genie, an agent trained with behaviour cloning (BC) alone already matches the "oracle BC" baseline (i.e. trained in the real environment) with only 200 expert samples, confirming the viability of Genie as a training environment.

机器人域结果Results on the Robotics Domain

在机器人演示数据上（Robotics 域），2.5B 参数版本 FVD 达到 82.7。模型在无任何物理先验的情况下自发学会了可形变物体的物理规律（如布料折叠、软体运动）以及平台游戏中的视差效果（parallax）——前景运动快于背景，体现出对 3D 深度的隐式理解。On robot demonstration data (the Robotics domain), the 2.5B-parameter version reaches an FVD of 82.7. Without any physical prior, the model spontaneously learns the physics of deformable objects (such as cloth folding and soft-body motion) as well as the parallax effect in platform games — the foreground moves faster than the background — which reflects an implicit understanding of 3D depth.

04 局限性Limitations

说明：以下局限性均为作者在论文中明确陈述（stated by the authors）。Note: all limitations below are explicitly stated by the authors in the paper.

长时生成的幻觉（Hallucinations over Long Horizons）Hallucinations over Long Horizons

自回归生成存在误差累积问题：模型记忆窗口仅为 16 帧，超出此范围后场景一致性明显下降，出现物体消失、背景突变等幻觉现象。Autoregressive generation suffers from error accumulation: the memory window of the model is only 16 frames, and beyond that range scene consistency degrades noticeably, with hallucinations such as disappearing objects and abrupt background changes.

推理速度慢（~1 FPS）Slow Inference (~1 FPS)

当前推理速度约 1 FPS，远低于真实游戏交互所需的帧率（30+ FPS）。每帧需执行 25 步 MaskGIT 采样，计算代价高昂，实时交互尚不可行，论文将提速列为未来工作。Inference currently runs at about 1 FPS, far below the frame rate required for real game interaction (30+ FPS). Each frame requires 25 MaskGIT sampling steps, which is computationally expensive, so real-time interaction is not yet feasible; the paper lists acceleration as future work.

模型未开源（Model Not Released）Model Not Released

受多种因素限制，完整模型权重未对外发布。作者提供了可复现的 CoinRun 案例研究（case study）作为替代，但社区无法直接在此基础上进行下游研究或微调。Because of a variety of constraints, the complete model weights have not been released publicly. The authors provide a reproducible CoinRun case study as a substitute, but the community cannot conduct downstream research or fine-tuning directly on top of it.

训练域偏窄（Limited to 2D Platformers / Robotics）Limited to 2D Platformers / Robotics

当前模型主要在 2D 游戏和机器人演示上验证，尚未扩展至 3D 场景、第一人称视角或真实世界复杂环境。潜在动作空间的语义是否能迁移到更复杂的动作空间（如连续动作）仍有待探索。The current model is validated mainly on 2D games and robot demonstrations and has not yet been extended to 3D scenes, first-person viewpoints or complex real-world environments. Whether the semantics of the latent action space can transfer to more complex action spaces (such as continuous actions) remains to be explored.