NitroGen: An Open Foundation Model for Generalist Gaming Agents

01 动机Motivation

电子游戏是训练通用具身智能体的理想沙盒：它们种类繁多、规则多样、且无需物理硬件。然而，现有方法要么依赖手工设计的 API（如 Voyager），要么需要昂贵的强化学习训练（如 DQN、AlphaStar），要么人工演示数据规模极为有限。本文的核心问题是：能否直接从互联网上海量、嘈杂的玩家录像中，自动学习跨越千款游戏的视觉-动作策略？Video games are an ideal sandbox for training generalist embodied agents: they are numerous, diverse in their rules, and require no physical hardware. Existing approaches, however, either rely on hand-designed APIs (e.g. Voyager), require expensive reinforcement learning (e.g. DQN, AlphaStar), or are limited to very small amounts of human demonstration data. The central question of this paper is: can a vision-action policy spanning a thousand games be learned automatically, directly from the massive and noisy player recordings available on the internet?

"We present a vision-action foundation model trained on 40,000 hours of gameplay videos across more than 1,000 games."

NitroGen overview — **图1：NitroGen 系统概览。**模型接受单帧画面输入，输出 16 步手柄动作序列，可在格斗、平台跳跃、探索等多类型游戏中表现出非平凡的成功率。数据集、评测套件与模型权重均开放发布。**Figure 1: Overview of the NitroGen system.** The model takes a single frame as input and outputs a 16-step gamepad action sequence, reaching non-trivial success rates across genres such as fighting, platforming and exploration. The dataset, the evaluation suite and the model weights are all released openly.

40K训练视频总时长（小时）Total training video duration (hours)

1,000+覆盖游戏数量Number of games covered

52%战斗任务迁移相对提升Relative gain on transferred combat tasks

30跨 10 款游戏的评测任务Evaluation tasks across 10 games

02 方法Method

NitroGen 由三个相互配合的核心模块构成：（1）互联网规模视频-动作数据集的自动构建流水线；（2）涵盖 10 款游戏、30 个任务的多游戏评测基准；（3）基于 flow matching 的统一视觉-动作基础模型。NitroGen consists of three mutually supporting core components: (1) an automated pipeline that builds an internet-scale video-action dataset; (2) a multi-game evaluation benchmark covering 10 games and 30 tasks; (3) a unified vision-action foundation model based on flow matching.

数据采集与动作提取Data collection and action extraction

研究者从视频平台抓取了 71,000 小时含手柄叠加层（gamepad overlay）的游戏录像，来自 818 位创作者的 38,739 个视频。经过三阶段流水线处理后，保留了 40,000 小时高质量数据：The authors crawled 71,000 hours of gameplay recordings carrying a gamepad overlay from video platforms, covering 38,739 videos from 818 creators. After a three-stage pipeline, 40,000 hours of high-quality data were retained:

模板匹配（Template Matching）：使用约 300 种常见手柄模板，以 SIFT 和 XFeat 特征进行匹配，至少需要 20 个内点（inliers）才视为有效检测。Template Matching: about 300 common gamepad templates are matched with SIFT and XFeat features; at least 20 inliers are required for a detection to count as valid.
手柄解析（Gamepad Parsing）：使用 SegFormer 分割模型处理连续帧对，输出 11×11 离散网格上的摇杆位置及各按键的二值状态。训练数据为通过程序化生成合成叠加层标注的 800 万帧。Gamepad Parsing: a SegFormer segmentation model processes consecutive frame pairs and outputs the stick position on an 11×11 discrete grid together with the binary state of each button. Its training data consists of 8000000 frames annotated from procedurally generated synthetic overlays.
质量过滤（Quality Filtering）：丢弃动作密度低于 50% 的片段（以避免 null-action 偏差），最终保留约 55% 的原始数据。Quality Filtering: clips whose action density falls below 50% are discarded (to avoid a null-action bias), which keeps about 55% of the raw data.

data pipeline — **图2：视频-动作数据采集流水线。**从网络视频中定位手柄叠加层，经模板匹配、分割解析、质量过滤三个阶段，自动提取高精度游戏动作标注。**Figure 2: The video-action data collection pipeline.** Gamepad overlays are located in web videos and, through the three stages of template matching, segmentation-based parsing and quality filtering, high-precision game action annotations are extracted automatically.

**图3：动作提取流水线详情。**对每一帧检测手柄区域，利用 SegFormer 分割摇杆和按键区域，解析出连续摇杆坐标（R²=0.84）和按键二值状态（accuracy=0.96）。**Figure 3: Details of the action extraction pipeline.** The gamepad region is detected in every frame, SegFormer segments the stick and button areas, and continuous stick coordinates (R²=0.84) and binary button states (accuracy=0.96) are parsed out.

dataset distribution — **图3（续）：数据集游戏分布。**（左）按游戏统计时长：846 款游戏超 1 小时，91 款超 100 小时，15 款超 1,000 小时。（右）按游戏类型：Action-RPG 占 34.9%，Platformer 占 18.4%，Action-Adventure 占 9.2%。**Figure 3 (continued): Game distribution of the dataset.** (Left) Duration per game: 846 games exceed 1 hour, 91 exceed 100 hours and 15 exceed 1,000 hours. (Right) By genre: Action-RPG accounts for 34.9%, Platformer for 18.4% and Action-Adventure for 9.2%.

视觉-动作基础模型架构Architecture of the vision-action foundation model

模型以 flow matching 框架为核心，通过扩散变换器（Diffusion Transformer, DiT）在给定单帧视觉观测的条件下，生成未来的动作序列块（action chunk）：The model is centred on a flow matching framework: a Diffusion Transformer (DiT) generates a future action chunk conditioned on a single-frame visual observation:

视觉编码器：SigLIP 2 视觉变换器，处理 256×256 RGB 输入，每帧生成 256 个图像 token。Visual encoder: a SigLIP 2 vision transformer processes 256×256 RGB input and produces 256 image tokens per frame.
动作生成：DiT 每次前向传播输出 16 步动作块，通过 MLP 将带噪声的动作块编码为每步一个 action token，再经过交替的 self-attention 与 cross-attention 层处理。Action generation: each DiT forward pass outputs a 16-step action chunk; an MLP encodes the noisy action chunk into one action token per step, which is then processed by alternating self-attention and cross-attention layers.
统一动作空间：标准化 16 维二值向量（手柄按键）+ 4 维连续摇杆向量，支持跨游戏泛化。Unified action space: a normalized 16-dimensional binary vector (gamepad buttons) plus a 4-dimensional continuous stick vector, which supports cross-game generalization.
训练：AdamW 优化器（weight decay 0.001），warmup-stable-decay 学习率调度（恒定学习率 1e-4），batch size 256，EMA 衰减系数 0.9999。推理时使用 16 步 Euler 积分去噪。Training: AdamW optimizer (weight decay 0.001), a warmup-stable-decay learning rate schedule (constant learning rate 1e-4), batch size 256, EMA decay 0.9999. At inference, denoising uses 16 Euler integration steps.

in-game rollouts — **图4：多款游戏中的策略展示（rollouts）。**NitroGen 在 2D 平台跳跃、3D 动作角色扮演、随机生成地图探索等多类型游戏中均能完成有效操作，展示出跨游戏类型的泛化能力。**Figure 4: Policy rollouts in several games.** NitroGen acts effectively across genres such as 2D platformers, 3D action role-playing games and exploration of procedurally generated maps, showing generalization across game types.

03 实验Experiments

评测在统一 Gymnasium API 封装的 10 款商业游戏（5 款 2D、5 款 3D）上进行，覆盖格斗、导航、游戏特有机制共 30 个任务，采用人工评估衡量任务完成率（task success rate）。Evaluation is run on 10 commercial games (5 in 2D, 5 in 3D) wrapped by a unified Gymnasium API, covering 30 tasks in total across fighting, navigation and game-specific mechanics, with the task success rate measured by human evaluation.

动作提取精度Accuracy of action extraction

**图5：手柄解析精度验证。**（a）摇杆 R² 相关分数，各手柄家族均值为 0.84；（b）按键帧级别精度，均值为 0.96——证明自动提取的动作标注质量足以支撑大规模行为克隆预训练。**Figure 5: Validation of gamepad parsing accuracy.** (a) Stick R² correlation scores, averaging 0.84 over gamepad families; (b) frame-level button accuracy, averaging 0.96 — evidence that the automatically extracted action labels are good enough to support large-scale behavior cloning pretraining.

零样本跨游戏性能Zero-shot cross-game performance

zero-shot performance — **图6：零样本（zero-shot）跨游戏表现。**NitroGen 在未经微调的情况下，在多款外观风格和类型迥异的游戏上均取得非平凡的成功率，证明了"trained on a very noisy internet dataset"的模型仍具有跨域泛化能力。**Figure 6: Zero-shot cross-game performance.** Without any fine-tuning, NitroGen reaches non-trivial success rates on games with very different visual styles and genres, showing that a model "trained on a very noisy internet dataset" still generalizes across domains.

预训练迁移（Transfer Learning）结果Transfer learning results from pretraining

游戏类型 / 任务类别Game type / task category	从头训练（Scratch）From scratch	NitroGen 预训练微调NitroGen pretrained + fine-tuned	相对提升Relative gain
等距俯视 Roguelike（平均）Isometric roguelike (average)	基线Baseline	+10% 相对提升+10% relative gain	+10%
3D Action-RPG（平均）3D Action-RPG (average)	基线Baseline	+25% 相对提升+25% relative gain	+25%
3D Action-RPG · 格斗任务3D Action-RPG · fighting tasks	基线Baseline	最高 +52% 相对提升up to +52% relative gain	+52%
3D Action-RPG · 导航任务3D Action-RPG · navigation tasks	基线Baseline	+25% 相对提升+25% relative gain	+25%
3D Action-RPG · 游戏特有任务3D Action-RPG · game-specific tasks	基线Baseline	+5% 相对提升+5% relative gain	+5%

评测基准构成Composition of the benchmark

研究者开发了通用 Gymnasium API 封装器，支持任意商业游戏接入，并设计了跨 10 款游戏的 30 个任务，覆盖格斗（combat）、导航（navigation）和游戏特有机制（game-specific mechanics）三大类别。所有任务均采用人工评估以保证评测可靠性。The authors developed a general Gymnasium API wrapper that lets arbitrary commercial games be plugged in, and designed 30 tasks across 10 games, covering the three categories of combat, navigation and game-specific mechanics. All tasks are scored by human evaluation to keep the evaluation reliable.

注意：由于商业游戏版权限制，当前评测基准和数据集详情无法完全开放，但模型权重已在 nitrogen.minedojo.org 开放发布。Note: because of copyright restrictions on commercial games, the current benchmark and the details of the dataset cannot be fully released, but the model weights are openly available at nitrogen.minedojo.org.

04 局限性Limitations

Note: 以下局限性部分由作者在论文中明确陈述（标注为 stated），部分由设计推断（标注为 inferred）。Some of the limitations below are explicitly stated by the authors in the paper (marked stated), while others are inferred from the design (marked inferred).

缺乏长程规划与语言条件化（stated）No long-horizon planning or language conditioning (stated)

NitroGen 是"System-1"反应式模型，依赖单帧视觉输入生成短期动作序列，无法进行多步推理、任务分解或响应自然语言指令。作者指出未来工作包括语言跟随（language following）和强化学习后训练（RL post-training）。NitroGen is a "System-1" reactive model: it relies on single-frame visual input to produce short-horizon action sequences and cannot perform multi-step reasoning, task decomposition, or follow natural language instructions. The authors note that future work includes language following and RL post-training.

数据集偏向动作类游戏（stated）The dataset is biased toward action games (stated)

数据集中 Action-RPG 占 34.9%，Platformer 占 18.4%，而策略游戏（strategy）和键盘主导（keyboard-centric）游戏代表性不足，导致模型在这些类型上的泛化能力可能受限。Action-RPG accounts for 34.9% of the dataset and Platformer for 18.4%, while strategy games and keyboard-centric games are under-represented, so the model’s generalization to these genres may be limited.

动作标注存在系统性噪声（inferred）Systematic noise in the action labels (inferred)

视频中的动作提取存在固有延迟（视频编码延迟、手柄叠加层刷新延迟）和各创作者手柄型号/布局差异带来的解析误差。尽管摇杆 R²=0.84、按键精度=0.96，但残余噪声仍可能限制精细操控任务的学习效果。Action extraction from video suffers from inherent latency (video encoding delay, gamepad overlay refresh delay) and from parsing errors caused by differing gamepad models and layouts across creators. Although the stick reaches R²=0.84 and button accuracy 0.96, the residual noise may still limit learning on fine-grained control tasks.

评测依赖人工标注，可扩展性有限（inferred）Evaluation depends on human annotation and scales poorly (inferred)

30 个评测任务的成功率判断依赖人工评估，难以扩展到大规模自动化测试。与 RL 环境不同，商业游戏缺乏程序化奖励信号，使得大规模、可重复的定量比较存在挑战。Success on the 30 evaluation tasks is judged by human evaluation, which is hard to scale to large automated testing. Unlike RL environments, commercial games lack programmatic reward signals, which makes large-scale, reproducible quantitative comparison challenging.