Yume: An Interactive World Generation Model

01 动机Motivation

现有视频扩散模型在生成交互式、可导航的真实世界场景时面临三大核心挑战：合成数据与真实场景之间的领域鸿沟、依赖绝对相机姿态参数导致的标注难与训练不稳定，以及城市复杂场景中普遍存在的视觉 artifacts（闪烁、纹理异常、几何扭曲）。Existing video diffusion models face three core challenges when generating interactive, navigable real-world scenes: the domain gap between synthetic data and real scenes; the annotation cost and training instability caused by relying on absolute camera pose parameters; and the visual artifacts (flickering, texture anomalies, geometric distortion) that pervade complex urban scenes.

"Yume aims to use images, text, or videos to create an interactive, realistic, and dynamic world, which allows exploration and control using peripheral devices or neural signals."

02 方法Method

Yume 的框架由四个相互协作的核心组件构成：相机运动量化、Masked Video Diffusion Transformer 架构、无训练采样器设计、以及模型加速。这四部分分别解决了控制稳定性、计算效率、生成质量和推理速度四个维度的问题。The Yume framework comprises four cooperating core components: camera motion quantization, the Masked Video Diffusion Transformer architecture, a training-free sampler design, and model acceleration. Together they address control stability, computational efficiency, generation quality, and inference speed.

1. Quantized Camera Motion (QCM)

现有方法依赖逐帧绝对相机矩阵（c2w）进行控制，标注成本高且对预训练模型不友好。QCM 将连续相机轨迹量化为 8 种离散动作（forward, backward, left, right, turn-left, turn-right, tilt-up, tilt-down），通过计算相邻帧的相对变换矩阵，选取与之最近的预定义典范动作，并转换为文本条件输入——无需引入额外可学习模块，即可在预训练 I2V 基础模型上实现键盘控制。Existing methods rely on per-frame absolute camera matrices (c2w) for control, which is expensive to annotate and unfriendly to pretrained models. QCM quantizes continuous camera trajectories into 8 discrete actions (forward, backward, left, right, turn-left, turn-right, tilt-up, tilt-down): it computes the relative transformation matrix between adjacent frames, picks the closest predefined canonical action, and converts it into a text condition input — enabling keyboard control on a pretrained I2V foundation model without introducing any extra learnable module.

2. Masked Video Diffusion Transformers (MVDT)

MVDT 在去噪过程中对 30% 的 token 进行 selective masking，分三阶段处理：Encoder 仅处理可见 token，Side-interpolator 通过可学习 token + self-attention 预测被遮盖内容，Decoder 对插值特征进行精化。该设计将计算集中于可见区域，同时保持时序一致性。配合基于 FramePack 思路的记忆压缩方案（近帧 (1,2,2)、远帧 (1,8,8) 不同压缩比），实现自回归无限视频生成。During denoising MVDT applies selective masking to 30% of the tokens, processed in three stages: the Encoder handles visible tokens only, the Side-interpolator predicts the masked content with learnable tokens plus self-attention, and the Decoder refines the interpolated features. The design concentrates computation on visible regions while preserving temporal consistency. Combined with a FramePack-inspired memory compression scheme (different compression ratios for near frames (1,2,2) and far frames (1,8,8)), it enables autoregressive infinite video generation.

3. Training-Free Anti-Artifact Mechanism (AAM)

AAM 是一个无需额外训练的两阶段去噪机制：首先进行标准去噪生成初始结构 z_orig，再利用 Gaussian blur 滤波将低频分量（整体结构）保留自初始生成，而高频分量（细节纹理）由精化步骤重新生成，从而在不引入训练开销的前提下大幅减少城市建筑场景中的视觉伪影。AAM is a two-stage denoising mechanism that requires no extra training: standard denoising first produces the initial structure z_orig, then a Gaussian blur filter keeps the low-frequency components (overall structure) from that initial generation while the high-frequency components (fine texture) are regenerated by a refinement step, greatly reducing visual artifacts in urban architectural scenes without any training overhead.

4. Time Travel Sampling based on SDE (TTS-SDE)

受 DDNM 和 OSV 启发，TTS-SDE 利用后期去噪阶段的信息引导前期步骤，同时注入随机微分方程噪声以增强采样随机性，从而提升文本/动作可控性。消融实验证明，相比标准 ODE 采样，TTS-SDE 将 Instruction Following 从 0.657 提升至 0.743。Inspired by DDNM and OSV, TTS-SDE uses information from the later denoising stages to guide the earlier steps while injecting stochastic differential equation noise to strengthen sampling randomness, thereby improving text/action controllability. Ablations show that, compared with standard ODE sampling, TTS-SDE lifts Instruction Following from 0.657 to 0.743.

5. 模型加速Model Acceleration

通过对抗蒸馏将采样步数从 50 步压缩至 14 步，推理时间从 583.1 秒降至 158.8 秒（3.7× 加速），视觉质量指标几乎不变（Imaging Quality 0.739 保持不变，Subject Consistency 仅下降 0.005）。此外还引入基于 Cache 的加速机制协同优化。Adversarial distillation compresses the sampling schedule from 50 steps to 14 steps, cutting inference time from 583.1 s to 158.8 s (3.7× speedup) with virtually unchanged visual quality metrics (Imaging Quality stays at 0.739, Subject Consistency drops by only 0.005). A Cache-based acceleration mechanism is additionally introduced for joint optimization.

03 实验Experiments

训练数据：Sekai-Real-HQ（来自 YouTube 行走视频 + 无人机视频，经过镜头检测、亮度过滤、轨迹过滤后精选 400 小时，含 139,019 个标注片段）。评测基准 Yume-Bench 覆盖六项指标：Instruction Following（人工评测）、Subject Consistency、Background Consistency、Motion Smoothness、Aesthetic Quality、Imaging Quality（后五项来自 VBench）。测试分辨率 544 × 960，帧率 16 FPS，96 帧，50 步推理。Training data: Sekai-Real-HQ (YouTube walking videos plus drone footage, curated down to 400 hours after shot detection, brightness filtering, and trajectory filtering, containing 139,019 annotated clips). The Yume-Bench benchmark covers six metrics: Instruction Following (human evaluation), Subject Consistency, Background Consistency, Motion Smoothness, Aesthetic Quality, and Imaging Quality (the last five taken from VBench). Testing uses a resolution of 544 × 960 at 16 FPS, 96 frames, and 50 inference steps.

Table 2：与 SOTA 方法的对比（质量对比）Table 2: Comparison with SOTA Methods (Quality)

Yume 的 Instruction Following（0.657）大幅领先两个 baseline，同时在 Subject Consistency、Background Consistency、Motion Smoothness、Aesthetic Quality 上均达到最优或接近最优，仅在 Imaging Quality 上略低于 MatrixGame（0.739 vs. 0.750）。Yume's Instruction Following (0.657) leads both baselines by a wide margin, and it is best or near-best on Subject Consistency, Background Consistency, Motion Smoothness, and Aesthetic Quality; only on Imaging Quality is it slightly below MatrixGame (0.739 vs. 0.750).

长视频生成验证Long Video Generation Validation

在 18 秒的长视频生成测试中（每次自回归生成 2 秒片段，共 9 次外推），前 8 秒运动与测试集一致，后 10 秒切换为持续前进（W 键）。结果显示：subject consistency 仅下降 0.5%，表明 Yume 的自回归生成机制在长时序下具备良好的时序连贯性。In an 18-second long-video generation test (each autoregressive pass produces a 2-second clip, 9 extrapolations in total), the motion of the first 8 seconds follows the test set while the last 10 seconds switch to continuous forward motion (the W key). The result: subject consistency drops by only 0.5%, showing that the autoregressive generation mechanism of Yume keeps good temporal coherence over long horizons.

消融实验Ablation Studies

Table 3：不同采样器对比（TTS-SDE 消融）Table 3: Sampler Comparison (TTS-SDE Ablation)

TTS-SDE 将 Instruction Following 从 0.657（ODE baseline）提升至 0.743，代价是 Subject Consistency 和 Aesthetic Quality 略有下降。这表明 TTS-SDE 通过策略性引入噪声扰动，增强了运动轨迹的精化效果。TTS-SDE lifts Instruction Following from 0.657 (the ODE baseline) to 0.743, at the cost of slight drops in Subject Consistency and Aesthetic Quality. This indicates that by strategically injecting noise perturbations TTS-SDE strengthens the refinement of motion trajectories.

Table 4：模型蒸馏消融（50 步 → 14 步）Table 4: Model Distillation Ablation (50 steps → 14 steps)

对抗蒸馏将推理时间压缩约 3.7×（583.1s → 158.8s），视觉质量几乎无损，仅 Instruction Following 下降 0.1（0.657 → 0.557），作者认为这可能因步数减少削弱了文本控制能力。Adversarial distillation compresses inference time by roughly 3.7× (583.1s → 158.8s) with almost no loss of visual quality; only Instruction Following falls by 0.1 (0.657 → 0.557), which the authors attribute to the reduced step count weakening text controllability.

04 局限性Limitations