首个以真实人体运动为条件的第一人称世界模拟器The first egocentric world simulator conditioned on real human motion
Yuanpeng Tu, Hao Luo, Xi Chen, Xiang Bai, Fan Wang, Hengshuang Zhao
PlayerOne 以单张第一人称（egocentric）图像和同步外部视角的人体运动序列为输入，生成与人体动作精确对齐的沉浸式视频。
核心创新在于 Part-Disentangled Motion Injection (PMI) 机制和 Scene-Frame Reconstruction (SR) 框架，
在 Diffusion Transformer 基座上实现粗到精（coarse-to-fine）的训练策略，显著优于 Cosmos、Aether 等现有方法。PlayerOne takes a single egocentric image together with a synchronized exocentric human motion sequence as input, and generates immersive video precisely aligned with the human motion. The core innovations are the Part-Disentangled Motion Injection (PMI) mechanism and the Scene-Frame Reconstruction (SR) framework, which realize a coarse-to-fine training strategy on top of a Diffusion Transformer backbone and clearly outperform existing methods such as Cosmos and Aether.
egocentric world simulator自我中心视角生成egocentric view generationdiffusion transformermotion injectionSMPL 人体姿态SMPL human pose视频生成video generation具身智能embodied intelligencescene consistency
01动机Motivation
现有世界模拟器要么局限于游戏环境（运动方式受限），要么缺乏精细的人体运动控制——玩家只能是"被动旁观者"而非"主动参与者"。
真实、沉浸、无限制的第一人称世界模拟尚未得到充分探索。Existing world simulators are either confined to game environments (with restricted modes of motion) or lack fine-grained human motion control — the player can only be a "passive observer" rather than an "active participant". Realistic, immersive and unrestricted egocentric world simulation remains underexplored.
"We introduce PlayerOne, the first egocentric realistic world simulator, facilitating immersive and unrestricted exploration within vividly dynamic environments."
图1：PlayerOne 系统总览。
输入为单张第一人称场景图像，以及包含身体、头部、双手运动的外部视角 SMPL 序列（用户已匿名化），
输出为与真实人体运动精确同步的沉浸式第一人称视频。Figure 1: Overview of the PlayerOne system. The input is a single egocentric scene image together with an exocentric SMPL sequence covering body, head and hand motion (the user is anonymized); the output is an immersive egocentric video precisely synchronized with the real human motion.
DINO 67.8视觉一致性（↑ vs Cosmos-14B 51.6）Visual consistency (↑ vs Cosmos-14B 51.6)
FVD 226视频质量（↓ vs Cosmos-14B 302）Video quality (↓ vs Cosmos-14B 302)
MPJPE 127关节误差（↓ vs Cosmos-14B 257）Joint error (↓ vs Cosmos-14B 257)
PlayerOne 以 Wanx2.1（1.3B Diffusion Transformer）为基座，引入两个核心模块：
Part-Disentangled Motion Injection (PMI) 用于精确的分部位运动控制，
Scene-Frame Reconstruction (SR) 用于保持长视频中的场景一致性。
训练采用粗到精的两阶段策略，先在大规模 egocentric 文本-视频对上做 LoRA 预训练，再冻结 LoRA 仅微调最后六个 block。PlayerOne builds on Wanx2.1 (a 1.3B Diffusion Transformer) and introduces two core modules: Part-Disentangled Motion Injection (PMI) for precise part-wise motion control, and Scene-Frame Reconstruction (SR) for preserving scene consistency in long videos. Training follows a coarse-to-fine two-stage strategy: LoRA pre-training on large-scale egocentric text-video pairs, then freezing the LoRA and fine-tuning only the last six blocks.
图2：PlayerOne 整体框架。
输入图像转为视觉 token；人体运动分解为身体/脚（66维）、头部（3维）、双手（各45维）三组，
分别经独立 3D 卷积编码器处理。头部参数经 Rodrigues 公式转为相机旋转序列，注入 noised 视频 latent。
场景帧重建（SR）通过 CUT3R 渲染 point map 序列，与视频 latent 联合去噪，推理时无需 point map。Figure 2: Overall framework of PlayerOne. The input image is converted into visual tokens; the human motion is decomposed into three groups — body/feet (66 dims), head (3 dims) and hands (45 dims each) — each processed by a separate 3D convolutional encoder. The head parameters are turned into a camera rotation sequence via the Rodrigues formula and injected into the noised video latent. Scene-Frame Reconstruction (SR) renders point map sequences with CUT3R and denoises them jointly with the video latent; no point map is needed at inference time.
Part-Disentangled Motion Injection (PMI)
传统方案将全身运动拼接为单一向量（"entangled"）送入 ControlNet，导致不同部位（尤其是头部视角与手部动作）互相干扰。
PMI 将运动分为三组：Conventional approaches concatenate whole-body motion into a single "entangled" vector fed to ControlNet, so that different parts — especially head viewpoint and hand motion — interfere with each other. PMI splits the motion into three groups:
Body/Feet（66维）：专用 3D 卷积编码器，捕捉步伐与全身姿态变化。Body/Feet (66 dims): A dedicated 3D convolutional encoder captures gait and whole-body pose changes.
Head（3维旋转）：通过 Rodrigues 公式转换为相机旋转序列，额外注入 noised 视频 latent，精确控制第一人称视角转动。Head (3-dim rotation): Converted into a camera rotation sequence via the Rodrigues formula and additionally injected into the noised video latent, giving precise control over egocentric viewpoint rotation.
Hands（各45维）：独立编码器处理双手精细动作，用于物体交互场景。Hands (45 dims each): Separate encoders handle fine-grained motion of the two hands, used for object-interaction scenarios.
消融实验表明，PMI 相比 ControlNet 基线将 MPJPE 从 241.73 降至 156.76，FVD 从 287.52 降至 245.72。Ablations show that, compared with the ControlNet baseline, PMI reduces MPJPE from 241.73 to 156.76 and FVD from 287.52 to 245.72.
Scene-Frame Reconstruction (SR)
为避免长视频中场景漂移，SR 在训练时用 CUT3R 从真值视频渲染 point map 序列，
通过带 5层 3D 卷积 adapter 的专用编码器与视频 latent 联合去噪，
迫使模型在时序上维持三维场景一致性。推理阶段不需要 point map 输入。To avoid scene drift in long videos, SR renders point map sequences from ground-truth videos with CUT3R during training and denoises them jointly with the video latent through a dedicated encoder equipped with a 5-layer 3D convolutional adapter, forcing the model to maintain 3D scene consistency over time. No point map input is required at the inference stage.
图3：自动数据集构建流水线。
利用 SAM2 检测、SMPLest-X 提取 SMPL 参数，过滤重投影误差最高的 10% 样本，
从大规模 egocentric-exocentric 配对数据集中构建高质量运动-视频配对，
最终用于 PMI 模块的精调训练。Figure 3: Automatic dataset construction pipeline. SAM2 is used for detection and SMPLest-X for extracting SMPL parameters; the 10% of samples with the highest reprojection error are filtered out, and high-quality motion-video pairs are built from large-scale egocentric-exocentric paired datasets, finally used for fine-tuning the PMI module.
Coarse-to-Fine 训练策略Coarse-to-Fine Training Strategy
阶段一（粗粒度）：在 Egovid-5M 等大规模 egocentric 文本-视频对上做 LoRA（rank 128，weight 4）预训练，
使模型具备第一人称视觉先验。 阶段二（细粒度）：冻结 LoRA，仅对最后六个 Transformer block 在精标的运动-视频配对数据上微调，
建立精确的运动-视觉对应关系。
此外，采用 Asymmetric Distillation（因果学生由双向教师监督）将推理步数压缩至 50 步，输出帧率保持 8 FPS。Stage 1 (coarse): LoRA pre-training (rank 128, weight 4) on large-scale egocentric text-video pairs such as Egovid-5M, giving the model an egocentric visual prior. Stage 2 (fine): The LoRA is frozen and only the last six Transformer blocks are fine-tuned on carefully annotated motion-video pairs, establishing a precise motion-vision correspondence.
In addition, Asymmetric Distillation (a causal student supervised by a bidirectional teacher) compresses inference to 50 steps, while the output frame rate stays at 8 FPS.
03实验Experiments
评估在自建 egocentric-exocentric 配对测试集上进行，基线包括 Aether、Cosmos（Diff-7B / Diff-14B）。
指标：DINO-Score（视觉一致性）、CLIP-Score（语义匹配）、MPJPE / MRRPE（运动对齐，单位 mm）、FVD（视频质量）、LPIPS（感知质量）。
用户研究由 20 名标注员对 25 组视频进行 1–4 分盲评（Quality / Fidelity / Smoothness / Alignment）。Evaluation is carried out on a self-built egocentric-exocentric paired test set, with Aether and Cosmos (Diff-7B / Diff-14B) as baselines. Metrics: DINO-Score (visual consistency), CLIP-Score (semantic matching), MPJPE / MRRPE (motion alignment, in mm), FVD (video quality) and LPIPS (perceptual quality). The user study has 20 annotators blindly rate 25 groups of videos on a 1-4 scale (Quality / Fidelity / Smoothness / Alignment).
与 State-of-the-Art 对比（Table 3）Comparison with the State of the Art (Table 3)
方法
Method
DINO-Score ↑
CLIP-Score ↑
MPJPE ↓
MRRPE ↓
FVD ↓
LPIPS ↓
Aether
38.0
64.2
415.70
431.05
397.40
0.1856
Cosmos (Diff-7B)
45.3
70.3
301.92
324.12
346.09
0.1630
Cosmos (Diff-14B)
51.6
79.7
256.73
253.06
302.17
0.1351
PlayerOne（本文）PlayerOne (Ours)
67.8
88.2
127.16
151.62
226.12
0.0663
用户研究（Table 4，20位标注员，1–4分）User Study (Table 4, 20 annotators, 1-4 scale)
方法
Method
Quality ↑
Fidelity ↑
Smoothness ↑
Alignment ↑
Aether
1.32
1.30
1.31
1.34
Cosmos (7B)
2.07
2.13
2.05
2.09
Cosmos (14B)
3.02
2.94
2.98
2.71
PlayerOne
3.59
3.63
3.65
3.86
图7：定性结果。
以相同首帧图像搭配不同人体运动序列为条件，PlayerOne 能生成各具差异、动作精确对齐的第一人称视频，
且真实地体现了人与环境的交互细节（如手持物体、行走时的视角晃动）。Figure 7: Qualitative results. Conditioned on the same first-frame image paired with different human motion sequences, PlayerOne generates distinct egocentric videos whose motion is precisely aligned, and faithfully reflects interaction details between the person and the environment (such as holding objects, or viewpoint shake while walking).图8：与基线方法的定性对比。
PlayerOne 在运动对齐精度和视频视觉质量上均显著优于 Cosmos 和 Aether，
尤其在视角转变和手部交互等细粒度动作上更为突出。Figure 8: Qualitative comparison with baseline methods. PlayerOne clearly outperforms Cosmos and Aether in both motion alignment accuracy and visual video quality, and the gap is most pronounced on fine-grained actions such as viewpoint changes and hand interactions.
消融实验（Table 2，关键组件贡献）Ablation Study (Table 2, contribution of key components)
逐步累加各模块，DINO-Score 从基线 51.3 提升至 67.8，MPJPE 从 376.14 降至 127.16：Adding the modules one by one raises DINO-Score from a baseline of 51.3 to 67.8 and lowers MPJPE from 376.14 to 127.16:
配置
Configuration
DINO ↑
CLIP ↑
MPJPE ↓
FVD ↓
LPIPS ↓
Baseline
51.3
65.6
376.14
394.16
0.1421
+ Pretrain
56.6
74.4
258.05
301.32
0.1146
+ ControlNet
57.1
75.2
241.73
287.52
0.1103
+ PMI (No Camera)
60.7
79.8
183.25
257.04
0.0902
+ PMI（含相机编码）+ PMI (with camera encoding)
62.5
81.3
156.76
245.72
0.0839
+ Filtering
64.2
83.8
141.56
230.50
0.0782
+ Recon (DUSt3R)
67.5
87.7
129.08
228.20
0.0685
PlayerOne Full (CUT3R)
67.8
88.2
127.16
226.12
0.0663
关键发现：PMI 中相机编码器对视角控制不可或缺（去掉后 MPJPE 从 156.76 上升至 183.25）；
CUT3R 的 point map 质量优于 DUSt3R，FVD 从 228.20 进一步降至 226.12；
数据过滤（排除重投影误差最高 10%）对运动对齐有显著正效果。Key findings: the camera encoder inside PMI is indispensable for viewpoint control (removing it raises MPJPE from 156.76 to 183.25); the point map quality of CUT3R is better than that of DUSt3R, lowering FVD further from 228.20 to 226.12; and data filtering (discarding the 10% of samples with the highest reprojection error) has a clearly positive effect on motion alignment.
04局限性Limitations
说明：以下局限性由论文作者在 Conclusion 部分明确陈述，标注为 stated；
其余为从系统设计中推断，标注为 inferred。Note: The limitations below that are explicitly stated by the authors in the Conclusion section are marked stated; the rest are inferred from the system design and marked inferred.
游戏场景表现较差（stated）Weaker performance in game scenes (stated)
模型在游戏风格场景下的生成质量明显弱于真实场景，作者归因于
"imbalanced distribution between realistic and game training data"。
解决方向是扩大游戏场景训练数据的比例。Generation quality in game-style scenes is clearly weaker than in realistic scenes; the authors attribute this to the "imbalanced distribution between realistic and game training data". The proposed direction is to increase the proportion of game-scene training data.
运动数据依赖外部视角同步采集（inferred）Motion data depends on synchronized exocentric capture (inferred)
精调阶段需要 egocentric-exocentric 同步配对视频（内外视角同时录制），
采集成本较高，限制了训练数据规模；自动化流水线虽已过滤质量较低样本，但仍依赖 SMPLest-X 和 CUT3R 的估计精度。The fine-tuning stage requires synchronized egocentric-exocentric paired videos (both viewpoints recorded at the same time), which is costly to collect and limits the scale of the training data; although the automatic pipeline already filters out lower-quality samples, it still depends on the estimation accuracy of SMPLest-X and CUT3R.
推理不支持 point map 输入（设计取舍，inferred）Inference does not accept point map input (design trade-off, inferred)
SR 模块在训练时利用真值 point map 进行场景对齐，但推理时无法获得真值 point map，
模型依赖训练时建立的隐式一致性。在极端视角变化或大范围场景中，一致性保障有限。The SR module uses ground-truth point maps for scene alignment during training, but no ground-truth point map is available at inference, so the model relies on the implicit consistency established during training. Under extreme viewpoint changes or in large-scale scenes, this consistency guarantee is limited.
输出分辨率与时长固定（inferred）Fixed output resolution and duration (inferred)
训练固定为 480×480、49帧（约6秒），推理输出同样受此约束。
对于更长时序的探索或高分辨率需求，需要额外的工程扩展（如滑动窗口续帧策略）。Training is fixed at 480×480 and 49 frames (about 6 seconds), and the inference output is bound by the same constraint. Longer-horizon exploration or higher-resolution demands would require additional engineering extensions, such as a sliding-window frame-continuation strategy.
