OSCAR: Omni-Embodiment Action-Conditioned World Model for Robotics

01 动机Motivation

现有视频世界模型在机器人策略真实评估中面临三大核心挑战：训练数据场景多样性不足、动作跟随不精准、以及跨机器人形态泛化能力差。Existing video world models face three core challenges for real-world robot policy evaluation: insufficient scenario diversity in training data, imprecise action following, and poor generalization across robot embodiments.

"Existing video world models face three main challenges for real-world robot evaluation: limited scenario diversity in current robot training datasets, imprecise action following, and poor generalization across embodiments for broad adoption."

OSCAR as real-world policy evaluation proxy on RoboArena — **图 1：OSCAR 作为 RoboArena 的真实世界策略评估代理。** 左：OSCAR 生成的 rollout（上）与 π₀-FAST 策略真实世界 rollout（下）对比；从 episode 中均匀采样三帧。右：七种通用机器人策略在 RoboArena 上的平均成功率——使用 OSCAR 世界模型评估结果与真实评估结果呈现强相关性。**Figure 1: OSCAR as a real-world policy evaluation proxy for RoboArena.** Left: rollouts generated by OSCAR (top) compared with real-world rollouts of the π₀-FAST policy (bottom); three frames uniformly sampled from each episode. Right: average success rates of seven generalist robot policies on RoboArena — results obtained with the OSCAR world model correlate strongly with real-world evaluation.

2B模型参数量（vs. 基线最大 14B）Model parameters (vs. up to 14B for baselines)

180K清洗后训练 episode 数量Cleaned training episodes

4覆盖机器人形态数（+ 人手）Robot embodiments covered (+ human hands)

2.165M原始数据源 episode 总量Total episodes in the raw data sources

三大挑战与对应设计Three Challenges and the Corresponding Designs

挑战 I：动作跟随不精准Challenge I: Imprecise Action Following

视频需要在帧级别（何时发生）和像素级别（何处发生）精确跟随动作序列，才能为策略评估提供有意义的信号。Latent-action 方法将动作压缩为隐变量，往往无法精确传达空间运动。Videos must follow the action sequence precisely at both the frame level (when something happens) and the pixel level (where it happens) in order to provide a meaningful signal for policy evaluation. Latent-action methods compress actions into latent variables and often fail to convey spatial motion precisely.

挑战 II：场景泛化不足Challenge II: Insufficient Scenario Generalization

策略评估需要覆盖不同任务、环境和动作序列。现有数据集（如 AgiBot）虽有百万条 clip，却集中在同一场景和任务，视觉多样性极低。Policy evaluation must cover diverse tasks, environments and action sequences. Existing datasets such as AgiBot contain millions of clips, yet they concentrate on the same scene and task and offer extremely low visual diversity.

挑战 III：跨形态泛化能力差Challenge III: Poor Cross-Embodiment Generalization

世界模型应能泛化到不同机器人形态（Franka Panda、KUKA iiwa、AgiBot G1 等），而非绑定单一机器人，才能支持广泛的策略评估应用。A world model should generalize to different robot embodiments (Franka Panda, KUKA iiwa, AgiBot G1, etc.) rather than being tied to a single robot, so that it can support a broad range of policy evaluation applications.

OSCAR 的两大设计支柱The Two Design Pillars of OSCAR

（1）大规模标准化数据流水线：跨 4 种机器人和 2 种人手数据源，经过去冗余、质量过滤，构建覆盖广泛的联合训练数据集。（2）2D 骨骼渲染统一条件：仅依赖运动链，无需机器人外观信息，可跨机器人乃至人手泛化。(1) A large-scale standardized data pipeline: spanning 4 robot and 2 human-hand data sources, with deduplication and quality filtering, to build a broadly covering joint training dataset. (2) 2D skeleton rendering as unified conditioning: it relies on the kinematic chain alone, needs no robot appearance information, and generalizes across robots and even to human hands.

02 方法Method

OSCAR 在 Cosmos-Predict2.5-2B（一个 2B 参数的视频 Diffusion Transformer）基础上微调，以第一帧 RGB 图像和 2D 骨骼渲染序列为条件，自回归生成未来视频帧。OSCAR is fine-tuned from Cosmos-Predict2.5-2B, a 2B-parameter video Diffusion Transformer, and autoregressively generates future video frames conditioned on the first RGB frame and a sequence of 2D skeleton renderings.

OSCAR method overview — **图 2：OSCAR 方法概览。** OSCAR 由三个组件构成：（1）**条件编码（Condition encoding）**：用 VAE 将第一帧 I₀ 和骨骼序列 S₁:T 编码为隐变量；（2）**条件注入（Conditioning injection）**：将骨骼隐变量与带噪视频隐变量叠加；（3）**视频生成（Video generation）**：DiT 对 token 去噪，VAE 解码最终视频。**Figure 2: Overview of the OSCAR method.** OSCAR consists of three components: (1) **Condition encoding**: a VAE encodes the first frame I₀ and the skeleton sequence S₁:T into latents; (2) **Conditioning injection**: the skeleton latents are added to the noisy video latents; (3) **Video generation**: the DiT denoises the tokens and the VAE decodes the final video.

3.1 骨骼渲染作为统一条件表示（Skeleton Rendering as Unified Conditioning）3.1 Skeleton Rendering as Unified Conditioning

选择正确的动作表示是动作条件世界模型的核心。OSCAR 采用 2D 运动学骨骼（kinematic skeleton）渲染，在泛化性与精度之间取得平衡：Choosing the right action representation is central to an action-conditioned world model. OSCAR adopts 2D kinematic skeleton rendering, which strikes a balance between generalization and precision:

泛化性：骨骼渲染仅依赖运动链（kinematic chain），改变机器人形态只需更新运动学规格；同一表示可用于不同机器人臂或人手。无纹理的渲染避免了模型过拟合到特定机器人的外观。Generalization: skeleton rendering depends only on the kinematic chain; changing the robot embodiment merely requires updating the kinematics specification, and the same representation serves different robot arms or human hands. Texture-free renderings keep the model from overfitting to the appearance of a particular robot.
精度：对比 latent-action（隐式压缩，精度低）和 pointmap/mesh（过于细致，泛化差），骨骼渲染是"只够用"的几何信息——它显式指导机器人动作，同时将外观细节留给视频 prior。Precision: compared with latent action (implicit compression, low precision) and pointmap/mesh (overly detailed, poor generalization), skeleton rendering carries just enough geometric information — it explicitly guides the robot motion while leaving appearance details to the video prior.
实现方式：给定 URDF 模型 M 与关节配置 q_t，通过正向运动学（FK）得到每个连杆的 SE(3) 位姿 T_{k,t}，将关键点投影到图像平面后光栅化为黑底线图 S_t，作为与目标视频帧对齐的 RGB 条件流输入 DiT。Implementation: given a URDF model M and joint configuration q_t, forward kinematics (FK) yields the SE(3) pose T_{k,t} of every link; keypoints are projected onto the image plane and rasterized into a line drawing S_t on a black background, which is fed to the DiT as an RGB conditioning stream aligned with the target video frames.

3.2 人手扩展（Extension to Human Hands）3.2 Extension to Human Hands

由于 S₁:T 仅编码 2D 关节投影，同样的条件表示可以无缝扩展到人手。MANO 人手模型的五指拓扑与机械臂运动链用同样的光栅化流程处理，使模型可以联合训练机器人和人体手部的 egocentric 视频，显著增加场景、任务和动作分布的多样性。Because S₁:T encodes only 2D joint projections, the same conditioning representation extends seamlessly to human hands. The five-finger topology of the MANO hand model is handled by the same rasterization pipeline as the robot-arm kinematic chain, letting the model train jointly on robot videos and egocentric human-hand videos, which substantially increases the diversity of scenes, tasks and action distributions.

3.3 条件注入（Conditioning Injection）3.3 Conditioning Injection

骨骼序列 S₁:T 经过与目标视频相同的 WAN 2.1 VAE 编码，得到骨骼隐变量 z_s，与带噪视频隐变量通过不同的 patch embedder（PE_v 和 PE_s）嵌入后相加，送入 DiT 进行去噪。训练时以概率 0.2 将 S₁:T 置零，以支持 classifier-free guidance（推理时 guidance scale w=6）。The skeleton sequence S₁:T is encoded by the same WAN 2.1 VAE as the target video, giving skeleton latents z_s, which are embedded by separate patch embedders (PE_v and PE_s), added to the noisy video latents, and fed to the DiT for denoising. During training S₁:T is zeroed out with probability 0.2 to support classifier-free guidance (guidance scale w=6 at inference).

Skeleton overlays on training data sources — **图 3：八个训练数据源上的骨骼覆盖可视化。** 上排（四种机器人录制）：DROID、RH20T-cfg5（Franka Panda）、RH20T-cfg7（KUKA iiwa）、InternData；下排（人形与人手数据）：AgiBot G1、AIROA-MoMa、EgoDex、EPIC-Kitchens。每格展示同一数据源的四个 episode 帧。**Figure 3: Skeleton overlays on the eight training data sources.** Top row (four robot recordings): DROID, RH20T-cfg5 (Franka Panda), RH20T-cfg7 (KUKA iiwa), InternData; bottom row (humanoid and human-hand data): AgiBot G1, AIROA-MoMa, EgoDex, EPIC-Kitchens. Each cell shows four episode frames from the same data source.

3.4 大规模数据流水线3.4 Large-Scale Data Pipeline

从 2,165,359 条原始视频中清洗出 180,657 条高质量 episode，经过四阶段过滤：From 2,165,359 raw videos, 180,657 high-quality episodes are cleaned out through four filtering stages:

长度过滤：每条 clip 至少 70 帧，保证足够的 rollout 时长。Length filtering: every clip must have at least 70 frames, guaranteeing a sufficient rollout duration.
静态相机过滤：剔除相机运动幅度超过阈值的 episode。Static-camera filtering: episodes whose camera motion exceeds a threshold are discarded.
有效动作过滤：每条 episode 必须包含非平凡的机械臂动作序列。Valid-action filtering: every episode must contain a non-trivial robot-arm action sequence.
可见骨骼过滤：骨骼可见比例须高于阈值。Visible-skeleton filtering: the visible fraction of the skeleton must exceed a threshold.
语义去冗余（两阶段）：先用 SigLIP 图像嵌入的余弦相似度（阈值 0.95）聚类视觉相似候选对，再用 64 步重采样轨迹的 RMS 距离验证轨迹相似性，仅当视觉和轨迹均相似时才标记为重复。Semantic deduplication (two stages): visually similar candidate pairs are first clustered by the cosine similarity of SigLIP image embeddings (threshold 0.95), then trajectory similarity is verified by the RMS distance of trajectories resampled to 64 steps; a pair is marked as duplicate only when both the visual and the trajectory similarity hold.

03 实验Experiments

在 200 条机器人操作 clip 组成的自建 benchmark 上评估（来自 Franka Panda、KUKA iiwa、AgiBot G1、Toyota HSR 四种形态），与七个基线对比，指标涵盖 PSNR、SSIM、LPIPS、tLPIPS、FVD、FID、L2_latent 和 FPS，全部在 NVIDIA GH200 GPU 上计时。Evaluation is run on a self-built benchmark of 200 robot manipulation clips (from four embodiments: Franka Panda, KUKA iiwa, AgiBot G1 and Toyota HSR) against seven baselines, with metrics covering PSNR, SSIM, LPIPS, tLPIPS, FVD, FID, L2_latent and FPS, all timed on an NVIDIA GH200 GPU.

5.2 与基线定量对比（Table 2）5.2 Quantitative Comparison with Baselines (Table 2)

方法Method	PSNR ↑	SSIM ↑	LPIPS ↓	tLPIPS ↓	FVD ↓	FID ↓	L2_lat ↓	FPS ↑
Cosmos-Predict2.5	14.78	0.563	0.370	0.022	18.01	47.59	0.435	0.292
TesserAct	16.26	0.730	0.277	0.055	24.50	51.90	0.364	0.343
IRASim	6.48	0.088	0.909	0.606	411.42	394.10	2.453	2.330
Ctrl-World	19.06	0.705	0.321	0.042	28.90	53.33	0.292	1.631
EnerVerse-AC	20.47	0.746	0.223	0.021	33.70	38.23	0.197	1.900
Genie Envisioner	23.29	0.838	0.140	0.007	15.37	22.92	0.129	1.382
Kinema4D	17.68	0.741	0.198	0.021	17.07	37.16	0.233	0.089
OSCAR（本文）OSCAR (ours)	24.24	0.846	0.094	0.015	7.08	15.07	0.096	2.214

粗体为最优，下划线为次优。OSCAR 在七项指标中均达到最优或次优，且仅用 2B 参数，远小于 Kinema4D（14B）。Bold marks the best and underline the second best. OSCAR is best or second best on all seven metrics while using only 2B parameters, far fewer than Kinema4D (14B).

Qualitative comparison across embodiments — **图 4：两种机器人形态（AgiBot G1 和 DROID）上的动作条件视频生成定性对比。** 列依次为：GT（真值）、OSCAR（本文）、Cosmos-P2.5、TesserAct、Ctrl-World、GE-Sim、Kinema4D。与五个基线相比，OSCAR 在保持精准动作跟随的同时，视觉质量显著更优。**Figure 4: Qualitative comparison of action-conditioned video generation on two robot embodiments (AgiBot G1 and DROID).** Columns from left to right: GT (ground truth), OSCAR (ours), Cosmos-P2.5, TesserAct, Ctrl-World, GE-Sim, Kinema4D. Compared with the five baselines, OSCAR achieves markedly better visual quality while keeping action following precise.

5.3 消融实验（Table 3）5.3 Ablation Study (Table 3)

消融两个关键因素：条件表示和数据策略。Two key factors are ablated: the conditioning representation and the data strategy.

维度Dimension	变体Variant	PSNR ↑	SSIM ↑	LPIPS ↓	FVD ↓	FID ↓
条件表示Conditioning representation	Latent action	19.22	0.784	0.170	12.03	26.11
	Mesh rendering	23.11	0.831	0.106	7.89	16.38
	Skeleton（最终）Skeleton (final)	23.48	0.832	0.106	7.69	16.37
数据策略Data strategy	+Human from beginning	23.87	0.842	0.097	7.65	15.72
数据策略Data strategy	+Human, warm-start（最终）+Human, warm-start (final)	24.24	0.846	0.094	7.08	15.07

关键发现：（i）Latent action 无法精准跟随动作；（ii）Mesh 和 Skeleton 在七项指标上统计上无显著差异，但 Mesh 依赖机器人 URDF 外观资产，无法引入人体数据；（iii）人体数据的引入一致提升所有指标，warm-start 策略（先用机器人数据训练再加入人体数据）优于从头混合训练。Key findings: (i) latent action cannot follow actions precisely; (ii) Mesh and Skeleton show no statistically significant difference across the seven metrics, but Mesh relies on the robot's URDF appearance assets and therefore cannot bring in human data; (iii) introducing human data consistently improves every metric, and the warm-start strategy (training on robot data first and adding human data afterwards) outperforms mixed training from scratch.

5.4 策略评估（Policy Evaluation on RoboArena）5.4 Policy Evaluation on RoboArena

将 OSCAR 部署到 RoboArena 公开排行榜，对 7 个通用 DROID 策略（π₀-flow、π₀-FAST、PG-flow、PG-FSQ、PG-FAST、PG-FAST+、PG-Bin）进行评估：在 65 个 session 上自回归展开 OSCAR，用 GPT-5 评估每条 episode 的成功率，计算 Pearson 相关系数 r 和 Spearman ρ。OSCAR is deployed on the public RoboArena leaderboard to evaluate 7 generalist DROID policies (π₀-flow, π₀-FAST, PG-flow, PG-FSQ, PG-FAST, PG-FAST+, PG-Bin): OSCAR is rolled out autoregressively over 65 sessions, GPT-5 rates the success of every episode, and the Pearson correlation coefficient r and Spearman ρ are computed.

条件表示Conditioning representation	MMRV ↓	ρ ↑	r ↑	SISR_Δ (pp) ↓
Latent action	1.429	+0.643	+0.867	1.98
Mesh	0.714	+0.679	+0.781	3.04
Skeleton（本文）Skeleton (ours)	0.571	+0.750	+0.852	1.73

骨骼条件与真实世界部署的相关性最强（MMRV=0.571，Pearson r=+0.852），证明 OSCAR 可有效替代真实机器人评估。Skeleton conditioning correlates most strongly with real-world deployment (MMRV=0.571, Pearson r=+0.852), showing that OSCAR can effectively substitute for real-robot evaluation.

04 局限性Limitations

说明：以下局限性均为作者在论文 Conclusion 节中明确陈述（stated）。Note: every limitation below is explicitly stated by the authors in the Conclusion section of the paper.

相机标定与运动学标注质量受限Camera calibration and kinematic annotation quality are limiting factors

"Our current data scale is limited by the availability and quality of per-dataset camera calibration and kinematic annotations: errors in camera intrinsics/extrinsics directly degrade skeleton–RGB alignment, limiting the availability of raw video data that can be reliably converted into usable training dataset." 即相机内外参数的误差会直接导致骨骼与 RGB 图像的对齐偏差，可用训练数据规模因此受限。That is, errors in the camera intrinsics and extrinsics directly cause misalignment between the skeleton and the RGB image, which limits the amount of raw video that can be turned into usable training data.

模型规模仅为 2B，扩展受限The model is only 2B in size, so scaling is limited

"Our model only uses a 2B-parameter backbone; scaling to larger backbones may further improve fidelity and generalization but requires more compute." 当前 backbone 为 2B 参数，更大规模的 backbone 可能进一步提升保真度和泛化能力，但计算成本相应增加。The current backbone has 2B parameters; a larger backbone could further improve fidelity and generalization, but at a correspondingly higher compute cost.

仅支持静态相机场景Only static-camera scenes are supported

数据过滤阶段主动剔除了相机运动的 episode（"We mainly focus on the robot action in our paper and defer the camera motion for future work"），因此当前模型不支持移动相机场景，限制了对移动操作任务的策略评估能力。The data filtering stage deliberately discards episodes with camera motion ("We mainly focus on the robot action in our paper and defer the camera motion for future work"), so the current model does not support moving-camera scenes, which limits its ability to evaluate policies for mobile manipulation tasks.