World Modeling in Condition Space for Action Generation
Yue Su, Sijin Chen, Haixin Shi, Mingyu Liu, Zhengshen Zhang, Ningyuan Huang, Weiheng Zhong, Zhengbang Zhu, Yuxiao Liu, Xihui Liu
WoG 将未来观测映射到紧凑的"条件向量空间"，注入 VLA 动作推理流水线，在无需完整视频重建的前提下为精细操控提供预测性引导，同时支持从无标注人类操作视频中高效学习。WoG maps future observations into a compact "condition vector space" and injects it into the VLA action inference pipeline, providing predictive guidance for fine-grained manipulation without full video reconstruction, while also enabling efficient learning from unlabeled human manipulation videos.
arXiv 2026.02SimplerEnv + Real Robot两阶段训练Two-stage training📄 arXiv:2602.22010
现有 VLA 模型在"未来预测"与"精细动作生成"之间存在根本矛盾：World Action Models 显式预测未来图像/视频，信息冗余大，难以应用于现实操控；Latent Action Models 将未来压缩为稀疏隐变量，精度不足，无法支撑需要厘米级控制的任务。Existing VLA models face a fundamental tension between "future prediction" and "fine-grained action generation": World Action Models explicitly predict future images/videos, which carry heavy redundant information and are hard to apply to real-world manipulation; Latent Action Models compress the future into sparse latent variables, whose precision is insufficient to support tasks requiring centimeter-level control.
"WoG maps future observations into compact conditions by injecting them into the action inference pipeline." ——论文摘要"WoG maps future observations into compact conditions by injecting them into the action inference pipeline." — Paper abstract
图 1：WoG 核心思路对比。左侧：WoG 先将未来观测压缩为条件向量，注入动作推理流水线（Stage I）；右侧：Stage II 阶段将条件预测与动作预测解耦，使模型在推理时仅需当前观测即可完成动作生成，无需访问真实未来帧。Figure 1: Comparison of the core idea behind WoG. Left: WoG first compresses future observations into condition vectors and injects them into the action inference pipeline (Stage I). Right: Stage II decouples condition prediction from action prediction, so that at inference the model only needs the current observation to generate actions, without access to ground-truth future frames.
78.0%Google Robot 平均成功率（WoG vs. 59.2% Moto）Google Robot average success rate (WoG vs. 59.2% Moto)
63.5%WidowX 平均成功率（WoG vs. 62.5% ViPRA）WidowX average success rate (WoG vs. 62.5% ViPRA)
+42%引入 UMI 人类数据后 Pick & Place 提升Gain on Pick & Place after adding UMI human data
+33%引入 UMI 人类数据后 Fold Towel 提升Gain on Fold Towel after adding UMI human data
02方法Method
WoG 采用两阶段训练范式：Stage I 以真实未来帧为监督信号训练条件提取器；Stage II 冻结条件空间定义，训练 VLM 同步预测条件与动作，推理时不再依赖未来帧。WoG adopts a two-stage training paradigm: Stage I trains the condition extractor with ground-truth future frames as the supervision signal; Stage II freezes the definition of the condition space and trains the VLM to predict conditions and actions jointly, so that inference no longer relies on future frames.
图 2：WoG 系统总览。Stage I（左）：冻结的 DINOv2 与 Wan VAE 分别提取未来帧的语义特征与生成特征，可训练 Q-Former 将其查询压缩为 N=16 个条件 token（维度 D=32），DiT 动作头以当前观测 + 条件 token 生成动作。Stage II（右）：冻结 Q-Former 与视觉编码器，VLM 同时预测条件向量（余弦相似度损失）与动作（flow-matching 损失），推理时不需要未来帧。Figure 2: Overview of the WoG system. Stage I (left): a frozen DINOv2 and a frozen Wan VAE respectively extract semantic and generative features from future frames, and a trainable Q-Former queries and compresses them into N=16 condition tokens (dimension D=32); the DiT action head generates actions from the current observation plus the condition tokens. Stage II (right): with the Q-Former and the vision encoders frozen, the VLM simultaneously predicts the condition vectors (cosine similarity loss) and the actions (flow-matching loss), requiring no future frames at inference.
Stage I — World Guidance（条件提取与动作生成）Stage I — World Guidance (Condition Extraction and Action Generation)
当前帧由 Prismatic VLM backbone 编码；未来帧由冻结的 DINOv2（语义特征）和 Wan VAE（生成特征）双路编码。可训练的 Q-Former 对两路特征进行跨注意力查询，得到 D=32 维条件向量。DiT 动作头以当前观测和条件向量为条件，通过 flow-matching 目标生成 16 步动作序列。训练损失：The current frame is encoded by the Prismatic VLM backbone; future frames are encoded by two branches, a frozen DINOv2 (semantic features) and a frozen Wan VAE (generative features). A trainable Q-Former cross-attends over both feature streams to obtain D=32 dimensional condition vectors. Conditioned on the current observation and the condition vectors, the DiT action head generates a 16-step action sequence through a flow-matching objective. Training loss:
LI = Eτ,A [‖vθ(Aτ, τ, z, Oc) − v*‖²₂]
默认超参数：预测视野 16 步动作、均匀采样 4 帧未来观测、N=16 个可学习查询。Default hyperparameters: a prediction horizon of 16 action steps, 4 uniformly sampled future observation frames, and N=16 learnable queries.
Stage II — World Inference（条件预测与动作解耦）Stage II — World Inference (Decoupling Condition Prediction from Actions)
Stage II 冻结 Q-Former 和视觉基础模型（定义稳定的目标条件空间），训练 VLM 同时输出：(1) 预测的条件向量（使用余弦相似度损失与 Q-Former 输出对齐）；(2) 动作（flow-matching）。联合损失：Stage II freezes the Q-Former and the vision foundation models (which define a stable target condition space) and trains the VLM to output both: (1) the predicted condition vectors (aligned with the Q-Former output through a cosine similarity loss); (2) the actions (flow-matching). Joint loss:
推理时 VLM 直接从当前帧预测条件向量，无需真实未来帧。At inference the VLM predicts the condition vectors directly from the current frame, with no need for ground-truth future frames.
图 5：Q-Former 查询机制详解。左：Future Encoder 中 N=16 个可学习 queries 通过 cross-attention 提取 DINOv2+Wan VAE 双路特征，压缩为低维、动作相关的条件表示；右：Stage II 中 VLM 如何以条件为监督目标进行自预测，并与 DiT 动作头联合优化。Figure 5: A closer look at the Q-Former query mechanism. Left: in the Future Encoder, N=16 learnable queries extract the DINOv2+Wan VAE dual-branch features via cross-attention and compress them into a low-dimensional, action-relevant condition representation. Right: in Stage II, how the VLM performs self-prediction with the conditions as supervision targets and is jointly optimized with the DiT action head.
扩展至人类操作视频Extension to Human Manipulation Videos
WoG 提供两种策略将无标注人类视频纳入训练：(1) Stage I 使用少量带动作标注的人类子集，Stage II 用无标注大规模视频；(2) 直接在 Stage II 对无标注人类视频进行训练（无需动作标签）。两者均利用 Stage II 的余弦相似度损失对条件向量进行自监督，从而吸收人类操作知识。WoG offers two strategies for bringing unlabeled human videos into training: (1) Stage I uses a small human subset with action annotations while Stage II uses large-scale unlabeled videos; (2) training directly on unlabeled human videos in Stage II (no action labels needed). Both exploit the cosine similarity loss of Stage II to self-supervise the condition vectors and thereby absorb human manipulation knowledge.
03实验Experiments
在 SimplerEnv 仿真环境（Google Robot + WidowX）和真实机器人平台（Close Microwave / Pick & Place / Fold Towel）上与 Moto、ViPRA、UniVLA、VPP、DeFI 等 baseline 进行对比。成功率（%）为主要指标。Comparisons against baselines such as Moto, ViPRA, UniVLA, VPP and DeFI are carried out in the SimplerEnv simulation environments (Google Robot + WidowX) and on a real robot platform (Close Microwave / Pick & Place / Fold Towel). Success rate (%) is the main metric.
SimplerEnv 仿真结果SimplerEnv Simulation Results
模型Model
Google Robot 平均Google Robot avg.
WidowX 平均WidowX avg.
DeFI
51.2%
—
Moto
59.2%
—
UniVLA
—
45.6%
ViPRA
—
62.5%
WoG（ours）WoG (ours)
78.0%
63.5%
Google Robot 子任务：Pick Coke 89.0%（WoG）vs. 74.0%（Moto）；Move Near 82.5% vs. 60.4%；Open/Close Drawer 62.5% vs. 43.1%。Google Robot subtasks: Pick Coke 89.0% (WoG) vs. 74.0% (Moto); Move Near 82.5% vs. 60.4%; Open/Close Drawer 62.5% vs. 43.1%.
真实机器人实验Real-Robot Experiments
图 3：真实机器人实验平台。左：机械臂与传感器配置；中：分布内（In-Distribution）三项任务执行过程；右：分布外（Out-of-Distribution）场景，包括背景变换、光照变化和新物体。Figure 3: The real-robot experimental platform. Left: the robot arm and sensor setup. Middle: execution of the three in-distribution tasks. Right: out-of-distribution scenes, including background changes, lighting variation and novel objects.
任务Task
UniVLA（ID）UniVLA (ID)
VPP（ID）VPP (ID)
WoG（ID）WoG (ID)
Close Microwave
80%
90%
100%
Pick and Place
25%
55%
60%
Fold Towel
20%
45%
60%
分布外（OOD）鲁棒性：Pick & Place 新物体场景 WoG 60→40%，VPP 55→15%；Fold Towel 光照变化 WoG 60→35%，VPP 45→20%。WoG 在多种 OOD 设置下均保持更高鲁棒性。Out-of-distribution (OOD) robustness: on Pick & Place with novel objects WoG goes 60→40% while VPP goes 55→15%; on Fold Towel under lighting changes WoG goes 60→35% while VPP goes 45→20%. WoG retains higher robustness across a range of OOD settings.
Ablations
Future Encoder 消融（Stage I）：带 Future Encoder 的 WoG 在 Google Robot 达到 82.0%，移除后降至 75.1%；WidowX 86.4% vs. 75.0%。Future Encoder ablation (Stage I): WoG with the Future Encoder reaches 82.0% on Google Robot and falls to 75.1% without it; on WidowX, 86.4% vs. 75.0%. 训练阶段消融：完整两阶段 WoG（ID：100% / 60% / 60%）显著优于 WoG w/o cotrain（95% / 45% / 30%）和 Vanilla VLA（90% / 45% / 40%）。Training-stage ablation: The full two-stage WoG (ID: 100% / 60% / 60%) clearly outperforms WoG w/o cotrain (95% / 45% / 30%) and Vanilla VLA (90% / 45% / 40%). 编码器配置：WoG (dino-vae) 在轨迹规划任务（Google Robot）得 70.9%，WoG (dino-siglip) 在空间精度任务（WidowX）得 63.5%，各有侧重。Encoder configuration: WoG (dino-vae) scores 70.9% on the trajectory-planning tasks (Google Robot) while WoG (dino-siglip) scores 63.5% on the spatial-precision tasks (WidowX); each has its own emphasis.
图 4：引入 UMI 人类操作数据后的性能提升。相比仅使用机器人数据，Pick & Place 成功率从 60% 提升至 85%（+42%），Fold Towel 从 60% 提升至 80%（+33%）。Figure 4: Performance gains after introducing UMI human manipulation data. Compared with using robot data alone, the Pick & Place success rate rises from 60% to 85% (+42%) and Fold Towel from 60% to 80% (+33%).
04局限性Limitations
说明：以下局限性均由论文作者明确陈述于正文 Discussion 及 Conclusion 部分。Note: All limitations below are explicitly stated by the authors in the Discussion and Conclusion sections of the paper.
WoG 在需要精确相对位置估计的任务（如精细叠放、抽屉开关）上提升幅度"comparatively smaller"，作者归因于"limited spatial resolution of the current backbone and inherent difficulty of modeling fine-grained geometry through current dynamic prediction alone"。On tasks that demand accurate relative position estimation (such as fine-grained stacking or opening and closing drawers), the gains of WoG are "comparatively smaller"; the authors attribute this to the "limited spatial resolution of the current backbone and inherent difficulty of modeling fine-grained geometry through current dynamic prediction alone".
人类数据迁移的任务依赖性（Task-Dependent Human Data Transfer）Task-Dependent Human Data Transfer
人类视频对 Pick & Place 任务有明显提升，但对形变物体操控（Fold Towel）效果不稳定，说明人机操作的相似性存在任务依赖性，不能简单假设人类数据在所有任务上均有益。Human videos yield a clear improvement on the Pick & Place task, but their effect on deformable-object manipulation (Fold Towel) is unstable, showing that the similarity between human and robot manipulation is task-dependent and that human data cannot simply be assumed beneficial on every task.
条件表示的表达能力上限（Expressive Power of Condition Representation）Expressive Power of Condition Representation
作者在结论中指出，未来工作应聚焦于"more expressive and efficient condition representations to better handle scenarios with strong spatial or action constraints"，暗示当前 D=32 维条件向量在强几何约束场景下存在信息瓶颈。In the conclusion the authors point out that future work should focus on "more expressive and efficient condition representations to better handle scenarios with strong spatial or action constraints", hinting that the current D=32 dimensional condition vectors form an information bottleneck under strong geometric constraints.
