World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry
WAV — 利用前向-逆向不对称性让世界模型自我识别误差并持续提升WAV — exploiting forward-inverse asymmetry so world models can spot their own errors and keep improving
Yuejiang Liu, Fan Feng, Lingjing Kong, Weifeng Lu, Jinzhou Tang, Kun Zhang, Kevin Murphy, Chelsea Finn, Yilun Du
通用世界模型（world model）承诺可用于策略评估、优化与规划，但在多样动作分布下保持鲁棒性仍极具挑战。WAV 将动作条件预测分解为"状态合理性（state plausibility）"与"动作可达性（action reachability）"两个可独立验证的因子，通过前向-逆向循环一致性（cycle consistency）自动发现模型预测误差并引导高效探索，无需额外标注。General-purpose world models promise to serve policy evaluation, optimization and planning, yet staying robust under diverse action distributions remains highly challenging. WAV decomposes action-conditioned prediction into two independently verifiable factors, "state plausibility" and "action reachability", and uses forward-inverse cycle consistency to automatically discover the model's prediction errors and guide efficient exploration, with no extra annotation.
world modelinverse dynamicsaction reachabilityself-improvingsample efficiencyrobot learningcycle consistency世界模型自我提升self-improving world models
01动机Motivation
通用世界模型要在策略学习中发挥作用，必须在包括次优动作和探索性动作在内的多样动作分布下稳定准确地预测结果。然而，现有方法在探索不足的状态区域难以识别哪些 transition 最具信息量，验证（verification）机制的缺失导致模型在分布外（OOD）场景下显著退化。For a general-purpose world model to be useful in policy learning, it must predict outcomes stably and accurately under diverse action distributions, including suboptimal and exploratory actions. Existing methods, however, struggle to tell which transitions are most informative in under-explored state regions, and the absence of a verification mechanism makes models degrade markedly in out-of-distribution (OOD) settings.
"General-purpose world models promise scalable policy evaluation, optimization, and planning, yet achieving the required level of robustness remains challenging."
图 1：WAV 框架总览。框架通过前向-逆向不对称循环使动作条件世界模型能够验证自身预测并自我提升：(i) 多样子目标生成器从视频数据中提出合理的未来状态；(ii) 稀疏逆模型（sparse inverse model）从有限状态特征推断动作；(iii) 前向模型检验两者的循环一致性，验证分数用于引导高效探索。Figure 1: Overview of the WAV framework. A forward-inverse asymmetric cycle lets an action-conditioned world model verify its own predictions and self-improve: (i) a diverse subgoal generator proposes plausible future states from video data; (ii) a sparse inverse model infers actions from a limited set of state features; (iii) the forward model checks the cycle consistency of the two, and the verification score is used to guide efficient exploration.
核心洞察：已知起止状态时，推断引发合理 transition 的动作（逆问题）往往比直接预测状态转移结果（前向问题）更容易。这种不对称性来源于：① 无动作视频数据远比带动作的交互数据丰富；② 动作相关特征的维度 d_z 远小于完整状态维度 d_s。WAV 正是利用这一不对称性构建高效的预测验证机制。Key insight: when the start and end states are known, inferring the action that induces a plausible transition (the inverse problem) is often easier than directly predicting the outcome of the state transition (the forward problem). This asymmetry has two sources: first, action-free video data is far more abundant than action-labelled interaction data; second, the dimensionality of action-relevant features d_z is much smaller than the full state dimensionality d_s. WAV builds an efficient prediction-verification mechanism precisely by exploiting this asymmetry.
02方法Method
WAV 框架由三个模块组成，通过前向-逆向循环一致性进行自我验证：子目标生成器 g_ϕ、稀疏逆动力学模型 h_ψ、以及前向世界模型 f_θ。验证分数衡量前向预测与子目标之间的偏差，高分样本优先被选入训练集以引导探索。The WAV framework comprises three modules that perform self-verification through forward-inverse cycle consistency: a subgoal generator g_ϕ, a sparse inverse dynamics model h_ψ, and a forward world model f_θ. The verification score measures the discrepancy between the forward prediction and the subgoal; high-scoring samples are selected into the training set first, so as to guide exploration.
图 2：将世界模型的预测验证分解为两个独立可验证的因子：状态合理性（state plausibility）——预测状态是否视觉上合理，可借助大规模无标注视频数据验证；动作可达性（action reachability）——在给定动作下状态转移是否物理上可行，通过稀疏逆动力学模型验证。Figure 2: Decomposing the verification of world-model predictions into two independently verifiable factors: state plausibility — whether the predicted state is visually plausible, which can be verified with large-scale unlabelled video data; action reachability — whether the state transition is physically feasible under the given action, which is verified by a sparse inverse dynamics model.
子目标生成器（Subgoal Generator）Subgoal Generator
子目标生成器 g_ϕ 建模状态转移先验分布 p(st+1|st)，同时在有动作标注的机器人交互数据和规模更大的无动作视频数据上训练。对给定当前状态 st，它采样 K 个合理未来状态 {s̃kt+1}，这些候选子目标不依赖动作，天然具有视觉合理性（state plausibility）。The subgoal generator g_ϕ models the state-transition prior p(st+1|st), and is trained jointly on action-labelled robot interaction data and on much larger action-free video data. Given a current state st, it samples K plausible future states {s̃kt+1}; these candidate subgoals do not depend on actions and are therefore naturally visually plausible (state plausibility).
稀疏逆动力学模型（Sparse Inverse Dynamics Model）Sparse Inverse Dynamics Model
稀疏 IDM 引入可学习稀疏掩码 M，仅利用与动作相关的紧凑特征子集推断动作：The sparse IDM introduces a learnable sparse mask M and infers the action from only a compact subset of action-relevant features:
at = h_ψ(M ⊙ st, M ⊙ st+1)
"Sparse" 的含义是：模型只关注对动作预测贡献最大的少数状态维度 z（d_z ≪ d_s），从而降低输入维度、提升样本效率与对环境噪声的鲁棒性。"Sparse" means that the model attends only to the few state dimensions z that contribute most to action prediction (d_z ≪ d_s), thereby lowering the input dimensionality and improving sample efficiency and robustness to environment noise.
偏差 ε̂k 衡量前向世界模型的预测与子目标生成器给出的"合理未来"之间的不一致程度，验证分数高（偏差大）的样本优先被加入训练集，引导世界模型在高误差区域自我提升。The discrepancy ε̂k measures how inconsistent the forward world model's prediction is with the "plausible future" given by the subgoal generator; samples with a high verification score (large discrepancy) are added to the training set first, driving the world model to self-improve in high-error regions.
理论支撑（Proposition 3.1）Theoretical Support (Proposition 3.1)
在线性-高斯设定下，若两个模型均用 OLS 在 n 个带标注转移上拟合，当 n > d_s + d_a + 1 且 n > 2d_z + 1 时，前向误差与逆向误差之比满足：Under a linear-Gaussian setting, if both models are fitted by OLS on n labelled transitions, then when n > d_s + d_a + 1 and n > 2d_z + 1 the ratio of forward error to inverse error satisfies:
三个因子分别对应维度不对称（d_z ≪ d_s 时逆向问题更简单）、随机性差异（环境噪声 σ_s 比动作恢复歧义 σ_a 更难克服）、以及有限样本稳定性（参数少时估计更准）。The three factors correspond respectively to dimensional asymmetry (the inverse problem is simpler when d_z ≪ d_s), a difference in stochasticity (environment noise σ_s is harder to overcome than the ambiguity of action recovery σ_a), and finite-sample stability (estimation is more accurate with fewer parameters).
03实验Experiments
在三个平台（MiniGrid、RoboMimic、ManiSkill）共 9 个任务上评测：MiniGrid 用于验证核心验证机制的鲁棒性，RoboMimic（Lift、Can、Square）与 ManiSkill（PullCube、PokeCube、LiftPeg）评测真实机器人场景的世界模型学习与 OOD 适应能力。基线包括 Random、Uncertainty（最高认知不确定性）、Progress（连续世界模型间的分歧）、Vanilla IDM（无稀疏掩码）及 Oracle（上界）。Evaluated on nine tasks in total across three platforms (MiniGrid, RoboMimic, ManiSkill): MiniGrid is used to verify the robustness of the core verification mechanism, while RoboMimic (Lift, Can, Square) and ManiSkill (PullCube, PokeCube, LiftPeg) assess world-model learning and OOD adaptation in realistic robot settings. Baselines include Random, Uncertainty (highest epistemic uncertainty), Progress (disagreement between successive world models), Vanilla IDM (no sparse mask) and Oracle (upper bound).
MiniGrid：探索方法对比MiniGrid: Comparison of Exploration Methods
图 4（中）：在 MiniGrid 上比较不同探索方法（WAV、Uncertainty、Progress、Random）下世界模型的预测精度。WAV 在多种探索策略中持续实现最低预测误差，结果在 5 个随机种子上取平均。Figure 4 (middle): Prediction accuracy of the world model on MiniGrid under different exploration methods (WAV, Uncertainty, Progress, Random). WAV consistently attains the lowest prediction error across the various exploration strategies; results are averaged over 5 random seeds.
RoboMimic & ManiSkill：世界模型学习RoboMimic & ManiSkill: World Model Learning
图 5：在 RoboMimic 和 ManiSkill 上报告 32 帧预测误差（MSE）随训练轨迹数增加的变化（3 个种子均值）。WAV 在绝大多数任务上实现 2× 采样效率提升：达到相同预测误差所需的训练样本量仅为基线的一半。Figure 5: On RoboMimic and ManiSkill, the 32-frame prediction error (MSE) is reported as the number of training trajectories grows (mean over 3 seeds). On the large majority of tasks WAV attains a 2× gain in sample efficiency: reaching the same prediction error takes only half as many training samples as the baseline.
下游策略性能Downstream Policy Performance
图 7：利用学习到的世界模型在 RoboMimic 和 ManiSkill 上的下游策略性能（error bar 为 3 个种子的标准误）。WAV 在所有任务上下游策略性能提升 超过 22%，在与新物体/新交互类型的泛化场景中优势尤为突出。Figure 7: Downstream policy performance on RoboMimic and ManiSkill using the learned world model (error bars are standard errors over 3 seeds). WAV improves downstream policy performance by more than 22% on all tasks, with a particularly clear advantage in generalization settings involving new objects or new interaction types.
OOD 适应OOD Adaptation
图 6：RoboMimic 上的 OOD 适应结果。在视觉偏移（背景/机体颜色变化）和对象/交互偏移（新物体、混合最优性演示）两类设定下评测世界模型的归一化预测误差与下游奖励。WAV 在两类 OOD 设定下均显著优于所有基线，在混合最优性场景下约实现 22% 奖励提升。Figure 6: OOD adaptation results on RoboMimic. The normalized prediction error and downstream reward of the world model are evaluated under two kinds of setting: visual shift (changes of background or robot-body color) and object/interaction shift (new objects, mixed-optimality demonstrations). WAV significantly outperforms all baselines under both kinds of OOD setting, achieving roughly a 22% reward gain in the mixed-optimality scenario.
验证分数与真实误差的对齐Alignment Between Verification Score and True Error
图 10：WAV 的验证分数与世界模型的真实预测误差呈强单调一致性，而其他基线（Uncertainty、Progress）分布分散且频繁出现排序错误（misranking）。这一定性结果说明 WAV 的循环一致性机制确实能准确识别模型的高误差样本。Figure 10: WAV's verification score is strongly monotonically consistent with the true prediction error of the world model, whereas the other baselines (Uncertainty, Progress) are widely scattered and frequently show misranking. This qualitative result indicates that WAV's cycle-consistency mechanism can indeed accurately identify the model's high-error samples.
消融实验Ablation Study
对比 WAV 与 Vanilla IDM（去掉稀疏掩码 M）的结果表明，稀疏性是关键：去掉掩码后模型需要从完整状态维度推断动作，既降低了样本效率又对环境噪声更为敏感。在 MiniGrid 上，随对象数量从 6 增加到 14，Sparse IDM 的验证精度基本稳定，而稠密前向模型的精度显著下降，验证了维度不对称理论分析的预测。Comparing WAV with Vanilla IDM (removing the sparse mask M) shows that sparsity is the key: without the mask the model has to infer actions from the full state dimensionality, which both lowers sample efficiency and makes it more sensitive to environment noise. On MiniGrid, as the number of objects grows from 6 to 14, the verification accuracy of the Sparse IDM stays essentially stable while the accuracy of the dense forward model drops markedly, confirming the prediction of the dimensional-asymmetry analysis.
平台 / 指标Platform / Metric
Random (基线)Random (baseline)
最优竞争方法Best competing method
WAV
提升Gain
采样效率（达到同等 MSE 所需样本）Sample efficiency (samples needed to reach the same MSE)
注：论文无独立的 Limitations 章节。以下局限性部分来自论文正文与附录中的明确陈述（标注 论文陈述），部分为从设计推断（标注 设计推断）。Note: The paper has no standalone Limitations section. Some of the limitations below come from explicit statements in the main text and the appendix (marked stated in the paper), others are inferred from the design (marked design inference).
稀疏逆模型依赖动作标注，无法充分利用无标注数据（论文陈述）The sparse inverse model relies on action labels and cannot fully exploit unlabelled data (stated in the paper)
子目标生成器可以利用大规模无动作视频预训练，但稀疏 IDM 本身需要带动作标注的交互数据才能训练，这限制了在动作标注极度稀缺的场景下的可扩展性。若无动作视频数据相对于带标注数据占比持续扩大，IDM 部分将成为瓶颈。The subgoal generator can be pre-trained on large-scale action-free video, but the sparse IDM itself needs action-labelled interaction data in order to be trained, which limits scalability in settings where action labels are extremely scarce. If the share of action-free video keeps growing relative to labelled data, the IDM component will become the bottleneck.
理论分析基于线性-高斯"风格化模型"，不完全覆盖真实环境（论文陈述）The theoretical analysis rests on a linear-Gaussian "stylized model" and does not fully cover real environments (stated in the paper)
Proposition 3.1 在线性高斯框架下推导；论文在附录 F.2 中明确标注 "Scope of the stylized model"，说明该理论结论对于高度非线性的视觉状态空间与复杂动力学系统的适用边界有待进一步研究。Proposition 3.1 is derived within a linear-Gaussian framework; in Appendix F.2 the paper explicitly flags "Scope of the stylized model", noting that the applicability of this theoretical conclusion to highly nonlinear visual state spaces and complex dynamical systems remains to be studied further.
性能依赖于动作可恢复性（action-recovery injectivity）（论文陈述）Performance depends on action-recovery injectivity (stated in the paper)
论文指出，若动作恢复的单射性（injectivity）弱（即多个动作均能产生相似 transition），逆向验证的精度会下降；若 OOD 偏移反向传播到验证组件本身，方法也可能退化。这意味着 WAV 在动作空间连续且高度冗余的任务上效果可能不稳定。The paper points out that if the injectivity of action recovery is weak (i.e. several actions can produce similar transitions), the accuracy of inverse verification drops; and if the OOD shift propagates back into the verification components themselves, the method may likewise degrade. This means WAV may be unstable on tasks whose action space is continuous and highly redundant.
需要同时获取无动作视频语料库与带标注交互数据（设计推断）Requires access to both an action-free video corpus and labelled interaction data (design inference)
WAV 的优势部分来自"无动作视频数据远比带标注数据丰富"这一前提假设。在专有工业场景或新型机器人平台中，大规模相关视频可能并不存在，此时子目标生成器的泛化能力及整体框架的优势可能受到限制。Part of WAV's advantage comes from the premise that "action-free video data is far more abundant than labelled data". In proprietary industrial settings or on novel robot platforms, large-scale relevant video may simply not exist, in which case the generalization ability of the subgoal generator and the advantage of the overall framework may be limited.
