通过内容-结构解耦，在潜在动作模型中同时实现高级动作抽象与高保真视频生成Achieving both high-level action abstraction and high-fidelity video generation in latent action models through content-structure disentanglement
Latent Action Models (LAMs) 面临"动作抽象"与"生成保真度"之间的根本性权衡。DiLA 通过将视频帧分解为结构（structure）通道和内容（content）通道，让潜在动作编码空间布局（运动信息），内容记忆保留视觉细节，从而在不牺牲生成质量的前提下学到连续且语义结构化的潜在动作空间。Latent Action Models (LAMs) face a fundamental trade-off between "action abstraction" and "generation fidelity". DiLA decomposes video frames into a structure pathway and a content pathway, letting the latent action encode spatial layout (motion information) while the content memory preserves visual detail, thereby learning a continuous and semantically structured latent action space without sacrificing generation quality.
4× NVIDIA A100 (80GB)123M 可训练参数123M trainable parameters无动作标注自监督训练Self-supervised training without action labels📄 arXiv:2605.15725🌐 Project Page
latent action modelworld modelcontent-structure disentanglementvideo generationvisual planningcross-embodiment transfer机器人操作robot manipulation世界模型world model
01动机 MotivationMotivation
Latent Action Models (LAMs) 是一类以无监督方式从视频中提取潜在动作表示的世界模型。然而，已有方法在动作抽象和生成保真度之间存在难以调和的权衡：通过 Vector Quantization 或 VAE 强化瓶颈可以提升动作抽象质量，却以牺牲视频帧的视觉细节为代价；而保留视觉细节又会导致结构化动作表示退化。Latent Action Models (LAMs) are world models that extract latent action representations from video in an unsupervised manner. Existing methods, however, suffer from an irreconcilable trade-off between action abstraction and generation fidelity: strengthening the bottleneck through Vector Quantization or a VAE improves the quality of action abstraction at the cost of the visual detail of the video frames, while preserving visual detail in turn degrades the structured action representation.
"The predictive bottleneck inherent in latent action learning serves as a driving force for disentanglement."
作者将这一现象称为 "LAM Trade-off"，并提出核心洞察：仅需预测帧间差异的结构变化即可完成视频预测，这天然驱动了内容与结构的解耦。通过显式地将两条路径分离——结构通道（dynamics）和内容通道（appearance）——DiLA 绕开了这一权衡。The authors call this phenomenon the "LAM Trade-off" and propose a key insight: video prediction only requires predicting the structural change between frames, which naturally drives the disentanglement of content and structure. By explicitly separating the two pathways — the structure pathway (dynamics) and the content pathway (appearance) — DiLA sidesteps this trade-off.
图1：潜在动作与解耦效果随受限预测瓶颈的共同演化。随着训练推进，结构通道逐渐专注于运动信息，内容通道保留静态外观，两者实现清晰分离。Figure 1: Co-evolution of latent actions and disentanglement under the constrained predictive bottleneck. As training proceeds, the structure pathway progressively focuses on motion information while the content pathway retains static appearance, so that the two become clearly separated.
0.009Push-T Linear Probing MSE↓（动作解码最优）Push-T Linear Probing MSE↓ (best action decoding)
02方法 MethodMethod
DiLA 由三个核心模块组成：结构通道（Structure Pathway）提取并预测帧间的空间布局变化；内容通道（Content Pathway）通过 Mamba-based memory 聚合静态视觉特征；Fusion Decoder 将两路信息融合重建目标帧。整个系统无需任何动作标注，以自监督 teacher-forcing 方式端到端训练。DiLA consists of three core modules: the Structure Pathway extracts and predicts the spatial layout changes between frames; the Content Pathway aggregates static visual features through a Mamba-based memory; and the Fusion Decoder fuses the two streams to reconstruct the target frame. The whole system needs no action labels at all and is trained end-to-end in a self-supervised teacher-forcing manner.
图2：DiLA 整体架构。输入视频帧经过编码后分流至结构通道（压缩为结构嵌入 s）和内容通道（Mamba 记忆模块聚合特征 cmem）。IDM（Inverse Dynamics Model）从相邻结构嵌入的差异中提取潜在动作 z（dz=256），FDM（Forward Dynamics Model）通过残差更新预测下一状态的结构嵌入；Fusion Decoder 最终将预测结构与内容记忆及初始帧融合重建目标帧。Figure 2: Overall architecture of DiLA. Encoded input video frames are routed to the structure pathway (compressed into the structure embedding s) and to the content pathway (a Mamba memory module aggregates features into cmem). The IDM (Inverse Dynamics Model) extracts the latent action z (dz=256) from the difference between adjacent structure embeddings, and the FDM (Forward Dynamics Model) predicts the structure embedding of the next state through a residual update; the Fusion Decoder finally fuses the predicted structure with the content memory and the initial frame to reconstruct the target frame.
视频帧经过编码后被压缩为结构嵌入 s，只保留空间布局信息。Inverse Dynamics Model (IDM) 接受相邻两帧的结构嵌入，提取潜在动作 z（维度 dz=256，连续表示）；Forward Dynamics Model (FDM) 则将当前结构嵌入与潜在动作结合，通过残差更新预测下一帧的结构嵌入。训练目标包含视觉潜在预测损失 ℒe（λ=2.0）、结构预测损失 ℒs（λ=0.03）、潜在动作一致性损失 ℒz（λ=0.03）以及含 inverse-temporal symmetry 的正则化损失 ℒreg（λ=0.001）。Video frames are encoded and compressed into the structure embedding s, which retains only spatial layout information. The Inverse Dynamics Model (IDM) takes the structure embeddings of two adjacent frames and extracts the latent action z (dimension dz=256, a continuous representation); the Forward Dynamics Model (FDM) combines the current structure embedding with the latent action and predicts the structure embedding of the next frame through a residual update. The training objective comprises a visual latent prediction loss ℒe (λ=2.0), a structure prediction loss ℒs (λ=0.03), a latent action consistency loss ℒz (λ=0.03), and a regularization loss ℒreg (λ=0.001) that includes inverse-temporal symmetry.
内容通道的设计目的是隔离静态视觉外观，不引入运动信息。Mamba-based memory 模块在时间维度上聚合来自多帧的静态特征，输出内容记忆向量 cmem。消融实验验证：冻结结构嵌入后，重建序列静止不动，确认内容通道中零运动泄漏（zero motion leakage）。The content pathway is designed to isolate static visual appearance without introducing motion information. The Mamba-based memory module aggregates static features from multiple frames along the temporal dimension and outputs the content memory vector cmem. An ablation confirms this: once the structure embedding is frozen, the reconstructed sequence stays completely still, verifying zero motion leakage in the content pathway.
动作迁移 — DINOv2 结构化嵌入Action Transfer — DINOv2 Structured Embeddings
跨场景/跨实体动作迁移通过 DINOv2 特征对齐实现：将源视频提取的结构变化（潜在动作序列）应用于目标初始帧，无需任何微调。这支持了 human→robot、跨视角、跨导航环境等多种 cross-embodiment 迁移场景。Cross-scene and cross-embodiment action transfer is achieved through DINOv2 feature alignment: the structural changes (the latent action sequence) extracted from a source video are applied to the initial frame of a target, without any fine-tuning. This supports a variety of cross-embodiment transfer settings such as human→robot, cross-view, and cross-navigation-environment.
图3：DiLA 的跨实体动作迁移实验。从人手操作视频中提取潜在动作，直接迁移到机器人手臂（human-to-robot）、不同物体/视角（semantic transfer），以及不同导航环境（navigation），证明所学动作空间具有跨域语义一致性。Figure 3: Cross-embodiment action transfer experiments of DiLA. Latent actions extracted from videos of human hand manipulation are transferred directly to a robot arm (human-to-robot), to different objects and viewpoints (semantic transfer), and to different navigation environments (navigation), showing that the learned action space is semantically consistent across domains.
03实验 ExperimentsExperiments
实验在 Something-Something v2 (SSv2)、RT-1 机器人操作、OmniObject3D、RECON/LoopNav 导航等多个数据集上展开，与 LAPA、Moto、AdaWorld、villa-x 等强基线对比，评估视频生成质量（SSIM↑、LPIPS↓）、动作解码能力（Linear Probing MSE↓）以及视觉规划成功率。Experiments are carried out on several datasets including Something-Something v2 (SSv2), RT-1 robot manipulation, OmniObject3D and RECON/LoopNav navigation, and compared against strong baselines such as LAPA, Moto, AdaWorld and villa-x, evaluating video generation quality (SSIM↑, LPIPS↓), action decoding ability (Linear Probing MSE↓) and visual planning success rate.
视频生成质量（Table 1）Video Generation Quality (Table 1)
模型Model
SSv2 SSIM↑
SSv2 LPIPS↓
RT-1 SSIM↑
RT-1 LPIPS↓
LAPA
0.637±0.035
0.565±0.021
0.491±0.014
0.595±0.005
Moto
0.555±0.043
0.593±0.022
0.762±0.023
0.284±0.016
AdaWorld(FDM)
0.625±0.029
0.576±0.016
0.554±0.014
0.549±0.012
AdaWorld
0.674±0.008
0.521±0.022
0.634±0.013
0.429±0.009
villa-x
0.636±0.036
0.515±0.026
0.576±0.023
0.477±0.021
DiLA w/o content
0.594±0.050
0.450±0.031
0.647±0.022
0.258±0.015
DiLA
0.660±0.037
0.356±0.027
0.774±0.010
0.206±0.013
DiLA 在感知质量指标 LPIPS 上全面领先（两数据集均最优），RT-1 SSIM 也达到最优。SSv2 SSIM 略低于 AdaWorld（0.660 vs. 0.674），但 SSv2 LPIPS 显著优于所有基线（0.356 vs. 0.515+）。DiLA leads across the board on the perceptual quality metric LPIPS (best on both datasets), and also attains the best RT-1 SSIM. Its SSv2 SSIM is slightly below AdaWorld (0.660 vs. 0.674), but its SSv2 LPIPS is markedly better than every baseline (0.356 vs. 0.515+).
图4：解耦验证实验。重绑定实验（rebinding）将一个视频的结构嵌入与另一视频的内容记忆组合，可产生融合两者的输出，证明结构通道捕获运动而内容记忆保留静态外观。孤立运动实验（motion isolation）冻结结构嵌入后序列保持静止，确认内容通道零运动泄漏。Figure 4: Disentanglement verification experiments. The rebinding experiment combines the structure embedding of one video with the content memory of another and produces an output that fuses the two, showing that the structure pathway captures motion while the content memory preserves static appearance. In the motion isolation experiment the sequence stays still once the structure embedding is frozen, confirming zero motion leakage in the content pathway.
连续潜在动作表示显著优于离散（Vector Quantization）和高斯变分形式。内容通道对于改善动作迁移质量不可或缺（w/o content 在所有指标上均显著下降）。The continuous latent action representation is markedly better than the discrete (Vector Quantization) and Gaussian variational forms. The content pathway is indispensable for improving action transfer quality (w/o content degrades significantly on every metric).
OOD 机器人基准线性探测（Table 3）Linear Probing on OOD Robot Benchmarks (Table 3)
方法Method
Franka Kitchen↓
Block Pushing↓
Push-T↓
LIBERO Goal↓
Discrete z
0.098±0.014
0.061±0.015
0.023±0.004
0.160±0.023
Gaussian z
0.125±0.020
0.102±0.023
0.041±0.006
0.190±0.022
DiLA
0.073±0.014
0.037±0.013
0.009±0.003
0.119±0.018
潜在动作流形分析Latent Action Manifold Analysis
图5：OmniObject3D 数据集上的潜在动作空间 UMAP 可视化。平移动作形成拓扑与实际运动方向一致的二维流形；缩放动作关于恒等点呈对称分布；旋转动作形成连续色谱；组合动作聚类可解释；导航任务（RECON/LoopNav）分别显示连续偏航角谱或离散簇结构。Figure 5: UMAP visualization of the latent action space on the OmniObject3D dataset. Translation actions form a 2D manifold whose topology agrees with the actual direction of motion; scaling actions are distributed symmetrically about the identity point; rotation actions form a continuous spectrum; composed actions cluster interpretably; and the navigation tasks (RECON/LoopNav) show a continuous yaw-angle spectrum or a discrete cluster structure respectively.
在 Model Predictive Control 视觉规划基准 VP² 上，DiLA 聚合成功率 41.44% 远超 AdaWorld 的 21.54%，在 5/6 个子任务上占优。仅 Upright block 任务（精细控制）AdaWorld 略胜（5.00% vs. 3.33%），与作者分析的局限性一致。On the Model Predictive Control visual planning benchmark VP², DiLA reaches an aggregate success rate of 41.44%, far above AdaWorld's 21.54%, and wins on 5/6 of the subtasks. Only on the Upright block task (fine-grained control) does AdaWorld edge ahead (5.00% vs. 3.33%), consistent with the limitation analyzed by the authors.
04局限性 LimitationsLimitations
说明：以下局限性均为作者在论文中明确陈述（stated by the authors）。Note:All limitations below are explicitly stated by the authors in the paper.
精细控制不稳定（Fine-grained Control）Unstable fine-grained control (Fine-grained Control)
潜在动作的高级抽象特性牺牲了控制精度，导致在要求精细操作的任务（如 VP² 中的 Upright block）上表现不稳定。抽象层面越高，对底层执行器的精确控制越难保证。The high-level abstraction of latent actions sacrifices control precision, which leads to unstable performance on tasks that demand fine manipulation (such as Upright block in VP²). The higher the level of abstraction, the harder it is to guarantee precise control of the low-level actuator.
当前框架将空间布局（结构）与外观（内容）分离，但不支持对多个物体的独立分解建模。场景中存在多个独立运动物体时，模型的解耦能力受限。The current framework separates spatial layout (structure) from appearance (content), but does not support independently factorized modeling of multiple objects. When several independently moving objects are present in a scene, the disentanglement ability of the model is limited.
长视频自回归误差累积（Long-horizon Video）Autoregressive error accumulation in long videos (Long-horizon Video)
以自回归方式逐帧预测时，预测误差会随时间步增加而累积（compounding errors），导致长时序视频生成质量下降。When predicting frame by frame in an autoregressive manner, prediction errors accumulate as the time step grows (compounding errors), degrading the quality of long-horizon video generation.
