表示学习 · 世界模型 · 理论Representation Learning · World Models · Theory
LeJEPA 何时学到世界模型？When Does LeJEPA Learn a World Model?
When Does LeJEPA Learn a World Model?
David Klindt (Cold Spring Harbor Laboratory) · Yann LeCun (New York University) · Randall Balestriero (Brown University)
本文为 Joint-Embedding Predictive Architectures（JEPA）建立首个可识别性保证：在潜变量服从高斯分布的广泛世界类中，LeJEPA（对齐损失 + 高斯正则化）可从非线性观测中线性恢复真实潜变量，且高斯分布是使该保证成立的唯一潜变量分布。This paper establishes the first identifiability guarantee for Joint-Embedding Predictive Architectures (JEPA): across a broad class of worlds whose latent variables follow a Gaussian distribution, LeJEPA (alignment loss + Gaussian regularization) linearly recovers the true latent variables from nonlinear observations, and the Gaussian is the only latent distribution for which this guarantee holds.
自监督学习（SSL）的愿景在于无需标注数据即可学到有用的世界表示。Joint-Embedding Predictive Architectures（JEPAs）通过训练表示使同一输入的相关视图产生相似嵌入，并辅以正则化防止塌缩。然而一个更深层的问题悬而未决：何时学到的表示才是"世界模型"——即对世界潜在结构的忠实映射？The promise of self-supervised learning (SSL) is to learn useful representations of the world without labeled data. Joint-Embedding Predictive Architectures (JEPAs) train representations so that related views of the same input produce similar embeddings, with regularization added to prevent collapse. Yet a deeper question remains open: when is the learned representation a “world model” — that is, a faithful map of the latent structure of the world?
"A representation that scrambles the true degrees of freedom of the world cannot support reliable planning or compositional generalization."
——论文原文— from the paper
每当通过 linear probing 评估表示质量时，隐含的问题正是：模型是否学到了潜变量的线性表示？没有线性可识别性（linear identifiability），线性探针就无法精确恢复潜变量。本文正是要给出这一性质的严格数学保证。Whenever representation quality is assessed via linear probing, the implicit question is precisely this: has the model learned a linear representation of the latent variables? Without linear identifiability, a linear probe cannot recover the latent variables exactly. This paper sets out to give a rigorous mathematical guarantee for that property.
图 1：LeJEPA 学到世界模型。（左）世界拥有独立高斯潜变量；（中）未知的非线性过程将其混合为我们观测到的数据；（右）LeJEPA 将潜变量恢复至旋转等价，即学到正确的世界模型（Thm. 5.1）。Figure 1: LeJEPA learns a world model. (Left) The world has independent Gaussian latent variables; (middle) an unknown nonlinear process mixes them into the data we observe; (right) LeJEPA recovers the latent variables up to a rotation, i.e. it learns the correct world model (Thm. 5.1).
首个FirstJEPA 可识别性理论保证Identifiability guarantee for JEPA
唯一Unique高斯分布是线性可识别的唯一潜变量分布（Thm. 5.2）The Gaussian is the only linearly identifiable latent distribution (Thm. 5.2)
R²>0.999SIGReg/VICReg 在 N=1024 维时仍保持Still maintained by SIGReg/VICReg at N=1024
R²=0.95DMC Reacher OU 数据对下最优规划表现Best planning performance on DMC Reacher OU pairs
02方法Method
论文从理论角度分析 LeJEPA：在满足独立性、平稳性、加性噪声三条假设的"世界"类中，证明带高斯正则化的对齐目标可线性恢复高斯潜变量，并给出近似可识别性界和最优规划保证。The paper analyzes LeJEPA theoretically: within the class of “worlds” satisfying the three assumptions of independence, stationarity and additive noise, it proves that an alignment objective with Gaussian regularization linearly recovers Gaussian latent variables, and it derives an approximate-identifiability bound and an optimal-planning guarantee.
图 2（示意）：LeJEPA 理论框架。（左）世界具有干净的高斯潜变量结构，正样本对高度相关；（中）未知非线性映射产生观测数据；（右）LeJEPA 以"拉近正样本对"（对齐）加"保持嵌入分布为高斯"（SIGReg）两个目标训练编码器——理论证明学到的表示必为真实潜变量的旋转。Figure 2 (schematic): the LeJEPA theoretical framework. (Left) The world has clean Gaussian latent structure and positive pairs are highly correlated; (middle) an unknown nonlinear map produces the observations; (right) LeJEPA trains the encoder with two objectives — pulling positive pairs together (alignment) and keeping the embedding distribution Gaussian (SIGReg) — and the theory proves the learned representation must be a rotation of the true latent variables.
世界模型假设World Model Assumptions
设世界潜变量 z ∈ ℝⁿ 通过未知非线性映射 g 生成观测 x = g(z)。论文对世界施加三条假设：Let the world latent variables z ∈ ℝⁿ generate observations x = g(z) through an unknown nonlinear map g. The paper imposes three assumptions on the world:
独立性（Independence）：各维潜变量及其转移相互独立，即 p(zᵢ) ⊥ p(zⱼ) 且转移独立。Independence: the latent dimensions and their transitions are mutually independent, i.e. p(zᵢ) ⊥ p(zⱼ) and the transitions are independent.
平稳性（Stationarity）：两个视图 z, z′ 共享相同的边缘分布 p(z) = p(z′)。Stationarity: the two views z, z′ share the same marginal distribution p(z) = p(z′).
加性噪声（Additive noise）：转移形如 z′ᵢ = mᵢ(zᵢ) + ηᵢ，噪声 ηᵢ 与 zᵢ 独立。Additive noise: transitions take the form z′ᵢ = mᵢ(zᵢ) + ηᵢ, with the noise ηᵢ independent of zᵢ.
在高斯潜变量世界中，满足上述假设的唯一转移为 Ornstein–Uhlenbeck（OU）过程：In a Gaussian-latent world, the only transition satisfying the assumptions above is the Ornstein–Uhlenbeck (OU) process: z′ = ρz + √(1−ρ²)η，η ∼ N(0,Iₙ)，η ⊥ z 其中 ρ ∈ (0,1) 控制视图间相关性。where ρ ∈ (0,1) controls the correlation between views.
LeJEPA 学习目标LeJEPA Learning Objective
编码器 h = f∘g: ℝⁿ → ℝⁿ 通过最小化如下目标训练：The encoder h = f∘g: ℝⁿ → ℝⁿ is trained by minimizing the following objective:
对于高斯世界，设 h: ℝⁿ→ℝⁿ 满足 h(z)∼N(0,Iₙ)，则 L(h) ≥ 2(1−ρ)n，等号成立当且仅当 h(z) = Qz，其中 Q ∈ O(n)（正交矩阵）。证明关键：Hermite 多项式分解使每个非线性度 d≥2 受到严格惩罚，线性映射是唯一最优解。For a Gaussian world, let h: ℝⁿ→ℝⁿ satisfy h(z)∼N(0,Iₙ); then L(h) ≥ 2(1−ρ)n, with equality if and only if h(z) = Qz, where Q ∈ O(n) (an orthogonal matrix). Key to the proof: a Hermite polynomial decomposition penalizes every nonlinear degree d≥2 strictly, so the linear map is the unique optimum.
在满足三条假设的所有世界中，若对齐约束加白化的唯一最小化器是线性映射 h(z) = Qz，则 z 必须是高斯分布。证明借助 Sturm–Liouville 谱理论，排除了全部非高斯替代方案。Among all worlds satisfying the three assumptions, if the unique minimizer of the alignment constraint plus whitening is a linear map h(z) = Qz, then z must be Gaussian. The proof draws on Sturm–Liouville spectral theory to rule out every non-Gaussian alternative.
设近似对齐间隙为 δ，近似白化误差为 ε，令 D = δ / (2ρ(1−ρ))。则存在 Q ∈ O(n) 使得：Let the approximate alignment gap be δ and the approximate whitening error be ε, and set D = δ / (2ρ(1−ρ)). Then there exists Q ∈ O(n) such that: 𝔼[‖h(z) − Qz‖²] ≤ D + (ε + D)² 恢复误差随 δ、ε 连续平滑退化。The recovery error degrades continuously and smoothly with δ and ε.
设 h(z) = Qz（Q ∈ O(n)），对任意旋转不变代价函数的有限水平最优控制问题，有 V̂*(h(z₀)) = V*(z₀)，即在学到的潜空间与真实潜空间中规划等价。Let h(z) = Qz (Q ∈ O(n)). For any finite-horizon optimal control problem with a rotation-invariant cost function, V̂*(h(z₀)) = V*(z₀), i.e. planning in the learned latent space is equivalent to planning in the true latent space.
图 3：2D 模拟实验（图示）。颜色编码真实潜变量的极坐标角度与半径。三种非线性混合（抛物线剪切、正弦剪切、RealNVP 耦合层）下，LeJEPA 均将各向同性高斯结构恢复至旋转等价，与定理 5.1 吻合。Figure 3: 2D simulation experiments (illustration). Color encodes the polar angle and radius of the true latent variables. Under three nonlinear mixings (parabolic shear, sinusoidal shear, RealNVP coupling layers), LeJEPA recovers the isotropic Gaussian structure up to a rotation, in agreement with Theorem 5.1.
03实验Experiments
实验涵盖：2D 非线性混合验证、维度扩展性（N=2→1024）、分布消融（广义正态族）、DMC Reacher 机器人像素控制。基线方法：SIGReg、VICReg、InfoNCE。评估指标：线性可识别性 R²（h→z 及 z→h）。The experiments cover: 2D nonlinear-mixing validation, dimensional scalability (N=2→1024), a distribution ablation (generalized normal family), and pixel-based robot control on DMC Reacher. Baselines: SIGReg, VICReg, InfoNCE. Metric: linear identifiability R² (h→z and z→h).
扩展性实验：不同正则化方法对比（Table 1 摘录）Scalability experiment: comparison of regularization methods (excerpt from Table 1)
共享 RealNVP 混合与匹配编码器，5 种随机种子，均值 ± 标准差：Shared RealNVP mixing and matched encoders, 5 random seeds, mean ± standard deviation:
潜变量维度 NLatent dimension N
SIGReg R²(h→z)
VICReg R²(h→z)
InfoNCE R²(h→z)
2
0.999998 ± 3.4e-6
0.999996 ± 8.4e-6
0.950961 ± 1.6e-3
64
0.999966 ± 7.4e-6
0.999968 ± 8.1e-6
0.648496 ± 3.1e-2
256
0.999884 ± 7.9e-6
0.999889 ± 7.2e-6
0.696587 ± 4.9e-3
512
0.999775 ± 6.7e-6
0.999785 ± 6.9e-6
0.704393 ± 2.6e-3
1024
0.999561 ± 1.2e-5
0.999582 ± 1.1e-5
0.720241 ± 2.0e-3
结论：SIGReg 和 VICReg 在 N=1024 时仍维持 R²>0.999；InfoNCE 在固定核宽 σ=1 下随维度增大明显退化。Takeaway: SIGReg and VICReg still maintain R²>0.999 at N=1024; InfoNCE degrades markedly with increasing dimension under a fixed kernel bandwidth σ=1.
RL 轨迹实验：OU 数据对 vs 策略轨迹（Table 2 摘录）RL trajectory experiment: OU pairs vs. policy trajectories (excerpt from Table 2)
数据类型Data type
相关系数 ρCorrelation ρ
R²(z→h)
R²(h→z)
OU（高斯正样本对）OU (Gaussian positive pairs)
0.30
0.67 ± 2e-2
0.67 ± 2e-2
OU（高斯正样本对）OU (Gaussian positive pairs)
0.90
0.95 ± 7e-4
0.95 ± 7e-4
OU（高斯正样本对）OU (Gaussian positive pairs)
0.99
0.95 ± 4e-4
0.95 ± 4e-4
RL 轨迹（非高斯）RL trajectories (non-Gaussian)
stride δ=1
-0.39 ± 1e-1
0.71 ± 3e-2
RL 轨迹（非高斯）RL trajectories (non-Gaussian)
stride δ=64
0.44 ± 4e-2
0.55 ± 3e-2
结论：高斯 OU 数据对下，R² 随 ρ 单调上升；RL 策略轨迹因各向异性（ρ₀ ≠ ρ₁）和非高斯转移导致可识别性显著下降，与理论预测一致。Takeaway: with Gaussian OU pairs, R² increases monotonically with ρ; RL policy trajectories show a marked drop in identifiability due to anisotropy (ρ₀ ≠ ρ₁) and non-Gaussian transitions, consistent with the theoretical prediction.
图 4：实验结果总览。
(a) 定理 5.3 界的验证：SIGReg 跨网格搜索、2D 混合、扩展性和分布消融的运行结果均在对角线以下，确认近似可识别性界成立（少数接近零的离群点源于有限样本估计噪声）。
(b) 高斯最优性：线性恢复 R²(h→z) 在广义正态分布 α=2（高斯）处达到尖锐峰值，与定理 5.2 吻合；SIGReg 的高斯化对非高斯潜变量分布比白化更鲁棒。
(c) 控制代价（30 组随机起始-目标对，路径长度 ≥1，理想值为 1）：高斯编码器与 oracle 在统计上无差异；Trajectory 编码器代价系统性偏高。
(d) 控制代价随线性可识别性 R² 单调下降，支持定理 5.4。Figure 4: overview of the experimental results.
(a) Validation of the Theorem 5.3 bound: runs across grid search, 2D mixings, scalability and the distribution ablation all fall below the diagonal, confirming that the approximate-identifiability bound holds (the few near-zero outliers stem from finite-sample estimation noise).
(b) Gaussian optimality: linear recovery R²(h→z) peaks sharply at α=2 (Gaussian) in the generalized normal family, in agreement with Theorem 5.2; the Gaussianization of SIGReg is more robust to non-Gaussian latent distributions than whitening.
(c) Control cost (30 random start-goal pairs, path length ≥1, ideal value 1): the Gaussian encoder is statistically indistinguishable from the oracle, while the Trajectory encoder incurs systematically higher cost.
(d) Control cost decreases monotonically with linear identifiability R², supporting Theorem 5.4.图：DMC Reacher 机器人控制环境。像素观测下的连续控制任务，用于验证理论在实际视觉输入上的适用性。Figure: the DMC Reacher robot control environment. A continuous control task from pixel observations, used to verify that the theory applies to real visual inputs.图 5：线性可识别性使能潜空间规划。
在三种编码器的潜空间中对固定起始帧和目标帧做插值，并通过最近邻检索解码。
上：Oracle（关节空间直线）；
中：高斯编码器（OU，ρ=0.99）紧密追随 oracle；
下：RL 轨迹编码器（stride δ=8）出现明显偏差。
此结果直观验证了定理 5.4：线性可识别性使潜空间规划与真实潜空间规划等价。Figure 5: linear identifiability enables latent-space planning.
Interpolating between a fixed start frame and a goal frame in the latent space of three encoders, decoded by nearest-neighbor retrieval.
Top: Oracle (a straight line in joint space);
middle: the Gaussian encoder (OU, ρ=0.99) tracks the oracle closely;
bottom: the RL-trajectory encoder (stride δ=8) deviates noticeably.
This result visually confirms Theorem 5.4: linear identifiability makes latent-space planning equivalent to planning in the true latent space.
分布消融Distribution Ablation
对广义正态分布族（generalized normal family，形状参数 α）进行扫描：线性恢复 R² 在 α=2（高斯）处达到尖锐峰值，SIGReg 和 InfoNCE 对重尾潜变量比 VICReg 拥有更宽的平坦区。这与定理 5.2 的"高斯唯一性"完全吻合。Sweeping the generalized normal family (shape parameter α): linear recovery R² peaks sharply at α=2 (Gaussian), and SIGReg and InfoNCE have a wider plateau for heavy-tailed latent variables than VICReg. This agrees exactly with the “Gaussian uniqueness” of Theorem 5.2.
2D 仿真2D Simulation
三种非线性混合（抛物线剪切、正弦剪切、RealNVP 耦合层）：学到的表示将各非线性混合逆转至旋转等价，与定理 5.1 一致。网格搜索显示：过强的高斯性（λ=0.5）会压缩表示，最优恢复在 低 λ + 高 ρ 时出现。Three nonlinear mixings (parabolic shear, sinusoidal shear, RealNVP coupling layers): the learned representation inverts each nonlinear mixing up to a rotation, consistent with Theorem 5.1. The grid search shows that overly strong Gaussianity (λ=0.5) compresses the representation, and the best recovery occurs at low λ + high ρ.
图：规划代价 vs 线性可识别性散点图。各模型的控制代价与线性可识别性 R² 呈单调递减关系，支持"线性可识别表示可实现最优潜空间规划"（定理 5.4）的理论结论。Figure: scatter plot of planning cost vs. linear identifiability. Control cost decreases monotonically with linear identifiability R² across models, supporting the theoretical conclusion that “linearly identifiable representations enable optimal latent-space planning” (Theorem 5.4).
04局限性Limitations
说明：Note:以下局限性由论文作者在正文中明确陈述（stated by the authors），非推断。the limitations below are stated explicitly by the authors in the paper (stated by the authors), not inferred.
高斯分布是给定均值与协方差情况下最大熵分布，是最少假设的先验。然而从观测数据中无法验证真实潜变量是否高斯。有一个尺度论据：单个微观变量可能非高斯，但任务相关的潜变量往往是大量微观变量的聚合，中心极限定理倾向于使其趋向高斯。尽管如此，这一假设本质上无法从观测中证伪。The Gaussian is the maximum-entropy distribution for a given mean and covariance, and hence the least-assumptive prior. However, whether the true latent variables are Gaussian cannot be verified from observational data. There is a scale argument: an individual microscopic variable may be non-Gaussian, but task-relevant latent variables are usually aggregates of many microscopic variables, and the central limit theorem tends to push them toward Gaussianity. Even so, this assumption is fundamentally unfalsifiable from observations.
嵌入维度不匹配时的行为（Dimension Mismatch）Behavior under dimension mismatch (Dimension Mismatch)
定理假设编码器输出维度 m 等于真实潜变量维度 n（m=n）。当 m<n 时，高斯性约束不能确定选取哪个子空间；当 m>n 时，多余维度必须塌缩或编码冗余。理解维度不匹配对可识别性的影响是一个重要的开放问题，对 JEPA 实践设计有直接影响。The theorems assume the encoder output dimension m equals the true latent dimension n (m=n). When m<n, the Gaussianity constraint does not determine which subspace is selected; when m>n, the surplus dimensions must collapse or encode redundancy. Understanding how dimension mismatch affects identifiability is an important open problem, with direct implications for the practical design of JEPAs.
理论结果是关于全局最优的总体水平（population-level）陈述。定理 5.3 证明保证随对齐间隙和协方差偏差连续退化，但未说明这些量如何随样本量或训练动态缩放。实验中观察到少量界违反情况（图 4a），与 ε 和 δ 的有限样本估计噪声一致。The theoretical results are population-level statements about the global optimum. Theorem 5.3 proves that the guarantee degrades continuously with the alignment gap and the covariance deviation, but says nothing about how these quantities scale with sample size or training dynamics. A small number of bound violations were observed in the experiments (Fig. 4a), consistent with finite-sample estimation noise in ε and δ.
本理论针对编码器学习，未涉及动作条件下的动态学习（action-conditioned dynamics），这是将 LeJEPA 从表示学习扩展到完整世界模型学习的重要后续方向。This theory targets encoder learning and does not address action-conditioned dynamics learning (action-conditioned dynamics), an important follow-up direction for extending LeJEPA from representation learning to full world-model learning.
