用贝叶斯框架自动识别扩散模型的低质量生成样本Automatically identifying low-quality samples of diffusion models with a Bayesian framework
Metod Jazbec, Eliot Wong-Toi, Guoxuan Xia, Dan Zhang, Eric Nalisnick, Stephan Mandt · arXiv 2025
扩散模型整体生成质量很高，但单次生成仍可能出现伪影或与条件不符的情况。本文提出"生成不确定性"（generative uncertainty）概念，借助 Laplace 近似将贝叶斯推断扩展到数十亿参数的大型扩散模型，通过语义似然度在潜空间中量化每张图像的不可靠程度，无需人工标注即可自动过滤低质量样本，从而显著提升生成集合的整体质量。Diffusion models achieve high generation quality overall, yet an individual generation may still contain artifacts or fail to match its conditioning. This paper introduces the notion of generative uncertainty and, via a Laplace approximation, extends Bayesian inference to large diffusion models with billions of parameters. A semantic likelihood quantifies in latent space how unreliable each image is, so that low-quality samples can be filtered automatically without any human annotation, substantially improving the overall quality of the generated set.
扩散模型平均生成质量虽高，但单次采样仍会产生含有伪影的低质量图像，现有方法只能依赖人工审核来筛选——这既费时又难以规模化。Although diffusion models are of high average generation quality, a single sampling run can still produce low-quality images with artifacts, and existing approaches can only rely on manual inspection to screen them out, which is both time-consuming and hard to scale.
"How can Bayesian principles help us detect poor generations?" 如何用贝叶斯原则自动检测低质量生成样本？"How can Bayesian principles help us detect poor generations?"
图1：方法总览。对于给定噪声输入 z，从 Laplace 后验中采样 M 组模型参数，分别生成 M 张图像，将其映射到语义嵌入空间，再计算该分布的熵作为"生成不确定性"。高不确定性图像被自动标记为低质量候选并过滤。Figure 1: Method overview. For a given noise input z, M sets of model parameters are drawn from the Laplace posterior and used to generate M images, which are mapped into a semantic embedding space; the entropy of that distribution is then computed as the generative uncertainty. Images with high uncertainty are automatically flagged as low-quality candidates and filtered out.
7.89UViT 过滤后 FID (n=10K)FID after filtering, UViT (n=10K)
9.45随机基线 FIDRandom-baseline FID
25最少额外 NFE（优化版）Minimum extra NFE (optimized setting)
–0.27不确定性与 Realism 相关性（互补信号）Correlation between uncertainty and Realism (a complementary signal)
02方法Method
核心思路：把分类任务中"预测不确定性"（predictive uncertainty）的概念迁移到生成模型——用后验预测分布的方差来衡量每个样本的可靠程度，并借助 last-layer Laplace approximation 和语义似然使之在大型模型上高效可行。Core idea: transfer the notion of predictive uncertainty from classification to generative models, using the variance of the posterior predictive distribution to measure how reliable each sample is, and making this efficient and feasible on large models through a last-layer Laplace approximation together with a semantic likelihood.
图2：二维高斯玩具示例。训练 ensemble (M=5) 后，过滤掉 50% 高不确定性样本，有效去除了两个模式之间的"幻觉"样本，分布质量大幅提升。Figure 2: A two-dimensional Gaussian toy example. After training an ensemble (M=5), filtering out the 50% of samples with the highest uncertainty effectively removes the hallucinated samples lying between the two modes, greatly improving the quality of the distribution.
生成不确定性定义Definition of Generative Uncertainty
对于给定的隐变量 z，生成不确定性定义为后验预测分布的变分度（variability）： u(z) := V(p(x | z, D))
其中 V(·) 为熵（entropy），p(x | z, D) 通过对参数后验积分得到。高 u(z) 意味着不同参数设置会产生差异显著的输出，即该样本是不可靠的。For a given latent variable z, the generative uncertainty is defined as the variability of the posterior predictive distribution: u(z) := V(p(x | z, D)) where V(·) is the entropy and p(x | z, D) is obtained by integrating over the parameter posterior. A large u(z) means that different parameter settings yield markedly different outputs, i.e. the sample is unreliable.
对超过 1 亿参数的扩散模型直接做完整贝叶斯推断计算上不可行。本文只在模型最后一层施加 Laplace 近似，将参数后验近似为高斯分布： q(θ | D) = N(θ | θ̂, Σ)，其中 Σ = (∇²_θ L(θ; D)|_θ̂)⁻¹
这样仅需在训练后一次性计算 Hessian 逆，无需重新训练，天然兼容任意预训练扩散或 Flow Matching 模型（post-hoc 方法）。Full Bayesian inference is computationally infeasible for diffusion models with more than one hundred million parameters. This paper applies the Laplace approximation only to the last layer of the model, approximating the parameter posterior by a Gaussian: q(θ | D) = N(θ | θ̂, Σ), where Σ = (∇²_θ L(θ; D)|_θ̂)⁻¹ Only a one-off computation of the inverse Hessian after training is required, with no retraining, which makes the approach naturally compatible with any pretrained diffusion or Flow Matching model (a post-hoc method).
语义似然（Semantic Likelihood）Semantic Likelihood
像素空间的似然在高维情况下失效（维度诅咒）。本文引入基于预训练编码器（如 CLIP）的语义似然： p(x | g_θ(z); φ) = N(e(x) | c_φ(g_θ(z)), σ²I)
将生成图像和目标图像都投影到语义特征空间，再计算 Gaussian 似然。这使不确定性估计关注视觉语义质量而非像素级细节，大幅提升了对低质量样本的识别能力。Likelihoods in pixel space break down in high dimensions (the curse of dimensionality). This paper introduces a semantic likelihood based on a pretrained encoder such as CLIP: p(x | g_θ(z); φ) = N(e(x) | c_φ(g_θ(z)), σ²I) Both the generated image and the target image are projected into a semantic feature space, where a Gaussian likelihood is computed. This makes uncertainty estimation focus on visual semantic quality rather than pixel-level detail, greatly improving the ability to identify low-quality samples.
03实验Experiments
在 ImageNet 256×256 上分别使用 UViT 和 ADM 两个扩散模型进行评估，与随机基线、BayesDiff 以及 aleatoric uncertainty (AU) 方法对比，指标为 FID、Precision、Recall。Evaluation is carried out on ImageNet 256×256 with two diffusion models, UViT and ADM, in comparison with a random baseline, BayesDiff and the aleatoric uncertainty (AU) method, using FID, Precision and Recall as metrics.
模型 / 方法Model / Method
n=10K FID ↓
Precision ↑
备注Notes
Ours (UViT)
7.89
~0.73
M=5, T=50
BayesDiff (UViT)
9.16
~0.67
—
AU / Aleatoric Unc.
9.20
—
—
随机基线Random baseline
9.45
—
—
Ours (ADM)
10.36
—
M=1, T=25（轻量版）M=1, T=25 (lightweight setting)
BayesDiff (ADM)
11.20
—
—
随机基线 (ADM)Random baseline (ADM)
11.31
—
—
图3：在 UViT 模型上，随过滤比例（filtering ratio）增大，本文方法的 FID 持续下降，始终优于 BayesDiff 和随机基线，Precision 也同步提升。Figure 3: On the UViT model, as the filtering ratio increases, the FID of this method keeps decreasing and remains consistently better than BayesDiff and the random baseline, while Precision improves in step.图4：计算效率 Pareto 图。横轴为额外 NFE（network function evaluations），纵轴为 FID。本文方法在不同计算预算下均优于 BayesDiff；优化配置 M=1, T=25 仅需 25 额外 NFE，即可超越所有基线。Figure 4: Pareto plot of computational efficiency. The horizontal axis is the extra NFE (network function evaluations) and the vertical axis is FID. This method outperforms BayesDiff across computational budgets; the optimized configuration M=1, T=25 needs only 25 extra NFE to surpass all baselines.图5：低不确定性样本示例（即模型最"有把握"的生成图像），视觉质量高、类别特征清晰，是各类别的"典范"图像。Figure 5: Examples of low-uncertainty samples, i.e. the generations the model is most confident about; they are of high visual quality with clear class characteristics and serve as exemplars of their classes.
Ablations 与补充分析Ablations and Additional Analyses
互补性：生成不确定性与 realism score 相关性仅为 -0.27，与 rarity score 相关性为 0.38，说明提供了独立的质量信号。联合过滤可进一步将 FID 降至 7.60（vs. 单独 realism+rarity 的 8.26）。Complementarity: generative uncertainty correlates with the realism score at only -0.27 and with the rarity score at 0.38, showing that it provides an independent quality signal. Joint filtering lowers FID further to 7.60 (vs. 8.26 for realism+rarity alone).
轻量化：将 M 从 5 减至 1、T 从 50 减至 25，额外 NFE 从 250 降至 25（降低 10×），性能仅略有损失，仍大幅超越随机基线。Lightweight variant: reducing M from 5 to 1 and T from 50 to 25 lowers the extra NFE from 250 to 25 (a 10× reduction) with only a slight loss in performance, still far surpassing the random baseline.
Flow Matching：方法无缝适配 Flow Matching 模型，验证了 post-hoc 框架的通用性。Flow Matching: the method transfers seamlessly to Flow Matching models, confirming the generality of the post-hoc framework.
类别分布偏移：过滤高不确定性样本后，不同类别被过滤比例不同，导致类别分布轻微偏移。Class distribution shift: after filtering out high-uncertainty samples, different classes are filtered at different rates, causing a slight shift in the class distribution.
04局限性Limitations
Note:以下局限性部分由作者在论文中明确指出，部分为方法设计的自然结论。Some of the limitations below are stated explicitly by the authors in the paper; others follow naturally from the design of the method.
对预训练编码器的强依赖性Strong dependence on a pretrained encoder
语义似然依赖 CLIP 等预训练图像编码器，导致方法仅适用于自然图像领域。对于分子结构、文本、音频等其他扩散模型擅长的模态，目前暂无合适的编码器，方法无法直接迁移。The semantic likelihood relies on pretrained image encoders such as CLIP, which restricts the method to the natural-image domain. For other modalities at which diffusion models excel, such as molecular structures, text and audio, no suitable encoder is currently available, so the method cannot be transferred directly.
Laplace 近似的理论正当性不完备Incomplete theoretical justification of the Laplace approximation
作者明确指出："Applying the Laplace approximation directly, without such reweighting, is not fully theoretically justified"——扩散模型的训练损失包含时间步加权，不严格符合 Laplace 理论要求的 likelihood + prior 形式，因此后验近似的理论保证存在缺口。The authors state explicitly: "Applying the Laplace approximation directly, without such reweighting, is not fully theoretically justified" — the training loss of diffusion models includes time-step weighting and does not strictly take the likelihood + prior form required by Laplace theory, so there is a gap in the theoretical guarantees of the posterior approximation.
对角近似可能低估后验复杂度The diagonal approximation may underestimate posterior complexity
为保持计算可行性，采用对角（diagonal）last-layer Laplace 近似，忽略参数间协方差。这可能无法完整捕捉真实后验的复杂结构，使不确定性估计的精度受限。论文作者也指出需要更系统地比较不同推断方法。To keep computation tractable, a diagonal last-layer Laplace approximation is used, ignoring covariance between parameters. This may fail to capture the full complexity of the true posterior and limits the accuracy of the uncertainty estimates. The authors also note that a more systematic comparison of different inference methods is needed.
过滤高不确定性样本会降低 recall（即样本多样性），这与其他基于 guidance 的过滤方法面临的问题一致。用户在追求质量提升的同时需接受一定程度的多样性损失。Filtering out high-uncertainty samples reduces recall, i.e. sample diversity, which is the same issue faced by other guidance-based filtering methods. Users pursuing better quality must accept a certain loss of diversity.
类别分布偏移Class distribution shift
在 ImageNet 1000 类的条件生成中，不同类别的高不确定性比例不同，过滤后类别分布发生偏移，可能影响某些需要均匀类别覆盖的下游应用。In class-conditional generation over the 1000 ImageNet classes, the proportion of high-uncertainty samples differs across classes, so filtering shifts the class distribution, which may affect downstream applications that require uniform class coverage.
