Flow-GRPO：通过在线强化学习训练 Flow Matching 模型Flow-GRPO: Training Flow Matching Models via Online RL

01 动机（Motivation）Motivation

Flow matching 模型（如 SD3.5、FLUX）在图像生成上表现出色，但在多对象组合、属性绑定和文字渲染等复杂场景下仍有明显短板。在线强化学习（online RL）已被证明能显著提升大语言模型的推理能力，然而将其应用于 flow matching 模型面临两大根本挑战。Flow matching models (such as SD3.5 and FLUX) excel at image generation, yet still fall clearly short in complex settings such as multi-object composition, attribute binding and text rendering. Online reinforcement learning (online RL) has been shown to markedly improve the reasoning ability of large language models, but applying it to flow matching models faces two fundamental challenges.

"This need for stochasticity in RL conflicts with the deterministic nature of flow matching models." — Flow matching 依赖确定性 ODE 采样，无法直接支持 RL 所需的随机探索；同时 flow model 推理需要大量去噪步数，数据收集代价极高。"This need for stochasticity in RL conflicts with the deterministic nature of flow matching models." — Flow matching relies on deterministic ODE sampling and cannot directly support the stochastic exploration that RL requires; at the same time, flow-model inference needs a large number of denoising steps, making data collection extremely costly.

核心挑战Key Challenges

• 确定性 vs. 随机性：RL 需要通过随机采样探索动作空间，而 flow matching 使用确定性 ODE，无法直接进行统计采样，policy ratio（Eq. 5 中的 r_t）在 deterministic 动态下计算代价极高。Determinism vs. stochasticity: RL has to explore the action space through random sampling, whereas flow matching uses a deterministic ODE that allows no direct statistical sampling, and computing the policy ratio (r_t in Eq. 5) under deterministic dynamics is prohibitively expensive.
• 采样效率瓶颈：Flow model 推理默认需要 T=40 个去噪步骤，在线 RL 每次收集训练数据都需完整运行，GPU 小时消耗极大，限制了实用性。Sampling-efficiency bottleneck: Flow-model inference needs T=40 denoising steps by default, and every round of online RL data collection has to run the full chain, which consumes an enormous number of GPU hours and limits practicality.
• Reward Hacking 风险：奖励模型可能被"钻空子"——奖励分数上涨而图像质量或多样性下降，这在 T2I 任务中尤为需要防范。Reward hacking risk: The reward model can be "gamed" — the reward score climbs while image quality or diversity degrades, which is particularly worth guarding against in T2I tasks.

02 方法（Method）Method

Flow-GRPO 将 GRPO 扩展到 flow matching 模型，核心包含两项技术创新：（1）ODE-to-SDE 转换，将确定性 ODE 采样转化为保持边缘分布不变的等价 SDE；（2）Denoising Reduction，训练时大幅减少去噪步数以加速数据收集，而推理时保留原始步数以保证质量。Flow-GRPO extends GRPO to flow matching models with two core technical innovations: (1) ODE-to-SDE conversion, which turns deterministic ODE sampling into an equivalent SDE that preserves the marginal distributions; (2) Denoising Reduction, which sharply cuts the number of denoising steps during training to speed up data collection, while keeping the original number of steps at inference time to guarantee quality.

1. Denoising as a Markov Decision Process

Flow matching 的去噪过程被建模为 MDP（S, A, P, R）：状态 s_t = (c, t, x_t)，动作 a_t = x_{t-1}（模型预测的去噪样本），策略 π_θ 由 flow model 参数化。奖励仅在最终时步给出：R(s_T, a_T) = r(x_0, c)。这一 MDP 视角使得策略梯度方法可以直接适用。The denoising process of flow matching is modeled as an MDP (S, A, P, R): the state is s_t = (c, t, x_t), the action is a_t = x_{t-1} (the denoised sample predicted by the model), and the policy π_θ is parameterized by the flow model. The reward is given only at the final timestep: R(s_T, a_T) = r(x_0, c). This MDP view makes policy-gradient methods directly applicable.

2. ODE-to-SDE 转换（实现随机性）2. ODE-to-SDE Conversion (Introducing Stochasticity)

原始 flow matching 使用确定性 ODE（dx_t = v_t dt），无法满足 RL 的随机采样需求。Flow-GRPO 将其转换为等价的反向时间 SDE：The original flow matching uses a deterministic ODE (dx_t = v_t dt), which cannot meet the random-sampling requirement of RL. Flow-GRPO converts it into an equivalent reverse-time SDE:

dx_t = [v_θ(x_t) + (σ_t²/2t)(x_t + (1-t)v_θ(x_t))] dt + σ_t dw

其中 σ_t = a√(t/(1-t))，a 为控制随机程度的超参数。经 Euler-Maruyama 离散化后，策略 π_θ(x_{t-1} | x_t, c) 成为各向同性高斯分布，可以闭式计算 KL 散度，无需额外打分网络。关键数学保证：此 SDE 与原 ODE 在每个时间步的边缘分布完全相同，即转换不改变模型的生成分布。Here σ_t = a√(t/(1-t)), with a a hyperparameter controlling the degree of stochasticity. After Euler-Maruyama discretization, the policy π_θ(x_{t-1} | x_t, c) becomes an isotropic Gaussian, so the KL divergence can be computed in closed form without an extra scoring network. The key mathematical guarantee: this SDE has exactly the same marginal distribution as the original ODE at every timestep, i.e. the conversion does not change the model's generative distribution.

3. GRPO 目标函数3. GRPO Objective

GRPO 无需 critic/value network，通过组内相对归一化来估计优势：对每个提示词 c，采样 G 张图像，优势 Â_i = (R_i − mean(R)) / std(R)。训练目标：GRPO needs no critic/value network and estimates the advantage by relative normalization within a group: for each prompt c, G images are sampled and the advantage is Â_i = (R_i − mean(R)) / std(R). The training objective:

J_Flow-GRPO = E[f(r, Â, θ, ε, β)]，其中 f 包含 clip 截断的 PPO-style 比率项与 KL 正则项 −β·D_KL(π_θ ‖ π_ref)J_Flow-GRPO = E[f(r, Â, θ, ε, β)], where f contains a clipped PPO-style ratio term and the KL regularizer −β·D_KL(π_θ ‖ π_ref)

KL 惩罚是防止 reward hacking 的关键：它将模型约束在预训练权重附近，从而保留图像质量和多样性。The KL penalty is the key to preventing reward hacking: it constrains the model to stay near the pretrained weights, thereby preserving image quality and diversity.

4. Denoising Reduction（训练加速）4. Denoising Reduction (Training Speedup)

在线 RL 训练时，每次采集样本需运行完整去噪链，成本高昂。实验发现：将训练时去噪步数从 T=40 减少到 T=10，可获得超过 4× 的采样加速，而最终奖励和图像质量不受影响。推理时仍使用 T=40，保持生成质量。进一步减少至 T=5 则不一致地降低训练效率。During online RL training, every round of sample collection has to run the full denoising chain, which is costly. Experiments find that reducing the training-time denoising steps from T=40 to T=10 gives more than 4× sampling speedup, while the final reward and image quality are unaffected. Inference still uses T=40 to keep generation quality. Reducing further to T=5 lowers training efficiency inconsistently.

03 实验（Experiments）Experiments

在三类任务上评估 Flow-GRPO：（1）组合图像生成（GenEval benchmark）；（2）视觉文字渲染（OCR 准确率）；（3）人类偏好对齐（PickScore）。骨干模型为 Stable Diffusion 3.5-Medium（SD3.5-M）。图像质量通过 DrawBench 上的 Aesthetic Score、DeQA、ImageReward 和 UnifiedReward 独立评估以检测 reward hacking。Flow-GRPO is evaluated on three kinds of tasks: (1) compositional image generation (GenEval benchmark); (2) visual text rendering (OCR accuracy); (3) human preference alignment (PickScore). The backbone model is Stable Diffusion 3.5-Medium (SD3.5-M). Image quality is assessed independently with Aesthetic Score, DeQA, ImageReward and UnifiedReward on DrawBench in order to detect reward hacking.

GenEval 基准（组合图像生成）GenEval Benchmark (Compositional Image Generation)

视觉文字渲染与人类偏好结果（Table 2）Visual Text Rendering and Human Preference Results (Table 2)

注：不加 KL 正则时（w/o KL），Aesthetic Score 从 5.39 降至 4.93，DeQA 从 4.07 降至 2.77，出现明显 reward hacking（图像质量下降）。加入 KL 后质量指标几乎与基线持平。Note: without KL regularization (w/o KL), the Aesthetic Score drops from 5.39 to 4.93 and DeQA drops from 4.07 to 2.77, showing obvious reward hacking (degraded image quality). Once KL is added, the quality metrics are almost on par with the baseline.

Ablation 分析Ablation Analysis

• Denoising Reduction：训练步数从 T=40 减至 T=10，可实现超过 4× 加速，最终性能持平；进一步减至 T=5 不能一致地提速，且有时拖慢训练。Denoising Reduction: reducing the training steps from T=40 to T=10 achieves more than 4× speedup with on-par final performance; reducing further to T=5 does not speed things up consistently and sometimes slows training down.
• 噪声水平 a：a=0.1 时探索不足，训练慢；a=0.7 时 OCR 准确率最高；a=1.0 时图像质量崩溃，奖励归零。推荐 a=0.7。Noise level a: at a=0.1 exploration is insufficient and training is slow; at a=0.7 the OCR accuracy is highest; at a=1.0 image quality collapses and the reward drops to zero. a=0.7 is recommended.
• Group Size G：G=24 时训练稳定；G=12 和 G=6 时训练不稳定甚至崩溃。较小的组产生不准确的优势估计，引发高方差。Group Size G: training is stable at G=24; at G=12 and G=6 training is unstable and can even collapse. Smaller groups produce inaccurate advantage estimates and induce high variance.
• 对比其他对齐方法：Flow-GRPO 一致超越有监督微调（SFT）、Flow-DPO 的离线与在线变体。Comparison with other alignment methods: Flow-GRPO consistently outperforms supervised fine-tuning (SFT) as well as the offline and online variants of Flow-DPO.
• 泛化性：在训练于 60 类物体、测试于 20 类未见物体时，Flow-GRPO Overall 0.90 vs. 基线 0.64；训练时计数 2-4 个物体，测试时 5-6 个物体也有显著提升（0.48 vs. 0.13）。Generalization: when trained on 60 object classes and tested on 20 unseen classes, Flow-GRPO reaches an Overall of 0.90 vs. 0.64 for the baseline; training with counts of 2-4 objects and testing with 5-6 objects also brings a clear gain (0.48 vs. 0.13).

04 局限性（Limitations）Limitations