GEAR: Guided End-to-End AutoRegression for Image Synthesis
让 VQ tokenizer 和自回归生成器联合端到端训练，用 representation alignment 反过来「指导」tokenizerTraining the VQ tokenizer and the autoregressive generator jointly end-to-end, using representation alignment to "guide" the tokenizer in return
Bin Lin, Zheyuan Liu, Chenguo Lin, Sixiang Chen, Yunyang Ge, Yunlong Lin, Jianwei Zhang, Miles Yang, Zhao Zhong, Liefeng Bo, Li Yuan · Peking University, Tencent Hunyuan
视觉生成模型通常分两阶段训练：先训 tokenizer 做重建，冻结后再训生成器。GEAR 打破这个分离，让 AR 生成器通过 representation-alignment 信号反向指导 VQ tokenizer，同时用 hard/soft 双读出机制绕开 VQ 索引不可微、straight-through estimator 会坍缩的难题，在 ImageNet 上把 gFID 收敛速度加速最多 10×，并推广到 VQVAE/LFQ/IBQ 等多种量化器与 text-to-image 生成。Visual generative models are usually trained in two stages: first a tokenizer is trained for reconstruction, then it is frozen and a generator is trained on top of it. GEAR breaks this separation and lets the AR generator guide the VQ tokenizer in reverse through a representation-alignment signal, while a hard/soft dual read-out mechanism sidesteps the non-differentiability of VQ indices and the collapse of the straight-through estimator, accelerating gFID convergence on ImageNet by up to 10× and generalizing to quantizers such as VQVAE/LFQ/IBQ as well as to text-to-image generation.
现代视觉生成模型几乎都是两阶段训练：先训一个 VQ-VAE（或连续 VAE）做图像重建，然后冻结它，再在其离散索引或连续 latent 上训练生成器。论文指出，这种解耦「让 tokenizer 对下游生成器认为『好建模』的东西一无所知」——重建偏好高方差、细节丰富的 latent，而生成偏好简单、可预测的结构，二者天然存在张力。Almost all modern visual generative models are trained in two stages: first a VQ-VAE (or a continuous VAE) is trained for image reconstruction, then it is frozen and a generator is trained on its discrete indices or continuous latents. The paper argues that this decoupling "leaves the tokenizer unaware of what the downstream generator finds easy to model" — reconstruction favors high-variance, detail-rich latents whereas generation favors simple, predictable structure, and the two are inherently in tension.
"This decoupling leaves the tokenizer unaware of what the generator finds easy to model."
扩散模型这边已经有 REPA、VA-VAE、MAETok、REPA-E 等工作尝试打破这个边界（对齐 latent 到 DINOv2 特征，甚至像 REPA-E 一样端到端联合训练 VAE 和 diffusion 模型）。但同样的机会在离散 VQ-AR 场景下更难实现：VQ 索引到 AR 输入是一个不可微的 arg max，梯度传不回 tokenizer；直接上 straight-through estimator（STE）这个常规补救方案，在 GEAR 的联合训练设定下会剧烈不稳定并坍缩（gFID ≈ 105）。On the diffusion side, works such as REPA, VA-VAE, MAETok and REPA-E have already tried to break this boundary (aligning latents to DINOv2 features, or even, as in REPA-E, training the VAE and the diffusion model jointly end-to-end). The same opportunity is harder to realize in the discrete VQ-AR setting: the mapping from VQ indices to AR inputs is a non-differentiable arg max, so gradients cannot flow back into the tokenizer; and the standard remedy of plugging in a straight-through estimator (STE) becomes violently unstable and collapses under GEAR's joint-training setup (gFID ≈ 105).
Figure 1：GEAR 加速并提升自回归图像生成。(a) ImageNet 上 gFID vs. 训练步数（无 CFG）：GEAR 比 LlamaGen-REPA 收敛快最多 10×，而通过 straight-through estimator 反传梯度到 tokenizer 的 naive 端到端方案会直接发散（gFID≈105）。(b) 1.5M 步时，B/L/XL 三个模型规模下 GEAR 在每个尺寸上都优于 baseline。Figure 1: GEAR accelerates and improves autoregressive image generation. (a) gFID vs. training steps on ImageNet (without CFG): GEAR converges up to 10× faster than LlamaGen-REPA, whereas the naive end-to-end variant that backpropagates gradients into the tokenizer through a straight-through estimator diverges outright (gFID≈105). (b) At 1.5M steps, GEAR outperforms the baseline at each of the three model scales B/L/XL.
10×ImageNet gFID 收敛加速（vs. LlamaGen-REPA）Speed-up of ImageNet gFID convergence (vs. LlamaGen-REPA)
11.1×REPA 对齐损失收缩加速（GPIC，同规格 tokenizer）Speed-up in shrinking the REPA alignment loss (GPIC, same-spec tokenizer)
2.5×NTP 损失收缩加速（GPIC，同规格 tokenizer）Speed-up in shrinking the NTP loss (GPIC, same-spec tokenizer)
GEAR（Guided End-to-end AutoRegression）联合、端到端训练 VQ tokenizer 和自回归（AR）生成器，核心是对「每个位置的 codebook assignment」做一个双读出（dual read-out）：一条 hard（one-hot）分支、一条 soft（温度缩放）分支，分别承担不同的梯度路径，从而绕开不可微索引的问题，同时避免 STE 式坍缩。GEAR (Guided End-to-end AutoRegression) trains the VQ tokenizer and the autoregressive (AR) generator jointly and end-to-end. Its core is a dual read-out over the "codebook assignment at each position": a hard (one-hot) branch and a soft (temperature-scaled) branch, which carry different gradient paths, thereby sidestepping the non-differentiable index while avoiding STE-style collapse.
Figure 2：GEAR 总览。(a) 传统流程：冻结预训练 VQ-VAE，只用 NTP + REPA 损失训练 AR。(b) Naive 端到端方案：通过 straight-through estimator（锐角箭头）把 AR 梯度直接传回 tokenizer，高度不稳定并坍缩。(c) GEAR：hard 分支承载 NTP 与 hard REPA 损失、只更新 AR；可微的 soft 分支承载一个绕过 AR 上层 block、反传（虚线箭头）只更新 tokenizer 的 REPA 损失，给出稳定的端到端指导信号。Figure 2: Overview of GEAR. (a) Conventional pipeline: the pretrained VQ-VAE is frozen and the AR is trained with NTP + REPA losses only. (b) Naive end-to-end variant: AR gradients are routed straight back into the tokenizer through a straight-through estimator (sharp arrows), which is highly unstable and collapses. (c) GEAR: the hard branch carries the NTP and hard REPA losses and updates the AR only; the differentiable soft branch carries a REPA loss that bypasses the upper AR blocks and backpropagates (dashed arrows) into the tokenizer only, providing a stable end-to-end guidance signal.
Hard 分支：训练 AR，保持推理一致Hard branch: trains the AR, stays consistent with inference
hard read-out 就是常规的 one-hot arg max 索引——推理时用的那套离散 token。它训练 AR 的 next-token prediction（NTP）损失，加上一个 hard REPA 对齐损失，只更新 AR 模型；论文特别强调「never route the NTP loss into the tokenizer, because letting it reshape the codebook invites a collapse to a few low-entropy codes that trades reconstruction for predictability」——如果把 NTP 损失也传给 tokenizer，会诱导 codebook 坍缩到少数低熵 code，用可预测性换重建质量。The hard read-out is just the conventional one-hot arg max index — the same discrete tokens used at inference. It drives the next-token prediction (NTP) loss of the AR plus a hard REPA alignment loss and updates the AR model only; the paper stresses "never route the NTP loss into the tokenizer, because letting it reshape the codebook invites a collapse to a few low-entropy codes that trades reconstruction for predictability" — feeding the NTP loss into the tokenizer as well would drive the codebook to collapse onto a handful of low-entropy codes, trading reconstruction quality for predictability.
Soft 分支：唯一能穿过不可微索引的梯度路径Soft branch: the only gradient path through the non-differentiable index
soft read-out 是对最近若干 codeword 的温度加权插值，因而可微。它承载一个 representation-alignment 损失，只反向更新 tokenizer（AR 主干、embedding 和投影头在这一步都固定），是整套流程中唯一能让端到端梯度真正流回 tokenizer 的路径——"succeeding precisely where the STE collapses"。有意思的是指导方向和 diffusion 侧（REPA-E/VA-VAE 让 latent 更「语义化」）恰恰相反：GEAR 里 tokenizer 自身特征反而更不像 DINOv2（尤其是 patch 级别），语义对齐被转移到了 AR 生成器内部——AR 的隐藏状态和 DINOv2 的逐 patch 相关性更强、空间结构更连贯，而这恰恰是让 next-token prediction 更容易的关键。The soft read-out is a temperature-weighted interpolation over the nearest few codewords and is therefore differentiable. It carries a representation-alignment loss that updates the tokenizer only in the backward pass (the AR backbone, the embeddings and the projection head are all held fixed at this step), and it is the one path in the whole pipeline through which end-to-end gradients really flow back into the tokenizer — "succeeding precisely where the STE collapses". Interestingly, the direction of guidance is exactly opposite to that on the diffusion side (where REPA-E/VA-VAE make the latents more "semantic"): in GEAR the tokenizer's own features instead become less DINOv2-like (especially at the patch level), and semantic alignment is relocated inside the AR generator — the AR hidden states correlate more strongly with DINOv2 patch by patch and are more spatially coherent, which is precisely what makes next-token prediction easier.
03 Experiments
主实验在 ImageNet-1K 256×256 类条件生成上，对比 LlamaGen 与 LlamaGen-REPA baseline，评测 gFID / sFID / IS / Precision / Recall（ADM 协议，50K 采样）。GEAR 与 LlamaGen-REPA 从同一个 warm-up tokenizer 出发；GEAR 先联合端到端微调 tokenizer 400K 步，再在这个改进后的、冻结的 tokenizer 上重新训练一个全新 AR，与 LlamaGen-REPA（在冻结 warm-up tokenizer 上训练）共享完全一致的 AR 结构与训练预算——这样任何差距都能归因到 tokenizer 本身，并证明它可以「即插」到标准冻结-tokenizer 流程中而不必支付端到端训练的额外成本。The main experiments are on ImageNet-1K 256×256 class-conditional generation, comparing against the LlamaGen and LlamaGen-REPA baselines and reporting gFID / sFID / IS / Precision / Recall (ADM protocol, 50K samples). GEAR and LlamaGen-REPA start from the same warm-up tokenizer; GEAR first fine-tunes the tokenizer jointly end-to-end for 400K steps, then trains a brand-new AR from scratch on this improved, frozen tokenizer, sharing exactly the same AR architecture and training budget as LlamaGen-REPA (trained on the frozen warm-up tokenizer) — so that any gap can be attributed to the tokenizer itself, and so as to show that it can be plugged into the standard frozen-tokenizer pipeline without paying the extra cost of end-to-end training.
规模 (Params)Scale (Params)
方法Method
gFID↓ (w/o CFG)
gFID↓ (w/ CFG)
B (111M)
LlamaGen-REPA
20.16
6.00
B (111M)
GEAR
16.96
4.95
L (343M)
LlamaGen-REPA
12.70
3.15
L (343M)
GEAR
8.66
2.95
XL (775M)
LlamaGen-REPA
8.20
2.68
XL (775M)
GEAR
6.76
2.52
300 epochs，同 AR 结构与训练预算，仅 tokenizer（frozen warm-up vs. GEAR 端到端改进后）不同。数据来自论文 Table 1。300 epochs, identical AR architecture and training budget; only the tokenizer differs (frozen warm-up vs. the end-to-end improved one from GEAR). Numbers from Table 1 of the paper.
Figure 3：在 GPIC 上的训练动态。两个 AR 分别在 GEAR 的端到端改进 tokenizer 与原始 tokenizer 上训练（tokenizer 均冻结，唯一变量是 tokenizer 质量）。要达到 baseline 的最终损失，GEAR 在 NTP 上快 2.5×，在 REPA 对齐损失上快 11.1×。Figure 3: Training dynamics on GPIC. Two ARs are trained on the end-to-end improved tokenizer of GEAR and on the original tokenizer respectively (both tokenizers frozen, so tokenizer quality is the only variable). To reach the final loss of the baseline, GEAR is 2.5× faster on NTP and 11.1× faster on the REPA alignment loss.
Ablations
论文的关键消融（Table 7）直接验证了 dual read-out 设计的必要性：把 AR 梯度通过 straight-through estimator 直接传回 tokenizer（w/ STE）会坍缩到 gFID 104.932（PSNR 12.596，SSIM 0.239），而 GEAR 的双读出方案把 gFID 拉回 10.630（PSNR 20.779，SSIM 0.558）——这正是 Figure 1 中「naive 端到端方案发散」的来源。此外论文还发现：随着 AR 规模从 B 增大到 XL，被 soft 指导信号间接监督的 tokenizer 本身的重建质量也在提升（rFID 1.658→1.624），说明「更强的 AR 模型提供了更好的指导信号」。The paper's key ablation (Table 7) directly validates the necessity of the dual read-out design: routing AR gradients straight back into the tokenizer through a straight-through estimator (w/ STE) collapses to gFID 104.932 (PSNR 12.596, SSIM 0.239), whereas the dual read-out of GEAR pulls gFID back to 10.630 (PSNR 20.779, SSIM 0.558) — exactly the origin of the "naive end-to-end variant diverges" curve in Figure 1. The paper further observes that as the AR scales from B up to XL, the reconstruction quality of the tokenizer itself, supervised only indirectly by the soft guidance signal, also improves (rFID 1.658→1.624), showing that "a stronger AR model provides a better guidance signal".
在 text-to-image 上，论文用 GPIC 语料做严格受控对比（同数据、同架构、同训练/推理设置，只换 tokenizer）：GEAR-1.0B 在 GenEval 上取得 short/long prompt 分别 0.086/0.227（w/o CFG）和 0.334/0.478（w/ CFG），均优于 LlamaGen-REPA-1.0B 的 0.074/0.218 与 0.272/0.419，验证了方法在 T2I 场景下的泛化性。For text-to-image, the paper runs a strictly controlled comparison on the GPIC corpus (same data, same architecture, same training/inference settings, only the tokenizer swapped): on GenEval, GEAR-1.0B reaches 0.086/0.227 for short/long prompts (w/o CFG) and 0.334/0.478 (w/ CFG), beating the 0.074/0.218 and 0.272/0.419 of LlamaGen-REPA-1.0B, which confirms that the method generalizes to the T2I setting.
Figure 4：端到端微调过程中的 codebook 使用情况。GEAR 的联合微调（τ=0.1）下，累计 top-1/10 assignment 质量、使用熵（nats）和有效 codebook 大小 exp(H) 随训练步数变化：分布先迅速变尖锐、在约 30K 步附近浓度达峰，然后放松并收敛到一个比 warm-up tokenizer 更集中、熵更低的状态。Figure 4: Codebook usage during end-to-end fine-tuning. Under the joint fine-tuning of GEAR (τ=0.1), the cumulative top-1/10 assignment mass, the usage entropy (nats) and the effective codebook size exp(H) evolve with training steps: the distribution first sharpens rapidly, peaks in concentration at around 30K steps, then relaxes and converges to a state that is more concentrated and lower in entropy than the warm-up tokenizer.
04 Limitations
Note：论文没有单独的 "Limitations" 小节，但第 5 节 "Discussion" 明确讨论了方法的边界与未解问题，以下均为论文原文陈述；仅最后一条标注为 inferred。Note: The paper has no standalone "Limitations" section, but Section 5, "Discussion", explicitly discusses the boundaries of the method and its open questions. Everything below is stated in the paper itself; only the last item is marked inferred.
在远小得多的训练预算下，GEAR 大幅缩小了 AR 生成与强 latent-diffusion baseline 的差距，但仍落后于最好的端到端 diffusion 方法 REPA-E：GEAR 的重建 rFID 1.64 上界了它的生成 gFID（2.52，w/ CFG），而 REPA-E 的连续 VAE 重建远更精确（rFID 0.28），达到 gFID 1.12。论文明确指出："Closing this reconstruction gap is the single largest lever for further improving VQ-AR generation."Under a far smaller training budget, GEAR substantially narrows the gap between AR generation and strong latent-diffusion baselines, yet still trails REPA-E, the best end-to-end diffusion method: the reconstruction rFID of 1.64 of GEAR upper-bounds its generation gFID (2.52, w/ CFG), whereas the continuous VAE of REPA-E reconstructs far more accurately (rFID 0.28) and reaches gFID 1.12. The paper states explicitly: "Closing this reconstruction gap is the single largest lever for further improving VQ-AR generation."
压缩率与计算量被架构耦合（Compression is coupled to compute in autoregression）Compression is coupled to compute in autoregression
当前 VQ-AR 流程中 tokenizer 做 16× 下采样，256×256 图像映射为 256 个 token，AR 模型的计算量恰好花在这 256 个 token 上——压缩率和序列长度被绑死。Latent diffusion 则用更温和的 8× tokenizer 保留 1024 个 latent 位置换取更高保真度，再用 2×2 patch embedding 让 transformer 仍操作在等效的 256 个 token 上，从而解耦了两者。论文将借鉴这种解耦（更温和的 tokenizer + patchified/multi-token prediction）列为提升重建上限的「promising route」，本文未实现。In the current VQ-AR pipeline the tokenizer downsamples by 16×, mapping a 256×256 image to 256 tokens, and the compute of the AR model is spent exactly on those 256 tokens — compression rate and sequence length are locked together. Latent diffusion instead uses a gentler 8× tokenizer that keeps 1024 latent positions in exchange for higher fidelity, then applies a 2×2 patch embedding so that the transformer still operates on an equivalent 256 tokens, thereby decoupling the two. The paper lists borrowing this decoupling (a gentler tokenizer + patchified/multi-token prediction) as a "promising route" toward a higher reconstruction ceiling, but does not implement it here.
Text-to-image 场景仍需扩展（stated as future work）Text-to-image still needs to be scaled up (stated as future work)
T2I 实验基于 GPIC 语料的单 epoch 严格受控对比，论文原文写道会将"stronger text conditioning"留给未来工作；结论部分也提出下一步是把 guided end-to-end training 扩展到更大的 text-to-image 模型和统一的理解-生成系统。The T2I experiments are a strictly controlled single-epoch comparison on the GPIC corpus, and the paper writes that "stronger text conditioning" is left to future work; the conclusion likewise proposes scaling guided end-to-end training to larger text-to-image models and to unified understanding-generation systems.
