QR-DQN: 基于分位数回归的分布式强化学习

01 动机（Motivation）Motivation

传统的 Q-learning 将回报（return）的随机性平均掉，只估计期望值。Bellemare 等人（C51）虽然提出了分布式 Bellman 算子并证明其在 Wasserstein 度量下是收缩的，却因随机梯度无法直接最小化 Wasserstein 损失而被迫退而求其次，使用 KL 散度加上启发式投影，留下了一个理论与算法之间的缺口。Classical Q-learning averages away the randomness of the return and estimates only its expectation. Bellemare et al. (C51) did propose the distributional Bellman operator and prove that it is a contraction under the Wasserstein metric, but because stochastic gradients cannot directly minimize the Wasserstein loss they had to settle for a KL divergence plus a heuristic projection, leaving a gap between the theory and the algorithm.

"This negative result left open the question as to whether it is possible to devise an online distributional reinforcement learning algorithm which takes advantage of the contraction result... In this paper, we answer this question affirmatively."

C51 projection step — **Figure 1（来自论文）：**C51 的投影方式——将目标分布按距最近支撑点的距离反比分配质量，然后最小化投影目标与预测之间的 KL 散度。此投影步骤是启发式的，与 Wasserstein 理论结果存在脱节。**Figure 1 (from the paper):** The C51 projection: mass of the target distribution is assigned to the nearest support points in inverse proportion to their distance, and the KL divergence between the projected target and the prediction is then minimized. This projection step is heuristic and is disconnected from the Wasserstein theoretical result.

1-Wasserstein minimizing projection — **Figure 2（来自论文）：**QR-DQN 的分位数投影——用 N=4 个等权 Dirac 的分位数中点最小化 1-Wasserstein 距离。阴影面积之和即为 1-Wasserstein 误差，可以通过分位数回归以无偏随机梯度直接优化。**Figure 2 (from the paper):** The QR-DQN quantile projection: the quantile midpoints of N=4 equally weighted Diracs minimize the 1-Wasserstein distance. The sum of the shaded areas is exactly the 1-Wasserstein error, which can be optimized directly by quantile regression with unbiased stochastic gradients.

+33%中位数得分相对 C51 提升（Huber 分位数损失）Median score improvement over C51 (Huber quantile loss)

211%QR-DQN-1 在 57 Atari 游戏上的中位数人类归一化得分（best agent）Median human-normalized score of QR-DQN-1 over 57 Atari games (best agent)

915%QR-DQN-1 平均人类归一化得分（best agent）Mean human-normalized score of QR-DQN-1 (best agent)

N=200最优分位数数量（超参数搜索结果）Optimal number of quantiles (from the hyper-parameter search)

为何 Wasserstein 度量具有吸引力？Why is the Wasserstein metric attractive?

Wasserstein 度量（又称 Earth Mover's Distance，EMD）是分布间的积分概率度量，能够考虑不同结果之间的距离，而 KL 散度在支撑不相交时会产生问题。Lemma 3（来自 C51）已经证明分布式 Bellman 算子 T^π 在 d̄_p（最大化形式的 p-Wasserstein 度量）下是 γ-收缩的：The Wasserstein metric (also known as the Earth Mover's Distance, EMD) is an integral probability metric between distributions that takes the distance between different outcomes into account, whereas the KL divergence breaks down when the supports are disjoint. Lemma 3 (from C51) already proved that the distributional Bellman operator T^π is a γ-contraction under d̄_p (the maximal form of the p-Wasserstein metric):
d̄_p(T^πZ₁, T^πZ₂) ≤ γ · d̄_p(Z₁, Z₂)

但随机梯度下降无法直接最小化 Wasserstein 损失（Theorem 1，Bellemare et al. 2017）：对于样本经验分布，期望样本损失的最优解与真实 Wasserstein 损失的最优解通常不同。因此，C51 实际用的是 KL 散度，而非 Wasserstein，导致理论与实现脱节。QR-DQN 的核心贡献是找到了一种参数化方式，使得分位数回归可以给出 Wasserstein 的无偏随机梯度。But stochastic gradient descent cannot directly minimize the Wasserstein loss (Theorem 1, Bellemare et al. 2017): for a sample empirical distribution, the minimizer of the expected sample loss generally differs from the minimizer of the true Wasserstein loss. C51 therefore actually uses the KL divergence rather than the Wasserstein metric, which disconnects the theory from the implementation. The core contribution of QR-DQN is a parameterization under which quantile regression yields unbiased stochastic gradients of the Wasserstein loss.

02 方法（Method）Method

QR-DQN 将 C51 的参数化"转置"：C51 用固定位置 + 可变概率，而 QR-DQN 用固定等概率（1/N）+ 可变位置。每个位置对应一个分位数中点，通过分位数回归以无偏随机梯度端到端最小化 1-Wasserstein 距离。QR-DQN "transposes" the C51 parameterization: C51 uses fixed locations with variable probabilities, whereas QR-DQN uses fixed uniform probabilities (1/N) with variable locations. Each location corresponds to one quantile midpoint, and the 1-Wasserstein distance is minimized end-to-end by quantile regression with unbiased stochastic gradients.

量化分位数分布（Quantile Distribution）The Quantile Distribution

对于固定的 N，定义分位数分布为：For a fixed N, the quantile distribution is defined as:
Z_θ(x,a) := (1/N) Σ_i=1..N δ_θᵢ(x,a)
其中 δ_z 是在 z 处的 Dirac 质量，θᵢ 是可学习的位置参数。where δ_z is a Dirac mass at z and the θᵢ are learnable location parameters. 分位数中点为 τ̂ᵢ = (τᵢ₋₁ + τᵢ) / 2，其中 τᵢ = i/N。The quantile midpoints are τ̂ᵢ = (τᵢ₋₁ + τᵢ) / 2, where τᵢ = i/N.

相比 C51，量化分位数分布有三大优势：Compared with C51, the quantile distribution has three major advantages:

不需要预先指定回报范围（V_min, V_max），分布可自适应扩展或收缩；the return range (V_min, V_max) need not be specified in advance, and the distribution can expand or contract adaptively;
不需要 C51 的启发式投影步骤（不存在支撑不相交的问题）；the heuristic projection step of C51 is not needed (disjoint supports are no longer an issue);
可以用分位数回归以无偏梯度最小化 Wasserstein 损失。the Wasserstein loss can be minimized by quantile regression with unbiased gradients.

分位数回归损失（Quantile Regression Loss）Quantile Regression Loss

对于分位数 τ ∈ [0,1]，分位数回归损失是一种非对称凸损失，过高估计以权重 τ 惩罚，过低估计以权重 1-τ 惩罚：For a quantile τ ∈ [0,1], the quantile regression loss is an asymmetric convex loss that penalizes overestimation with weight τ and underestimation with weight 1-τ:
L^τ_QR(θ) = E_Ẑ∼Z[ρ_τ(Ẑ - θ)]，其中 ρ_τ(u) = u(τ - 𝟙{u<0})L^τ_QR(θ) = E_Ẑ∼Z[ρ_τ(Ẑ - θ)], where ρ_τ(u) = u(τ - 𝟙{u<0})

由 Lemma（w1_midpoint）可知，令 θᵢ = F^-1_Z(τ̂ᵢ) 即可最小化 1-Wasserstein 距离。By the Lemma (w1_midpoint), setting θᵢ = F^-1_Z(τ̂ᵢ) minimizes the 1-Wasserstein distance. 因此，以下目标函数的最小化等价于最小化 W₁(Z, Z_θ)：Minimizing the following objective is therefore equivalent to minimizing W₁(Z, Z_θ):
Σᵢ E_Ẑ∼Z[ρ_τ̂ᵢ(Ẑ - θᵢ)]

此损失提供无偏样本梯度，可直接用随机梯度下降优化。This loss provides unbiased sample gradients and can be optimized directly by stochastic gradient descent.

Quantile Huber 损失Quantile Huber Loss

标准分位数回归损失在零点不光滑，可能限制非线性函数近似的性能。论文引入 Quantile Huber Loss：在 [-κ, κ] 区间内用非对称平方损失，超出区间回退到标准分位数损失：The standard quantile regression loss is not smooth at zero, which may limit the performance of non-linear function approximation. The paper introduces the Quantile Huber Loss: an asymmetric squared loss on the interval [-κ, κ] that falls back to the standard quantile loss outside that interval:
ρ^κ_τ(u) = |τ - 𝟙{u<0}| · L_κ(u)
其中 L_κ(u) = u²/2 if |u|≤κ, else κ(|u| - κ/2)（Huber 损失）。实验中 κ=1，记为 QR-DQN-1。where L_κ(u) = u²/2 if |u|≤κ, else κ(|u| - κ/2) is the Huber loss. The experiments use κ=1, denoted QR-DQN-1.

理论保证：收缩性Theoretical Guarantee: Contraction

论文证明（Proposition），分位数投影 Π_W₁ 与分布式 Bellman 算子 T^π 的组合在 d̄_∞ 下是 γ-收缩的：The paper proves (Proposition) that the composition of the quantile projection Π_W₁ with the distributional Bellman operator T^π is a γ-contraction under d̄_∞:
d̄_∞(Π_W₁ T^π Z₁, Π_W₁ T^π Z₂) ≤ γ · d̄_∞(Z₁, Z₂)
这意味着组合算子存在唯一不动点，算法（及其随机近似）收敛到该不动点，且对所有 p ∈ [1,∞] 收敛。这是 distributional RL 在 Wasserstein 度量下端到端保证的首个实例。This means that the composed operator has a unique fixed point, that the algorithm (and its stochastic approximation) converges to that fixed point, and that convergence holds for all p ∈ [1,∞]. This is the first instance of an end-to-end guarantee for distributional RL under the Wasserstein metric.

QR-DQN 算法The QR-DQN Algorithm

QR-DQN 相对 DQN 有三处修改：QR-DQN makes three modifications to DQN:

输出层大小改为 |A| × N（每个动作 N 个分位数）；the output layer size becomes |A| × N (N quantiles per action);
Huber 损失替换为 Quantile Huber 损失（κ=1）；the Huber loss is replaced by the quantile Huber loss (κ=1);
优化器从 RMSProp 改为 Adam。the optimizer is changed from RMSProp to Adam.

Algorithm: Quantile Regression Q-Learning (QR-DQN)
# 输入: x, a, r, x', N 个分位数数量, κ (Huber 参数)# Input: x, a, r, x', N number of quantiles, κ (Huber parameter)
Q(x', a') := Σ_j q_j θ_j(x', a') # 计算期望 Q 值# compute the expected Q-value
a* ← argmax_a' Q(x', a') # 贪心动作选择# greedy action selection
TΘ_j ← r + γ θ_j(x', a*) ∀j # 分布式 Bellman 目标# distributional Bellman target
输出：Output: Σ_i=1..N E_j[ρ^κ_τ̂ᵢ(TΘ_j - θ_i(x,a))] # Quantile Huber Loss

03 实验（Experiments）Experiments

实验分两部分：（1）在经典的两室风格子世界（windy gridworld）上验证 QR-TD 确实学习到真实回报分布；（2）在 57 个 Atari 2600 游戏上对比 DQN、DDQN、Dueling、Prioritized Replay、C51 等 baseline，使用人类归一化得分评估。The experiments come in two parts: (1) a classical two-room windy gridworld verifying that QR-TD really does learn the true return distribution; (2) a comparison against baselines such as DQN, DDQN, Dueling, Prioritized Replay and C51 on 57 Atari 2600 games, evaluated with human-normalized scores.

Atari 2600 — Best Agent 性能Atari 2600 — Best Agent Performance

超参数搜索结果：α = 0.00005，ε_ADAM = 0.01/32，N = 200。以下数据来自论文 Table 1（200 million 训练帧，57 游戏）：Hyper-parameter search results: α = 0.00005, ε_ADAM = 0.01/32, N = 200. The figures below are taken from Table 1 of the paper (200 million training frames, 57 games):

算法Algorithm	Mean (human-norm.)	Median (human-norm.)	>Human	>DQN
DQN	228%	79%	24	0
DDQN	307%	118%	33	43
Dueling	373%	151%	37	50
Prioritized	434%	124%	39	48
Pr. Dueling	592%	172%	39	44
C51	701%	178%	40	50
QR-DQN-0 (κ=0)	881%	199%	38	52
QR-DQN-1 (κ=1)	915%	211%	41	54

Online evaluation results on 57 Atari 2600 games — **Figure（来自论文）：**在线评估结果，57 个 Atari 2600 游戏，200 million 训练样本，人类归一化得分。左：单 seed 测试性能，显示各游戏中位数。右：三 seed 平均训练性能，显示第 10、20、30、40、50 百分位数。QR-DQN 在几乎所有阶段和百分位均优于 C51。**Figure (from the paper):** Online evaluation results, 57 Atari 2600 games, 200 million training samples, human-normalized score. Left: test performance for a single seed, showing the median across games. Right: training performance averaged over three seeds, showing the 10th, 20th, 30th, 40th and 50th percentiles. QR-DQN beats C51 at almost every stage and percentile.

Value Distribution 近似误差（Windy Gridworld）Value Distribution Approximation Error (Windy Gridworld)

在含随机转移的两室风格子世界中，以 1K Monte-Carlo rollout 估计真实分布，运行 TD 和 QR-TD 各 10K episodes（N=32，学习率 α=0.1）。结果显示 QR-TD 正确收敛并最小化了与 MC 估计之间的 1-Wasserstein 距离，而标准 TD 仅收敛均值。In a two-room windy gridworld with stochastic transitions, the true distribution is estimated with 1K Monte-Carlo rollouts, and TD and QR-TD are each run for 10K episodes (N=32, learning rate α=0.1). The results show that QR-TD converges correctly and minimizes the 1-Wasserstein distance to the MC estimate, while standard TD converges only in the mean.

Windy gridworld value distribution experiment — **Figure（来自论文）：**(a) 两室风格子世界，底行显示风力大小；蓝色路径为策略轨迹。(b,c) 起始状态 x_S 的（累积）价值分布，MC 估计 Z^π_MC 与 QR-TD 估计 Z_θ 对比。(d,e) TD 与 QR-TD 的价值函数/分布近似误差随 episode 数变化。两种算法在均值上均收敛，而 QR-TD 额外最小化了 1-Wasserstein 距离。**Figure (from the paper):** (a) The two-room windy gridworld; the bottom row shows the wind strength, and the blue path is the policy trajectory. (b,c) The (cumulative) value distribution of the start state x_S, comparing the MC estimate Z^π_MC with the QR-TD estimate Z_θ. (d,e) Value function/distribution approximation error of TD and QR-TD as a function of the number of episodes. Both algorithms converge in the mean, but QR-TD additionally minimizes the 1-Wasserstein distance.

Online 性能分析Online Performance Analysis

论文在 online 评估协议下得出三个关键发现：Under the online evaluation protocol the paper draws three key findings:

训练早期，所有算法在至少 10% 的游戏上表现低于随机；early in training, every algorithm performs below random on at least 10% of the games;
QR-TD 在样本复杂度上的提升与 Prioritized Replay 相当，同时还改善了最终性能；the sample-complexity gain of QR-TD is comparable to that of Prioritized Replay, while it also improves final performance;
即使到 200 million 帧，仍有 10% 的游戏所有算法均低于人类水平的 10%，说明近年进展在一小部分 Atari 游戏上仍严重受限。even at 200 million frames, on 10% of the games all algorithms stay below 10% of human performance, showing that recent progress remains severely limited on a small subset of Atari games.

04 局限性（Limitations）Limitations

注：论文无独立的 Limitations 节。以下第 1 条为论文明确指出；第 2–4 条为从设计推断（inferred from the design）。Note: The paper has no standalone Limitations section. Item 1 below is explicitly stated in the paper; items 2–4 are inferred from the design.

仍需期望值进行动作选择（stated）Action selection still requires the expectation (stated)

QR-DQN 学习了完整的回报分布，但动作选择仍依赖期望值：a* = argmax_a' E_z∼Z(x',a')[z]。论文明确指出这是当前设计的限制，并提出更丰富的策略类——基于完整分布进行风险敏感决策（risk-sensitive decision making）——是重要的未来方向。QR-DQN learns the full return distribution, but action selection still relies on the expectation: a* = argmax_a' E_z∼Z(x',a')[z]. The paper explicitly notes this as a limitation of the current design and proposes richer policy classes — risk-sensitive decision making based on the full distribution — as an important future direction.

尚未与 DQN 改进技术结合（stated）Not yet combined with DQN improvement techniques (stated)

论文测试的是"纯"QR-DQN，未叠加 Dueling architecture、Prioritized Replay 等近年对 DQN 的改进。论文明确指出："A natural next step would be to combine QR-DQN with the non-distributional methods found in Table 1。"（QR-DQN 在 54 个游戏上超过 DQN，但未必在所有改进组合上达到最优。）The paper tests "pure" QR-DQN, without stacking recent DQN improvements such as the Dueling architecture or Prioritized Replay. It states explicitly: "A natural next step would be to combine QR-DQN with the non-distributional methods found in Table 1." (QR-DQN beats DQN on 54 games, but is not necessarily optimal in combination with every improvement.)

收缩性仅在 d̄_∞ 下成立，p<∞ 时不直接成立（inferred）Contraction holds only under d̄_∞, not directly for p<∞ (inferred)

论文证明了 Π_W₁ T^π 在 d̄_∞ 下是收缩的，但明确指出 "the contraction property does not directly hold for p < ∞"（详见附录 Lemma）。这意味着理论保证最强在 ∞-Wasserstein 意义下成立，1-Wasserstein 下的收敛性需要额外论证。The paper proves that Π_W₁ T^π is a contraction under d̄_∞, but notes explicitly that "the contraction property does not directly hold for p < ∞" (see the appendix Lemma). This means the theoretical guarantee is strongest in the ∞-Wasserstein sense, and convergence under 1-Wasserstein requires additional argument.

N 是额外超参数，计算成本随 N² 增长（inferred）N is an extra hyper-parameter and the computational cost grows with N² (inferred)

QR-DQN 将输出层从 |A| 扩展到 |A|×N，且损失对所有 N² 对 (θᵢ, TΘⱼ) 求和，计算与内存成本均随 N² 增长。最优 N=200 是通过在五个训练游戏上超参数搜索得到的，增加了调参代价。QR-DQN widens the output layer from |A| to |A|×N, and the loss sums over all N² pairs (θᵢ, TΘⱼ), so both computation and memory cost grow with N². The optimal N=200 was obtained by a hyper-parameter search over five training games, which adds to the tuning cost.