Upside-Down Reinforcement Learning

01 动机Motivation

强化学习（RL）与监督学习（SL）之间有一道根本鸿沟：SL 提供误差信号（error signal）， 而 RL 提供评估信号（evaluation signal）——智能体只知道动作好不好，却不知道最优动作是什么。 能否完全用 SL 来解决 RL 问题？Barto（2004）曾断言「一般情况下这是不可能的」。 UDRL 正面回应了这一挑战。There is a fundamental gap between reinforcement learning (RL) and supervised learning (SL): SL provides an error signal, whereas RL provides an evaluation signal — the agent only learns whether an action was good, not what the optimal action would have been. Can RL problems be solved entirely with SL? Barto (2004) once asserted that this is impossible in general. UDRL takes that challenge head-on.

"Is it possible to learn to act in high-dimensional environments efficiently using only SL, avoiding the issues arising from non-stationary learning objectives common in traditional RL algorithms?" — Srivastava et al., 1912.02877

现有大多数将 SL 引入 RL 的工作（如 target network、RUDDER）只是用 SL 来辅助 RL， 而非替代它。UDRL 则更进一步：将过去的轨迹以「事后解释」（hindsight）的方式标注， 视之为智能体成功跟随命令的示范样本，从而完全在 SL 框架内学习。 Schmidhuber（arXiv:1912.02875）给出了理论框架，Srivastava et al.（arXiv:1912.02877） 则提供了可落地的算法与实验验证。Most existing work that brings SL into RL (target networks, RUDDER and the like) only uses SL to assist RL rather than to replace it. UDRL goes further: past trajectories are relabeled in hindsight and treated as demonstrations of an agent successfully following commands, so that learning happens entirely inside the SL framework. Schmidhuber (arXiv:1912.02875) supplies the theoretical framework, while Srivastava et al. (arXiv:1912.02877) contribute a practical algorithm and experimental validation.

02 方法Method

UDRL 的核心是行为函数（behavior function）B：在给定当前状态 s 和命令 c = (d^r, d^h) 的情况下，输出动作概率分布。 这里 d^r 是期望回报（desired return），d^h 是期望步长（desired horizon）。 整个学习过程无需预测值函数，无需策略梯度，用 SL 在历史轨迹上直接优化 B。The core of UDRL is the behavior function B: given the current state s and a command c = (d^r, d^h), it outputs a probability distribution over actions. Here d^r is the desired return and d^h is the desired horizon. The whole learning process needs no value-function prediction and no policy gradient; B is optimized directly by SL on past trajectories.

行为函数（Behavior Function）Behavior Function

形式上，给定状态 s、期望回报 d^r、期望步长 d^h，行为函数定义为：Formally, given a state s, a desired return d^r and a desired horizon d^h, the behavior function is defined as:

B(a, s, d^r, d^h) = P(A = a | R_d^h = d^r, S = s; π)

即：在状态 s 下，若希望未来 d^h 步内获得总回报 d^r，应选哪个动作？ 训练时，从 replay buffer 中随机采样片段 [t₁, t₂]， 以 (s_t1, d^r=∑r, d^h=t₂−t₁) 为输入， a_t1 为目标，用交叉熵损失做监督训练——这正是「事后标注」的精髓。That is: in state s, if one wants a total return of d^r over the next d^h steps, which action should be taken? At training time a segment [t₁, t₂] is sampled at random from the replay buffer; (s_t1, d^r=∑r, d^h=t₂−t₁) is the input and a_t1 the target, trained with a cross-entropy loss — this is the essence of hindsight relabeling.

实践中，神经网络直接拼接命令与状态容易导致学习不稳定； 论文发现采用fast weights（gating 或 bilinear 形式）在第一层融合命令输入， 可显著提升可靠性——这使得网络在任何状态下都必须依赖命令输入来决定动作。In practice, simply concatenating command and state in the network easily makes learning unstable; the paper finds that fusing the command input in the first layer with fast weights (a gating or bilinear form) markedly improves reliability — this forces the network to rely on the command input to decide the action in any state.

高层算法流程High-Level Algorithm

• 初始化：用随机动作填充 replay buffer（warm-up episodes）Initialize: fill the replay buffer with random actions (warm-up episodes)
• 训练 B：对 replay buffer 中最佳 Z 条轨迹做 SL，最小化动作预测损失Train B: run SL on the best Z trajectories in the replay buffer, minimizing the action-prediction loss
• 生成探索命令：从 replay buffer 中最高回报轨迹计算均值 M 和标准差 S， 将 d^r₀ 从 U[M, M+S] 中采样，d^h₀ 取最佳轨迹长度均值Generate exploratory commands: compute the mean M and the standard deviation S from the highest-return trajectories in the replay buffer, sample d^r₀ from U[M, M+S], and set d^h₀ to the mean length of the best trajectories
• 收集经验：用 B 按命令生成新轨迹，更新命令 c_t = (d^r_t − r_t−1, d^h_t − 1)，加入 bufferCollect experience: use B to generate new trajectories from the commands, update the command c_t = (d^r_t − r_t−1, d^h_t − 1), and add them to the buffer
• 评估：将 d^r₀ 设为最近探索命令的下界 M，执行贪心评估Evaluate: set d^r₀ to the lower bound M of the recent exploratory commands and run a greedy evaluation

架构细节Architecture Details

所有实验使用全连接前馈网络（TakeCover-v0 使用卷积网络）。 离散动作环境（LunarLander-v2、TakeCover-v0）与基线 DQN、A2C 对比； 连续动作环境（Swimmer-v2、InvertedDoublePendulum-v2）与 TRPO、PPO、DDPG 对比。 所有智能体均训练 10M 步（LunarLander/TakeCover）或 5M 步（MuJoCo）， 分别使用独立的训练和评估随机种子。All experiments use fully connected feed-forward networks (TakeCover-v0 uses a convolutional network). The discrete-action environments (LunarLander-v2, TakeCover-v0) are compared against the DQN and A2C baselines; the continuous-action environments (Swimmer-v2, InvertedDoublePendulum-v2) against TRPO, PPO and DDPG. All agents are trained for 10M steps (LunarLander/TakeCover) or 5M steps (MuJoCo), using separate random seeds for training and for evaluation.

03 实验Experiments

四个经典基准任务，涵盖离散和连续动作空间、低维和高维（视觉）观测。 每个任务均进行超参数搜索，并以 20 次独立实验 + 1000 bootstrap 样本的 95% 置信区间进行汇报。Four classic benchmark tasks, covering discrete and continuous action spaces and both low-dimensional and high-dimensional (visual) observations. Every task goes through a hyperparameter search, and results are reported as 95% confidence intervals over 20 independent runs with 1000 bootstrap samples.

稀疏延迟奖励实验（Sparse Delayed Reward）Sparse Delayed Reward Experiments

将所有环境奖励延迟至 episode 最后一步才发放（中间步骤奖励为 0）， 测试各算法对极端信用分配（credit assignment）的鲁棒性。All environment rewards are delayed until the final step of the episode (intermediate steps receive reward 0), testing how robust each algorithm is to extreme credit assignment.

结果：DQN（包括 LSTM 版本）、A2C（50/100-step returns）等基线均不稳定、极慢或完全失败。 唯一表现尚可的基线是 A2C with 20-step returns（但对超参数非常敏感）。 UDRL 则「未作任何修改，保留了大部分密集奖励下的性能」——因为它天然地跨长时间步分配信用， 无需时序差分（temporal difference）。Result: the baselines — DQN (including the LSTM variant) and A2C (50/100-step returns) — are unstable, extremely slow, or fail outright. The only baseline that does reasonably well is A2C with 20-step returns (but it is very sensitive to hyperparameters). UDRL, in contrast, retained most of its dense-reward performance without any modification, because it naturally assigns credit across long time spans and needs no temporal difference.

命令跟随能力（Different Returns with a Single Agent）Command-Following Ability (Different Returns with a Single Agent)

训练结束后，对同一 LunarLander-v2 智能体设置不同的 d^r₀（期望回报）， 统计实际获得的平均 episode 回报（100 episodes）。 结果显示 desired vs. obtained return 相关系数 R ≈ 0.98， 说明智能体确实记住了如何按命令执行，而非退化为单一最大回报策略。After training, the same LunarLander-v2 agent is given different values of d^r₀ (the desired return) and the average episode return actually obtained is measured over 100 episodes. The result is a correlation coefficient of R ≈ 0.98 between desired and obtained return, showing that the agent really did learn how to act on command instead of degenerating into a single maximum-return policy.

04 局限性Limitations