传统 RL 依赖时序差分（TD）学习进行长程信用分配（credit assignment），存在"deadly triad"（函数近似 + bootstrapping + 离策略）带来的不稳定性，且需要折扣因子，容易导致短视行为。
本文探索一个根本性的范式转移：能否用 Transformer 的序列建模能力直接替代传统 RL 算法？Conventional RL relies on temporal difference (TD) learning for long-horizon credit assignment, which suffers from the instability of the "deadly triad" (function approximation + bootstrapping + off-policy learning) and needs a discount factor that easily induces myopic behavior. This paper explores a fundamental paradigm shift: can the sequence modeling capability of Transformers directly replace conventional RL algorithms?
"Instead of training a policy through conventional RL algorithms like temporal difference (TD) learning, we will train transformer models on collected experience using a sequence modeling objective. This will allow us to bypass the need for bootstrapping for long term credit assignment — thereby avoiding one of the 'deadly triad' known to destabilize RL."
直观示例：在有向图上寻找最短路径（等价于 RL 问题）。训练集仅为随机游走轨迹及其 return-to-go。
通过条件生成最大 return，Decision Transformer 能在测试时自动规划出最优路径——无需任何专家演示或动态规划。
（来源：Figure 1，论文原图）An intuitive example: finding the shortest path on a directed graph (equivalent to an RL problem). The training set consists only of random-walk trajectories and their return-to-go. By conditioning generation on the maximum return, Decision Transformer automatically plans the optimal path at test time — without any expert demonstration or dynamic programming. (Source: Figure 1, original figure of the paper)
267.5DT on Breakout （专家归一化分，1% DQN replay）DT on Breakout (expert-normalized score, 1% DQN replay)
74.7D4RL 平均得分 （不含 Reacher，超越 CQL 63.9）D4RL average score (excluding Reacher, above CQL at 63.9)
94.6%Key-to-Door 成功率 （10K 随机轨迹训练）Key-to-Door success rate (trained on 10K random trajectories)
Decision Transformer 在思想上与 Upside-Down RL 一脉相承：两者都通过条件期望回报来驱动动作生成，而非显式优化值函数。
核心区别在于 DT 使用 GPT 风格的因果 Transformer，将 return-to-go、状态、动作显式组织为长序列 token，借助 self-attention 直接完成长程信用分配；
Upside-Down RL 则通常用更简单的前馈网络将"命令"（目标回报）与状态拼接后预测动作，缺乏长上下文建模能力。Decision Transformer is conceptually of a piece with Upside-Down RL: both drive action generation by conditioning on the expected return rather than explicitly optimizing a value function. The key difference is that DT uses a GPT-style causal Transformer that explicitly organizes return-to-go, state and action into a long token sequence, performing long-horizon credit assignment directly through self-attention; Upside-Down RL, by contrast, usually concatenates a "command" (target return) with the state in a simpler feedforward network to predict the action, and lacks long-context modeling capability.
02Method — Decision Transformer 架构Method — The Decision Transformer Architecture
Decision Transformer 的核心思想极为简洁：将轨迹表示为 (R̂₁, s₁, a₁, R̂₂, s₂, a₂, …, R̂ₜ, sₜ, aₜ) 的 token 序列，
用 GPT（因果 Transformer）自回归地预测动作——其中 R̂ₜ 为 return-to-go（当前时刻到终止的累积奖励之和），
既作为"目标"约束策略行为，也替代了传统 RL 中对折扣回报的 bootstrapping。The core idea of Decision Transformer is remarkably simple: represent a trajectory as the token sequence (R̂₁, s₁, a₁, R̂₂, s₂, a₂, …, R̂ₜ, sₜ, aₜ) and let GPT (a causal Transformer) autoregressively predict actions, where R̂ₜ is the return-to-go (the sum of rewards from the current step to termination); it both acts as a "goal" that constrains policy behavior and replaces the bootstrapping of discounted returns used in conventional RL.
Decision Transformer 架构（Figure 2）：每个时间步产生三个 token（return-to-go、state、action），
分别通过可学习的线性嵌入层映射到 embedding 空间，并叠加时间步级别的位置编码（注意：一个时间步对应三个 token，
而非标准 Transformer 的逐 token 位置编码）。随后由 GPT（因果掩码 Transformer）处理，
从 state token 对应的 hidden state 预测该时步的动作。Decision Transformer architecture (Figure 2): every timestep produces three tokens (return-to-go, state, action), which are mapped into the embedding space by separate learnable linear embedding layers and combined with a timestep-level positional encoding (note: one timestep corresponds to three tokens, rather than the per-token positional encoding of a standard Transformer). They are then processed by GPT (a causally masked Transformer), and the action of that timestep is predicted from the hidden state of the state token.
将原始奖励替换为 return-to-go：R̂ₜ = Σ_{t'=t}^{T} rₜ'。
这样模型学习的是"若未来总回报为 R̂，当前应采取什么动作"，而非被动拟合过去奖励。
每次执行动作后，将目标 return 减去实际获得的奖励，动态更新下一步的 R̂ 条件。Replace the raw reward with the return-to-go: R̂ₜ = Σ_{t'=t}^{T} rₜ'. The model thus learns "what action to take now if the total future return is R̂", instead of passively fitting past rewards. After each executed action, the target return is reduced by the reward actually obtained, dynamically updating the R̂ condition of the next step.
架构细节（Architecture）Architecture Details
每步输入最近 K 个时间步，共 3K 个 token：Each step takes the most recent K timesteps as input, for a total of 3K tokens:
每种模态（return-to-go、state、action）各用独立的线性层投影到 embedding 维度，后接 layer normalization。Each modality (return-to-go, state, action) is projected to the embedding dimension by its own linear layer, followed by layer normalization.
视觉任务（如 Atari）中，state 经卷积编码器处理后再线性投影。In visual tasks (such as Atari), the state is first processed by a convolutional encoder and then linearly projected.
位置编码为时间步级别（per-timestep）的可学习嵌入，一个时间步的三个 token 共享同一位置嵌入。The positional encoding is a per-timestep learnable embedding: the three tokens of one timestep share the same positional embedding.
backbone 为标准 GPT（因果自注意力掩码），输出 action token 的 hidden state，经线性层预测动作——连续动作用 MSE loss，离散动作用 cross-entropy loss。The backbone is a standard GPT (causal self-attention mask); the hidden state of the action token is passed through a linear layer to predict the action — MSE loss for continuous actions, cross-entropy loss for discrete actions.
训练与推理（Training & Inference）Training & Inference
训练阶段：在离线轨迹数据集上随机采样长度为 K 的片段，仅优化 action 预测损失（论文发现同时预测 state 或 return-to-go 并不提升性能）。
推理阶段：以目标 return（如专家级别）作为初始条件，通过自回归采样生成动作序列；每执行一步后，将 R̂ 减去实际奖励，循环直至终止。Training: randomly sample length-K segments from the offline trajectory dataset and optimize only the action prediction loss (the paper finds that additionally predicting state or return-to-go does not improve performance). Inference: take a target return (e.g. expert level) as the initial condition and generate the action sequence by autoregressive sampling; after each executed step, subtract the actual reward from R̂ and repeat until termination.
方法示意（来源：论文图示）：return-to-go、state、action 三种模态的 token 交错排列，
因果注意力确保每个 action token 只能看到其之前的 token，从而实现自回归生成。Method illustration (source: figure of the paper): the tokens of the three modalities — return-to-go, state and action — are interleaved, and causal attention ensures that each action token can only see the tokens preceding it, thereby realizing autoregressive generation.
03 Experiments
在三大离线 RL 基准上评估：Atari（高维视觉输入，需长程信用分配）、D4RL/OpenAI Gym（连续控制，MuJoCo 仿真）、Key-to-Door（稀疏奖励，极端长程信用分配）。
主要对比方法：Conservative Q-Learning（CQL，TD 学习的 SOTA）、行为克隆（BC）、BEAR、BRAC、AWR。Evaluated on three major offline RL benchmarks: Atari (high-dimensional visual input, requiring long-horizon credit assignment), D4RL/OpenAI Gym (continuous control, MuJoCo simulation) and Key-to-Door (sparse reward, extreme long-horizon credit assignment). Main baselines: Conservative Q-Learning (CQL, the SOTA of TD learning), behavior cloning (BC), BEAR, BRAC and AWR.
Figure 3（归一化得分汇总）：跨 Atari、OpenAI Gym 和 Minigrid 的平均归一化 episode return。
"On a diverse set of tasks, Decision Transformer performs comparably or better than traditional approaches."Figure 3 (summary of normalized scores): average normalized episode return across Atari, OpenAI Gym and Minigrid. "On a diverse set of tasks, Decision Transformer performs comparably or better than traditional approaches."
使用 DQN-replay 数据集的 1%（约 50 万条轨迹）训练，以专业玩家为 100 分进行归一化。
上下文长度 K=30（Pong 用 K=50）。Trained on 1% of the DQN-replay dataset (roughly half a million trajectories), with scores normalized so that a professional gamer scores 100. Context length K=30 (K=50 for Pong).
游戏Game
DT（本文）DT (ours)
CQL
QR-DQN
REM
BC
Breakout
267.5 ± 97.5
211.1
17.1
8.9
138.9 ± 61.7
Qbert
15.4 ± 11.4
104.2
0.0
0.0
17.3 ± 14.7
Pong
106.1 ± 8.1
111.9
18.0
0.5
85.2 ± 20.0
Seaquest
2.5 ± 0.4
1.7
0.4
0.7
2.1 ± 0.3
均值 ± 标准差（3 seeds）。论文原数据，Table 1。Mean ± standard deviation (3 seeds). Original data of the paper, Table 1.
评估 HalfCheetah、Hopper、Walker 及 Reacher 在 Medium、Medium-Replay、Medium-Expert 三种数据集上的表现。得分归一化（100 = 专家策略）。Evaluates HalfCheetah, Hopper, Walker and Reacher on the Medium, Medium-Replay and Medium-Expert datasets. Scores are normalized (100 = expert policy).
数据集Dataset
环境Environment
DT（本文）DT (ours)
CQL
BEAR
BRAC-v
BC
Medium-Expert
HalfCheetah
86.8 ± 1.3
62.4
53.4
41.9
59.9
Medium-Expert
Hopper
107.6 ± 1.8
111.0
96.3
0.8
79.6
Medium-Expert
Walker
108.1 ± 0.2
98.7
40.1
81.6
36.6
Medium
HalfCheetah
42.6 ± 0.1
44.4
41.7
46.3
43.1
Medium
Hopper
67.6 ± 1.0
58.0
52.1
31.1
63.9
Medium
Walker
74.0 ± 1.4
79.2
59.1
81.1
77.3
Medium-Replay
HalfCheetah
36.6 ± 0.8
46.2
38.6
47.7
4.3
Medium-Replay
Hopper
82.7 ± 7.0
48.6
33.7
0.6
27.6
Medium-Replay
Walker
66.6 ± 3.0
26.7
19.2
0.9
36.9
Average（不含 Reacher）Average (excluding Reacher)
74.7
63.9
48.2
36.9
46.4
论文原数据，Table 2。"Decision Transformer (DT) outperforms conventional RL algorithms on almost all tasks."Original data of the paper, Table 2. "Decision Transformer (DT) outperforms conventional RL algorithms on almost all tasks."
三阶段网格环境：拾取钥匙（阶段一）→空房间（阶段二）→到达门（阶段三）。只有拾取钥匙后到达门才能获得二值奖励。
训练数据全为随机游走轨迹，评估成功率（3 seeds）。A three-phase grid environment: pick up the key (phase one) → empty room (phase two) → reach the door (phase three). A binary reward is given only if the door is reached after the key has been picked up. The training data consists entirely of random-walk trajectories, and the success rate is evaluated (3 seeds).
数据集Dataset
DT（本文）DT (ours)
CQL
BC
%BC
Random
1K 随机轨迹1K random trajectories
71.8%
13.1%
1.4%
69.9%
3.1%
10K 随机轨迹10K random trajectories
94.6%
13.3%
1.6%
95.1%
3.1%
"Methods using hindsight (Decision Transformer, %BC) can learn successful policies, while TD learning struggles to perform credit assignment."（Table 4）"Methods using hindsight (Decision Transformer, %BC) can learn successful policies, while TD learning struggles to perform credit assignment." (Table 4)
Ablation — 上下文长度（Context Length K）的重要性Ablation — The Importance of Context Length K
对比 K=1（无历史）与标准 K（K=30 或 K=50）。实验表明长上下文对性能至关重要，尤其在 Breakout（267.5 vs 73.9）和 Pong（106.1 vs 2.5）上差异显著。
论文假设：在建模策略分布时，上下文帮助 Transformer 识别轨迹来自哪类策略，从而实现更好的学习与生成。Comparison of K=1 (no history) with the standard K (K=30 or K=50). Experiments show that a long context is critical to performance, with especially pronounced gaps on Breakout (267.5 vs 73.9) and Pong (106.1 vs 2.5). The paper hypothesizes that, when modeling a distribution of policies, the context helps the Transformer identify which kind of policy a trajectory comes from, thereby enabling better learning and generation.
关键时刻的注意力权重（Key-to-Door）：Transformer 将注意力集中在"拾取钥匙"与"到达门"等关键事件附近，
验证了模型能够形成状态-奖励关联（state-reward association），实现精准的价值预测与长程归因。
（来源：论文 Figure 7 右图，attention weights 叠加可视化）Attention weights at critical moments (Key-to-Door): the Transformer concentrates its attention around key events such as "picking up the key" and "reaching the door", confirming that the model can form a state-reward association and achieve accurate value prediction and long-horizon attribution. (Source: right panel of Figure 7 of the paper, overlaid visualization of attention weights)
稀疏/延迟奖励下的鲁棒性Robustness under Sparse / Delayed Rewards
在 D4RL Hopper 的延迟奖励设置下（所有中间奖励为 0，仅终止时给出累积奖励），CQL 性能崩溃（Medium-Expert: 111.0 → 9.0），
而 Decision Transformer 几乎不受影响（107.6 → 107.3 ± 3.5）。这证明 DT 对奖励稀疏性天然具有鲁棒性。In the delayed-reward setting of D4RL Hopper (all intermediate rewards are 0 and the cumulative reward is given only at termination), CQL collapses (Medium-Expert: 111.0 → 9.0), whereas Decision Transformer is almost unaffected (107.6 → 107.3 ± 3.5). This shows that DT is inherently robust to reward sparsity.
04 Limitations
Note：以下局限性部分来自论文"Conclusion"节中作者的明确陈述（标注为 stated），部分为从设计中推断（标注为 inferred）。Note: Some of the limitations below are explicit statements by the authors in the "Conclusion" section of the paper (marked stated), while others are inferred from the design (marked inferred).
需要准确的目标 return 估计（stated）Requires an accurate target return estimate (stated)
测试时需要指定合理的目标 return-to-go 初始值。若目标设置过高（超出数据分布）或不合理，
模型可能生成低质量动作。论文提到"conditioning on return distributions to model stochastic settings
instead of deterministic returns"是未来值得研究的方向。A reasonable initial value of the target return-to-go must be specified at test time. If the target is set too high (outside the data distribution) or is unreasonable, the model may generate low-quality actions. The paper mentions "conditioning on return distributions to model stochastic settings instead of deterministic returns" as a direction worth studying in the future.
局限于离线 RL，在线探索未研究（stated）Limited to offline RL; online exploration not studied (stated)
本文仅研究了离线 RL 场景，未扩展至在线 RL。论文指出"Decision Transformer can meaningfully improve
online RL methods by serving as a strong model for behavior generation"，但这仅为展望而非实验验证。This paper studies only the offline RL setting and does not extend to online RL. It points out that "Decision Transformer can meaningfully improve online RL methods by serving as a strong model for behavior generation", but this is only an outlook rather than an experimental validation.
Decision Transformer 本质是监督学习：它能复现数据集中高回报轨迹对应的行为，但无法通过优化学到的价值函数
来发现超出数据集的更优策略（Qbert 上明显落后于 CQL，104.2 vs 15.4，印证了这一点）。
TD 方法在数据质量高、状态覆盖好的任务上仍有优势。Decision Transformer is supervised learning in essence: it can reproduce the behavior of the high-return trajectories in the dataset, but it cannot discover better policies beyond the dataset by optimizing a learned value function (its clear gap behind CQL on Qbert, 104.2 vs 15.4, confirms this). TD methods still hold an advantage on tasks with high-quality data and good state coverage.
计算复杂度随上下文长度增长（inferred）Computational complexity grows with context length (inferred)
Transformer 的 self-attention 复杂度为 O(K²)（K = 上下文时间步数）。在需要超长上下文（如 Key-to-Door 使用整集长度作为上下文）的任务中，
计算和内存开销显著增大，限制了在超长时程任务上的扩展能力。The self-attention of a Transformer has O(K²) complexity (K = number of context timesteps). In tasks that require a very long context (e.g. Key-to-Door, which uses the full episode length as context), compute and memory cost grow markedly, limiting scalability to very long-horizon tasks.
数据偏差与对抗性风险（stated）Data bias and adversarial risk (stated)
论文明确提出："reward design by nefarious actors can potentially generate unintended behaviors as our model
generates behaviors by conditioning on desired returns."
训练数据的来源与质量直接决定生成行为的安全性，存在被恶意设计的数据或奖励函数所利用的风险。The paper explicitly states: "reward design by nefarious actors can potentially generate unintended behaviors as our model generates behaviors by conditioning on desired returns." The provenance and quality of the training data directly determine the safety of the generated behavior, and there is a risk of exploitation via maliciously designed data or reward functions.
