ReWiND：语言引导奖励无需新示范即可训练机器人策略ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations
Language-Guided Rewards Teach Robot Policies without New Demonstrations
Jiahui Zhang, Yusen Luo, Abrar Anwar, Sumedh A. Sontakke, Joseph J. Lim, Jesse Thomason, Erdem Bıyık, Jesse Zhang · University of Southern California & Amazon Robotics & KAIST
ReWiND 是一个框架，让机器人仅凭语言指令就能学会操控任务——无需针对每个新任务收集示范数据。它学习一个高数据效率的语言条件奖励函数，并结合离线与在线强化学习，在未见过的任务上实现泛化。ReWiND is a framework that lets robots learn manipulation tasks from language instructions alone—without collecting demonstration data for every new task. It learns a highly data-efficient language-conditioned reward function and, by combining offline and online reinforcement learning, generalizes to unseen tasks.
仿真成功率提升 2×2× higher success rate in simulation真实机器人提升 5×5× improvement on a real robot奖励泛化提升 2.4×2.4× better reward generalizationarXiv 论文arXiv paper
机器人学习长期面临两难困境：要么需要为每个任务人工设计奖励函数，要么需要大量专家示范数据。现有语言条件奖励学习方法往往依赖真值状态信息或数千条示范，难以推广到真实场景。Robot learning has long faced a dilemma: either a reward function must be hand-designed for every task, or large amounts of expert demonstration data are needed. Existing language-conditioned reward learning methods typically rely on ground-truth state information or thousands of demonstrations, which makes them hard to carry over to real-world settings.
"我们的框架仅需少量示范（例如每个任务五条），便能让机器人通过语言指令学习未见过的任务变体，无需额外收集新示范。" —— 论文核心主张"Our framework needs only a handful of demonstrations (five per task, for example) to let a robot learn unseen task variants from language instructions, without collecting any new demonstrations." — the core claim of the paper
图1：ReWiND 整体概览。系统在少量语言标注示范上预训练，随后仅凭语言指令驱动的奖励函数，即可通过强化学习适应未见过的新任务——无需为新任务收集任何新示范数据。Figure 1: Overview of ReWiND. The system is pretrained on a small set of language-annotated demonstrations; afterwards, driven only by a language-instructed reward function, it adapts to unseen new tasks through reinforcement learning—without collecting any new demonstration data for the new task.
2.4×奖励泛化提升（相对基线）Reward generalization gain (over the baseline)
68%真实机器人微调后成功率Real-robot success rate after fine-tuning
5×真实世界相对预训练策略提升Real-world gain over the pretrained policy
现有方法的不足Shortcomings of existing approaches
标准强化学习（RL）：每个新任务都需要人工设计奖励函数，工程成本高昂。Standard reinforcement learning (RL): every new task needs a hand-designed reward function, at high engineering cost.
模仿学习（Imitation Learning）：依赖大量专家示范，采集成本高，且难以泛化到未见变体。Imitation learning: it depends on large numbers of expert demonstrations, which are costly to collect, and generalizes poorly to unseen variants.
现有语言条件奖励方法（LIV、RoboCLIP、VLC、GVL 等）：通常需要真值状态信息，或对数千条示范的需求，在真实机器人部署中不切实际。Existing language-conditioned reward methods (LIV, RoboCLIP, VLC, GVL, etc.): they usually require ground-truth state information, or thousands of demonstrations, which is impractical for real-robot deployment.
02方法Method
ReWiND 包含三个阶段：（1）从少量示范中学习语言条件奖励函数；（2）用该奖励函数对语言条件策略进行离线 RL 预训练；（3）对未见新任务使用在线 RL 进行微调，全程无需额外示范。ReWiND consists of three stages: (1) learning a language-conditioned reward function from a small set of demonstrations; (2) using that reward function to pretrain a language-conditioned policy with offline RL; (3) fine-tuning on unseen new tasks with online RL, without extra demonstrations at any point.
图2：ReWiND 三阶段流程。（a）奖励模型在增广的示范数据集上训练，学习预测帧级别任务进度；（b）学到的奖励函数为示范数据打标签，用于 IQL 离线预训练语言条件策略；（c）对于语言指定的未见新任务，策略通过在线 RL 微调，奖励信号完全来自已学习的奖励函数。Figure 2: The three-stage ReWiND pipeline. (a) The reward model is trained on the augmented demonstration dataset, learning to predict frame-level task progress; (b) the learned reward function labels the demonstration data, which is used for offline IQL pretraining of the language-conditioned policy; (c) for an unseen new task specified in language, the policy is fine-tuned with online RL, and the reward signal comes entirely from the learned reward function.
奖励函数 Rψ(o1:t, z) 以观测序列和语言指令为输入，预测每帧对应的任务进度（范围 0→1）。对于匹配的视频-指令对，模型学习预测单调递增的进度值；对于不匹配对，则预测为零进度。这种"stable, fixed targets"设计将进度直接转化为归一化奖励，避免了奖励尺度不稳定的问题。The reward function Rψ(o1:t, z) takes an observation sequence and a language instruction as input and predicts the task progress of each frame (ranging from 0 to 1). For matching video-instruction pairs the model learns to predict monotonically increasing progress values; for mismatched pairs it predicts zero progress. This "stable, fixed targets" design turns progress directly into a normalized reward and avoids the problem of unstable reward scales.
架构关键设计：Key architectural choices:
图像编码器：冻结的 DINOv2；语言编码器：冻结的 all-MiniLM-L12-v2Image encoder: a frozen DINOv2; language encoder: a frozen all-MiniLM-L12-v2
仅首帧位置编码：避免模型过拟合于帧位置而非语义内容Positional encoding on the first frame only: prevents the model from overfitting to frame position rather than semantic content
阶段二：视频回绕增强（Video Rewind Augmentation）Stage 2: Video Rewind Augmentation
这是 ReWiND 最核心的创新之一。在线 RL 期间，策略不可避免地会产生各种失败轨迹，但示范数据仅含成功序列，导致奖励函数无法正确评估失败行为。This is one of ReWiND's most central innovations. During online RL the policy inevitably produces all kinds of failure trajectories, yet the demonstration data contains only successful sequences, so the reward function cannot correctly evaluate failure behavior.
图3：视频回绕增强（Video Rewind Augmentation）示意。对成功示范视频，在随机中间时刻将后续帧"倒放"若干步，合成出"尝试抓取后掉落"等失败轨迹。论文原文举例："If we rewind the video for a few frames right when the robot grabs the cup, it now looks like one in which the robot attempted to grasp the cup and then dropped it." 这使奖励模型学会在失败发生时降低奖励分值。Figure 3: Illustration of Video Rewind Augmentation. For a successful demonstration video, the following frames are "played backwards" for a few steps at a random intermediate moment, synthesizing failure trajectories such as "attempted a grasp and then dropped it". The paper gives the example: "If we rewind the video for a few frames right when the robot grabs the cup, it now looks like one in which the robot attempted to grasp the cup and then dropped it." This teaches the reward model to lower the reward score when a failure occurs.
为提升语言泛化能力，ReWiND 还整合了来自 Open-X Embodiment 数据集的 356k 条轨迹（含 59k 个唯一任务字符串），并通过 LLM 为每个任务生成 5–10 条多样化语言描述。训练目标结合了进度损失和回绕损失，使奖励函数对指令措辞的变化保持鲁棒。To improve language generalization, ReWiND also incorporates 356k trajectories from the Open-X Embodiment dataset (containing 59k unique task strings), and uses an LLM to generate 5–10 diverse language descriptions for each task. The training objective combines a progress loss and a rewind loss, making the reward function robust to variation in instruction phrasing.
训练数据组成Composition of the training data
Ddemos：目标环境 15–20 个任务，每任务约 5 条示范Ddemos: 15–20 tasks in the target environment, about 5 demonstrations per task
Dopen-x：精选 Open-X 子集，包含以物体/方向为中心的指令Dopen-x: a curated Open-X subset containing object- and direction-centric instructions
增强数据：视频回绕合成的失败轨迹Augmented data: failure trajectories synthesized by video rewinding
实验分三部分：（Q1）奖励函数质量评估；（Q2）策略学习性能；（Q3）消融研究。基线包括 LIV、RoboCLIP、VLC、GVL 等主流方法。The experiments have three parts: (Q1) evaluating reward function quality; (Q2) policy learning performance; (Q3) ablation studies. Baselines include mainstream methods such as LIV, RoboCLIP, VLC and GVL.
Q1：奖励函数质量Q1: Reward function quality
指标Metric
VLC（基线）VLC (baseline)
LIV-FT（基线）LIV-FT (baseline)
ReWiND（本文）ReWiND (ours)
任务进度 Pearson 相关系数 rTask-progress Pearson correlation r
0.74 (方差 0.04)，提升 23%0.74 (variance 0.04), a 23% gain
Q2：策略学习性能Q2: Policy learning performance
图5：MetaWorld 仿真最终成功率对比（100k 步，3 个随机种子，IQM）。ReWiND 以 79% 的 IQM 成功率领先所有基线：VLC 40%、LIV-FT 45%、Sparse Reward 接近 0%。ReWiND 相对 VLC 提升约 97.5%。Figure 5: Final success rates in MetaWorld simulation (100k steps, 3 random seeds, IQM). With a 79% IQM success rate, ReWiND leads all baselines: VLC 40%, LIV-FT 45%, Sparse Reward close to 0%. ReWiND improves over VLC by about 97.5%.图6：真实机器人（Koch 双臂）在线 RL 结果。五个任务覆盖分布内任务、视觉泛化、空间/动作序列和语言泛化。预训练策略平均成功率仅 12%，ReWiND 微调后达到 68%（提升 5.6×）；VLC 微调仅得 10%（ReWiND 对 VLC 提升 6.7×）。Figure 6: Online RL results on a real robot (Koch bimanual). Five tasks cover in-distribution tasks, visual generalization, spatial/action sequencing and language generalization. The pretrained policy reaches an average success rate of only 12%, while ReWiND reaches 68% after fine-tuning (a 5.6× gain); VLC fine-tuning reaches only 10% (ReWiND is 6.7× better than VLC).
分布内任务："separate the blue and orange cups"（分离蓝橙杯子）In-distribution task: "separate the blue and orange cups"
难分布内任务："fold the blue towel"（折叠蓝色毛巾）Hard in-distribution task: "fold the blue towel"
视觉泛化："open the red trash bin"（打开红色垃圾桶）Visual generalization: "open the red trash bin"
空间/动作序列："put the orange cup on the red plate"（将橙杯放到红盘上）Spatial / action sequencing: "put the orange cup on the red plate"
语言泛化："put the fruit-colored object in the box"（将水果色物体放入盒子）Language generalization: "put the fruit-colored object in the box"
Q3：消融研究Q3: Ablation studies
图7：BRIDGE 数据集毛巾展开任务的 ReWiND 奖励预测示例。当机器人未能成功抓取时，奖励停滞不前；成功抓取并展开后，奖励稳步上升，体现了奖励函数对接触密集任务的准确建模能力。Figure 7: Example ReWiND reward predictions on the towel-unfolding task of the BRIDGE dataset. When the robot fails to grasp, the reward stays flat; once it grasps and unfolds successfully, the reward rises steadily, showing that the reward function models contact-rich tasks accurately.
消融组件Ablated component
影响Effect
移除 Video Rewind 增强Remove the Video Rewind augmentation
策略成功率下降 33%；轨迹排序 ρ 从 0.82 降至 0.56Policy success rate drops by 33%; trajectory-ranking ρ falls from 0.82 to 0.56
移除 LLM 指令生成Remove LLM instruction generation
语言鲁棒性相关系数从 0.74 降至 0.52Language-robustness correlation falls from 0.74 to 0.52
移除 Open-X 子集Remove the Open-X subset
未见任务对齐从 0.79 降至 0.64；鲁棒性降至 0.55Unseen-task alignment falls from 0.79 to 0.64; robustness drops to 0.55
移除目标环境数据Remove target-environment data
训练对齐从 1.00 降至 0.55；轨迹排序失效Training alignment falls from 1.00 to 0.55; trajectory ranking breaks down
改用完整位置编码Switch to full positional encoding
策略成功率下降 21%（过拟合帧位置）Policy success rate drops by 21% (overfitting to frame position)
04局限性Limitations
说明：论文在正文及附录中明确讨论了以下局限性，均为作者明确陈述（stated）；仅最后一项为从设计推断（inferred）。Note: the paper explicitly discusses the following limitations in the main text and the appendix; all of them are stated by the authors, except the last item, which is inferred from the design.
依赖合理的零样本初始策略能力Depends on a reasonable zero-shot initial policy capability
ReWiND 的在线微调效果取决于预训练策略在新任务上是否具备一定的初始探索能力。论文指出："If the ReWiND reward function could be combined with stronger policies that are easy to learn online in the loop, performance could improve significantly." 当前架构相较于现代 Vision-Language-Action 模型仍较为简单。How well ReWiND's online fine-tuning works depends on whether the pretrained policy has some initial exploration capability on the new task. The paper notes: "If the ReWiND reward function could be combined with stronger policies that are easy to learn online in the loop, performance could improve significantly." The current architecture is still relatively simple compared with modern Vision-Language-Action models.
视频进度标签无法直接利用真实失败轨迹Video progress labels cannot directly exploit real failure trajectories
进度预测目标依赖成功示范的视频标注，无法直接整合带有明确进度标签的失败轨迹。视频回绕增强是一种有效的近似，但合成失败与真实失败之间仍存在分布差异。The progress-prediction objective relies on video annotations of successful demonstrations and cannot directly incorporate failure trajectories carrying explicit progress labels. Video rewind augmentation is an effective approximation, but a distribution gap remains between synthetic and real failures.
冻结编码器可能对特定任务欠拟合Frozen encoders may underfit particular tasks
为防止少量示范导致过拟合，ReWiND 使用冻结的预训练视觉（DINOv2）和语言（all-MiniLM-L12-v2）编码器。但若编码器缺乏对应领域的先验知识，可能出现欠拟合。论文举例：双臂海绵擦洗任务因缺乏双臂操作数据且存在摄像头遮挡，表现较差。To prevent a small number of demonstrations from causing overfitting, ReWiND uses frozen pretrained vision (DINOv2) and language (all-MiniLM-L12-v2) encoders. But if the encoders lack prior knowledge of the relevant domain, underfitting can occur. The paper gives an example: the bimanual sponge-scrubbing task performs poorly because bimanual manipulation data is lacking and the camera is occluded.
在线 RL 需要人工重置与成功检测Online RL requires manual resets and success detection
当前系统要求人工操作员进行环境重置，并在在线 RL 期间监督成功检测，限制了完全自主部署。论文认可这一局限性，并指出"recent reset-free RL works demonstrate promising solutions"作为未来方向。The current system requires a human operator to reset the environment and to supervise success detection during online RL, which limits fully autonomous deployment. The paper acknowledges this limitation and points to "recent reset-free RL works demonstrate promising solutions" as a future direction.
（推断）扩散策略集成尚不成熟(Inferred) Diffusion policy integration is not yet mature
初步的 Diffusion Policy 实验（Diffusion Steering RL, DSRL）显示，ReWiND 可将扩散策略成功率从 0% 提升到 20%，但在潜空间中进行指令条件化仍具挑战性，该方向有待进一步探索。Preliminary Diffusion Policy experiments (Diffusion Steering RL, DSRL) show that ReWiND can raise the success rate of a diffusion policy from 0% to 20%, but conditioning on instructions in the latent space remains challenging, and the direction needs further exploration.
