ThinkAct

01 动机Motivation

现有 VLA 模型通常以端到端方式将多模态输入直接映射到低层动作，缺乏显式推理过程，导致在多步规划和复杂任务变体上表现不足。尽管思维链（Chain-of-Thought, CoT）方法有所改进，但依赖昂贵的人工标注；基于问答（QA）风格的强化学习奖励则难以支撑具身规划的视觉基础。Existing VLA models typically map multimodal inputs directly to low-level actions in an end-to-end fashion, without an explicit reasoning process, which leaves them underperforming on multi-step planning and complex task variations. Chain-of-Thought (CoT) methods bring improvements but rely on expensive human annotation, while QA-style reinforcement learning rewards struggle to supply the visual grounding that embodied planning requires.

"Existing approaches typically train VLA models in an end-to-end fashion, directly mapping inputs to actions without explicit reasoning, which hinders their ability to plan over multiple steps or adapt to complex task variations."

02 方法Method

ThinkAct 包含两个协同工作的子系统：推理 MLLM（Qwen2.5-VL 7B）负责在视觉观测与语言指令输入下生成具身推理计划，并通过动作对齐的视觉奖励进行强化训练；DiT-based Action Model（432M 参数）以视觉计划潜变量为条件，在目标环境中预测可执行动作序列。ThinkAct consists of two cooperating subsystems: the reasoning MLLM (Qwen2.5-VL 7B) generates embodied reasoning plans from visual observations and language instructions and is reinforced with action-aligned visual rewards; the DiT-based Action Model (432M parameters) is conditioned on the visual plan latent and predicts executable action sequences in the target environment.

动作对齐的视觉奖励（Action-Aligned Visual Rewards）Action-Aligned Visual Rewards

ThinkAct 设计了两类视觉奖励取代传统 QA-style 奖励：ThinkAct designs two kinds of visual rewards to replace conventional QA-style rewards:

• Goal Reward (r_goal)：将 MLLM 预测的轨迹起止位置与检测器提取的真实轨迹点进行基于距离的相似度对比，衡量目标完成程度。Goal Reward (r_goal): compares the start and end positions of the MLLM-predicted trajectory against ground-truth trajectory points extracted by a detector through a distance-based similarity, measuring the degree of goal completion.
• Trajectory Reward (r_traj)：使用动态时间规整距离（Dynamic Time Warping, DTW）正则化预测轨迹分布，提升轨迹一致性。Trajectory Reward (r_traj): regularizes the distribution of predicted trajectories with the Dynamic Time Warping (DTW) distance, improving trajectory consistency.

综合奖励公式为：r = 0.9·rvisual + 0.1·rformat，其中 rvisual = 0.5·rgoal + 0.5·rtraj。The overall reward is given by r = 0.9·rvisual + 0.1·rformat, where rvisual = 0.5·rgoal + 0.5·rtraj.

训练策略（Training Strategy）Training Strategy

训练分两阶段进行：Training proceeds in two stages:

• Cold-start SFT：在 20K 轮次（batch size 32）的监督微调中初始化轨迹解读能力，使模型具备基础的视觉计划生成能力。Cold-start SFT: supervised fine-tuning over 20K steps (batch size 32) initializes the ability to interpret trajectories, giving the model a basic capacity for visual plan generation.
• GRPO Fine-tuning：在 6K 轮次（batch size 64，rollout 5）的强化训练中，利用动作对齐视觉奖励优化推理质量。动作模型则在 OXE 数据上以模仿学习方式训练，推理模块在此阶段保持冻结。GRPO Fine-tuning: reinforcement training over 6K steps (batch size 64, rollout 5) optimizes reasoning quality with the action-aligned visual rewards. The action model is trained by imitation learning on OXE data, while the reasoning module stays frozen at this stage.

03 实验Experiments

实验涵盖机器人操作基准（SimplerEnv、LIBERO）和具身推理基准（EgoPlan-Bench2、RoboVQA、OpenEQA），与 DiT-Policy、CoT-VLA、Magma 等基线对比，并考察少样本自适应（10-shot）和自我纠错能力。The experiments cover robot manipulation benchmarks (SimplerEnv, LIBERO) and embodied reasoning benchmarks (EgoPlan-Bench2, RoboVQA, OpenEQA), compare against baselines such as DiT-Policy, CoT-VLA and Magma, and examine few-shot adaptation (10-shot) and self-correction ability.

机器人操作（SimplerEnv）Robot Manipulation (SimplerEnv)

LIBERO 综合成功率LIBERO Overall Success Rate

具身推理基准Embodied Reasoning Benchmarks

少样本自适应（10-shot）Few-Shot Adaptation (10-shot)

消融实验（Ablations，SimplerEnv）Ablations (SimplerEnv)

消融结果表明，目标奖励和轨迹奖励均对最终性能有正向贡献；移除任一奖励均导致性能下降，完全去除视觉奖励后性能下降最为明显（60.1 → 56.9）。The ablations show that both the goal reward and the trajectory reward contribute positively to the final performance; removing either one degrades results, and removing the visual rewards altogether causes the largest drop (60.1 → 56.9).

04 局限性Limitations