Towards Universal Humanoid Control with Visual Integration
Pengxiang Ding*, Jianfei Ma*, Xinyang Tong* 等 · Westlake University / Westlake Robotics / Zhejiang UniversityPengxiang Ding*, Jianfei Ma*, Xinyang Tong* et al. · Westlake University / Westlake Robotics / Zhejiang University
Humanoid-VLA 是首个面向人形机器人的 VLA（Vision-Language-Action）框架，将语言理解、第一人称（egocentric）视觉感知与全身运动控制统一于单一系统中。针对现有方法依赖反应式机制、缺乏自主感知能力以及标注数据稀缺等问题，该框架通过语言-动作预对齐与视觉条件微调两阶段训练，并引入自监督数据增强策略，有效利用大规模无标注视频数据，使人形机器人能够在真实场景中完成物体交互与环境探索任务。Humanoid-VLA is the first VLA (Vision-Language-Action) framework targeting humanoid robots, unifying language understanding, egocentric visual perception and whole-body motion control within a single system. Addressing the reliance of existing methods on reactive mechanisms, their lack of autonomous perception, and the scarcity of annotated data, the framework is trained in two stages—language-motion pre-alignment and vision-conditioned fine-tuning—and introduces a self-supervised data augmentation strategy that effectively exploits large-scale unlabeled video data, enabling humanoid robots to accomplish object interaction and environment exploration tasks in real-world scenes.
当前人形机器人控制框架主要依赖反应式机制——跟踪人体演示或响应语言命令，但无法自主感知环境、识别交互目标。这一根本局限使机器人在需要物体操作或复杂环境探索的场景中举步维艰。同时，数据稀缺是另一瓶颈：现有动作捕捉数据集缺乏同步的第一人称视觉信息，遥操作采集成本极高，导致训练数据在数量和多样性上均严重不足。Current humanoid control frameworks rely mainly on reactive mechanisms—tracking human demonstrations or responding to language commands—but cannot autonomously perceive the environment or identify interaction targets. This fundamental limitation makes robots struggle in scenarios that require object manipulation or complex environment exploration. Data scarcity is another bottleneck: existing motion capture datasets lack synchronized egocentric visual information, and teleoperated collection is prohibitively expensive, leaving training data severely insufficient in both quantity and diversity.
"Current data acquisition methods, focusing mainly on human joint poses, lack integration with egocentric vision. Thus, they can only teach robots what actions are performed, not the underlying intent or context."
图1：先前方法 vs. Humanoid-VLA。
先前方法通过模仿人体演示执行运动，依赖反应式机制。Humanoid-VLA 具备自主感知能力，可主动识别交互目标并执行物体交互任务，显著超越了基于演示模仿的方法。Figure 1: Prior methods vs. Humanoid-VLA. Prior methods execute motions by imitating human demonstrations and rely on reactive mechanisms. Humanoid-VLA possesses autonomous perception: it can actively identify interaction targets and perform object interaction tasks, substantially surpassing demonstration-imitation methods.
0.929M训练片段总数（含合成数据）Total training clips (including synthetic data)
Humanoid-VLA 由三个主要模块构成：语言-动作预对齐（Language-Motion Pre-Alignment）、视觉条件微调（Vision-Conditioned Fine-Tuning），以及集成已有工作的全身控制器（Whole-Body Controller）。整体以 Llama3-70B 为基座 LLM，将运动码本与语言词汇表统一为共享词表，实现运动 token 与文本 token 的无缝融合。Humanoid-VLA consists of three main modules: Language-Motion Pre-Alignment, Vision-Conditioned Fine-Tuning, and a Whole-Body Controller integrated from prior work. It builds on Llama3-70B as the backbone LLM and unifies the motion codebook with the language vocabulary into a shared vocabulary, so that motion tokens and text tokens fuse seamlessly.
图2：Humanoid-VLA 系统总览。
左侧为语言-动作预对齐阶段：使用非第一人称人体运动数据集与文本描述对进行预训练，学习通用运动模式和动作语义。右侧为视觉条件微调阶段：冻结预训练 transformer 层，引入视觉编码器，通过 cross-attention 将视觉特征 $X_v$ 与语言特征 $X_d$ 融合，实现上下文感知的运动生成。Figure 2: Overview of the Humanoid-VLA system. Left is the language-motion pre-alignment stage: pre-training on pairs of non-egocentric human motion data and textual descriptions to learn general motion patterns and action semantics. Right is the vision-conditioned fine-tuning stage: the pre-trained transformer layers are frozen, a visual encoder is introduced, and visual features $X_v$ are fused with language features $X_d$ through cross-attention to achieve context-aware motion generation.
模型将每帧身体姿态分解为五个部位（左腿、右腿、躯干、左臂、右臂），为每个部位独立训练编码器 $\mathcal{E}_b$ 和码本 $V_b$，将部位数据 $c_t$ 压缩为离散表示 $\hat{z}_t \in \mathbb{R}^5$。优化目标 $\mathcal{L}_{hvq}$ 结合了重建损失、嵌入损失和承诺损失。这种分解式编码的核心优势在于：可在 token 级别对特定身体部位进行替换、扰动或重排，为后续自监督数据增强奠定灵活的操作基础。The model decomposes the body pose of each frame into five parts (left leg, right leg, torso, left arm, right arm) and trains an independent encoder $\mathcal{E}_b$ and codebook $V_b$ for every part, compressing the part data $c_t$ into a discrete representation $\hat{z}_t \in \mathbb{R}^5$. The optimization objective $\mathcal{L}_{hvq}$ combines a reconstruction loss, an embedding loss and a commitment loss. The core advantage of this decomposed encoding is that specific body parts can be replaced, perturbed or reordered at the token level, providing a flexible operational basis for the subsequent self-supervised data augmentation.
自监督数据增强（Self-Supervised Data Augmentation）Self-Supervised Data Augmentation
图3：数据获取流程。
提出的自监督数据增强方案将大量纯运动数据转换为带有标注的运动数据。框架包含两个核心模块：组合式动作量化方法和自动数据增强方法，共同实现数据集的可扩展扩充。四类增强任务（<Track>、<Time>、<Occlusion>、<State>）从原始运动数据中自动生成问答对，无需人工标注。Figure 3: Data acquisition pipeline. The proposed self-supervised data augmentation scheme converts large amounts of motion-only data into annotated motion data. The framework contains two core modules: the compositional motion quantization method and the automatic data augmentation method, which together enable scalable dataset expansion. Four augmentation tasks (<Track>, <Time>, <Occlusion>, <State>) automatically generate question-answer pairs from raw motion data without any manual annotation.
框架设计了四类增强任务：The framework designs four augmentation tasks:
<Track>：提取特定关节（如根关节）的时序轨迹，生成"Please move your center position along the trajectory of <Track>"等指令-答案对。<Track>: extracts the temporal trajectory of a specific joint (e.g. the root joint) and generates instruction-answer pairs such as "Please move your center position along the trajectory of <Track>".
<Occlusion>：临时遮蔽特定身体关节，训练模型重建被遮挡的运动，生成如"missing left arm <Occlusion> motion data. Please complete the motion"的指令。<Occlusion>: temporarily masks specific body joints and trains the model to reconstruct the occluded motion, generating instructions such as "missing left arm <Occlusion> motion data. Please complete the motion".
<Time>：结合时间条件，约束生成运动的时长。<Time>: incorporates a temporal condition to constrain the duration of the generated motion.
<State>：以目标姿态 token $z_t$ 为条件，引导模型生成以特定状态结束的运动序列。<State>: conditions on a target pose token $z_t$ to guide the model toward generating motion sequences that end in a specific state.
训练分两阶段：首先使用从视频提取的大规模低质量数据建立初步语言-动作对齐；随后用小规模高质量 Mocap 数据进行精调，确保动作符合正确的人体运动学规律。Training proceeds in two stages: large-scale low-quality data extracted from videos first establishes a preliminary language-motion alignment; small-scale high-quality Mocap data is then used for refinement, ensuring that the motions obey correct human kinematics.
冻结预对齐阶段的 transformer 层权重，引入视觉编码器。在解码器每一层插入 cross-attention 模块，以语言 token $X_d^l$ 为 query，视觉 token $X_v^l$ 为 key 和 value，融合得到统一表示 $X_u^l$：The transformer layer weights from the pre-alignment stage are frozen and a visual encoder is introduced. A cross-attention module is inserted into every decoder layer, taking the language tokens $X_d^l$ as queries and the visual tokens $X_v^l$ as keys and values, and fusing them into a unified representation $X_u^l$:
仅训练新引入的 cross-attention 参数，实现参数高效的视觉融合，将学到的运动知识迁移到视觉引导的真实场景中。Only the newly introduced cross-attention parameters are trained, achieving parameter-efficient visual fusion and transferring the learned motion knowledge to vision-guided real-world scenes.
全身控制器（Whole-Body Controller）Whole-Body Controller
集成目标条件强化学习策略（RL policy），将 VLA 生成的运动序列映射为人形机器人的关节力矩 $j_t \in \mathbb{R}^{24}$，使用 PPO（Proximal Policy Optimization）在 IsaacGym 物理仿真器中训练，实现端到端的物理可行运动执行。A goal-conditioned reinforcement learning policy (RL policy) is integrated to map the motion sequences generated by the VLA into joint torques $j_t \in \mathbb{R}^{24}$ of the humanoid robot; it is trained with PPO (Proximal Policy Optimization) in the IsaacGym physics simulator, achieving end-to-end physically feasible motion execution.
03实验Experiments
实验从两个维度评估 Humanoid-VLA：（1）运动生成质量（运动学保真度 + 物理合理性）；（2）视觉集成效果（在 Unitree G1 真实机器人上的任务成功率）。基线模型包括 MDM（扩散式）和 T2M-GPT（自回归式），使用 HumanML3D 和自建 Humanoid-S 数据集评估。The experiments evaluate Humanoid-VLA along two dimensions: (1) motion generation quality (kinematic fidelity + physical plausibility); (2) vision integration effectiveness (task success rate on the real Unitree G1 robot). The baselines include MDM (diffusion-based) and T2M-GPT (autoregressive), evaluated on HumanML3D and the self-built Humanoid-S dataset.
运动学保真度（Kinematic Fidelity）Kinematic Fidelity
使用 FID（分布相似度）和 Diversity（生成多样性，200 个随机运动的平均欧氏距离）作为评估指标。FID (distributional similarity) and Diversity (generation diversity, the average Euclidean distance over 200 random motions) are used as evaluation metrics.
方法Method
HumanML3D FID↓
HumanML3D DIV↑
Humanoid-S FID↓
Humanoid-S DIV↑
MDM
0.889±.026
3.855±.053
2.351±.590
4.111±.261
T2M-GPT
0.531±.020
4.555±.058
1.101±.189
4.199±.218
Humanoid-VLA
0.467±.018
4.585±.086
1.037±.147
4.466±.213
在 HumanML3D 上，Humanoid-VLA 的 FID 为 0.467，相比 MDM 提升 47.5%，相比 T2M-GPT 提升 12%；在 Humanoid-S 上 Diversity 达到 4.466，超越 MDM 6%。On HumanML3D, the FID of Humanoid-VLA is 0.467, an improvement of 47.5% over MDM and of 12% over T2M-GPT; on Humanoid-S its Diversity reaches 4.466, surpassing MDM by 6%.
物理合理性（Physical Plausibility）Physical Plausibility
在 IsaacGym 中追踪模型生成的运动学轨迹，评估全局 MPJPE（$E_\text{mpjpe}^g$，mm）、PA-MPJPE（$E_\text{mpjpe}^\text{pa}$，mm）、加速度误差（$E_\text{accel}$，mm/s²）和速度误差（$E_\text{vel}$，mm/s）：The kinematic trajectories generated by the model are tracked in IsaacGym, and global MPJPE ($E_\text{mpjpe}^g$, mm), PA-MPJPE ($E_\text{mpjpe}^\text{pa}$, mm), acceleration error ($E_\text{accel}$, mm/s²) and velocity error ($E_\text{vel}$, mm/s) are evaluated:
条件难度Condition difficulty
输入条件Input condition
$E_\text{mpjpe}^g$↓
$E_\text{mpjpe}^\text{pa}$↓
$E_\text{accel}$↓
$E_\text{vel}$↓
Easy
D（文本描述）D (text description)
36.13
1.53
34.42
18.73
Medium
D + T（描述+时长）D + T (description + duration)
31.07
1.18
27.84
14.76
Hard
D + $S_1$ + $S_N$（描述+始末状态）D + $S_1$ + $S_N$ (description + start and end states)
37.14
1.34
34.69
18.08
RL 策略在中等难度下（D+T 条件）达到最优追踪精度：全局位置误差 $E_\text{mpjpe}^g$ 为 31.07 mm，姿态精度误差 $E_\text{mpjpe}^\text{pa}$ 仅 1.18 mm，体现平滑且物理一致的运动生成能力。The RL policy attains the best tracking accuracy at medium difficulty (the D+T condition): the global position error $E_\text{mpjpe}^g$ is 31.07 mm and the pose accuracy error $E_\text{mpjpe}^\text{pa}$ is only 1.18 mm, reflecting smooth and physically consistent motion generation.
图4：真实环境机器人实验。
Humanoid-VLA 在 Unitree G1 机器人上展示物体交互能力：精确执行踢球任务（kick ball）及绕障碍导航任务（avoid obstacle），验证 VLA 框架利用视觉信息生成适当运动的有效性。Figure 4: Robot experiments in real environments. Humanoid-VLA demonstrates object interaction capability on the Unitree G1 robot: it precisely performs the kick ball task and the avoid obstacle navigation task, verifying the effectiveness of the VLA framework in exploiting visual information to generate appropriate motions.
在 Unitree G1 机器人上评估 4 大类、8 项代表性任务，每项任务重复测试 10 次：On the Unitree G1 robot, 4 major categories covering 8 representative tasks are evaluated, with each task repeated 10 times:
任务Task
成功率（SR）Success rate (SR)
Turn to an object（转向物体）Turn to an object
10/10
Hold an object（抓握物体）Hold an object
9/10
Wave to people（向人挥手）Wave to people
10/10
Avoid an obstacle（绕障碍）Avoid an obstacle
9/10
Jump over an object（跨越障碍）Jump over an object
9/10
Dance with a partner（与人共舞）Dance with a partner
8/10
Punch an obstacle（击打障碍）Punch an obstacle
10/10
Kick a ball（踢球）Kick a ball
9/10
消融实验（Ablation on Data Augmentation）Ablation on Data Augmentation
在数据增强的消融实验中，仅使用低质量视频数据训练时 FID 为 0.698；仅使用高质量 Mocap 数据时 FID 为 0.557；结合两者后 FID 降至 0.467，代表 16% 的改善。
这有力验证了将大规模视频运动数据纳入训练的重要性，以及自监督数据增强策略的有效性。In the ablation study on data augmentation, training with low-quality video data alone gives an FID of 0.698; training with high-quality Mocap data alone gives an FID of 0.557; combining the two lowers the FID to 0.467, which amounts to a 16% improvement. This strongly validates the importance of incorporating large-scale video motion data into training, as well as the effectiveness of the self-supervised data augmentation strategy.
04局限性Limitations
说明：论文未设独立局限性章节。以下各点综合了论文在结论部分明确提及（stated）的内容，以及从方法与实验设计中合理推断（inferred）的局限。Note: The paper has no dedicated limitations section. The points below combine what the paper explicitly mentions (stated) in its conclusion with limitations reasonably inferred (inferred) from its method and experimental design.
复杂灵巧操作任务成功率仍有提升空间（stated）Success rates on complex dexterous manipulation tasks still leave room for improvement (stated)
论文结论明确指出："In the future, we aim to enhance the success rate of humanoid robots in performing more complex loco-manipulation tasks."——说明当前框架在精细运动操作（loco-manipulation）任务上仍存在不足，尤其在涉及手部精细控制的场景中。The conclusion of the paper states explicitly: "In the future, we aim to enhance the success rate of humanoid robots in performing more complex loco-manipulation tasks."—indicating that the current framework remains inadequate on fine-grained loco-manipulation tasks, especially in scenarios involving fine hand control.
依赖 RGB 单目摄像头，缺乏深度感知（inferred）Reliance on a monocular RGB camera, with no depth perception (inferred)
视觉感知模块仅使用 RGB 摄像头采集第一人称图像，缺乏深度信息。对于需要精确三维定位的交互任务（如精确抓握），单目视觉可能引入定位误差，限制了交互精度。The visual perception module captures egocentric images using an RGB camera only and lacks depth information. For interaction tasks that require precise 3D localization (such as precise grasping), monocular vision may introduce localization errors and limit interaction precision.
对齐数据规模与质量的平衡问题（inferred）Balancing the scale and the quality of alignment data (inferred)
从视频中提取的低质量运动数据（human motion recovery）精度不足，而高质量 Mocap 数据规模有限。尽管两阶段训练策略有效缓解了这一矛盾，但数据质量的根本瓶颈仍制约着模型在极细粒度运动任务上的表现。The low-quality motion data extracted from videos (human motion recovery) is insufficiently accurate, while high-quality Mocap data is limited in scale. Although the two-stage training strategy effectively mitigates this tension, the fundamental data-quality bottleneck still constrains the model on extremely fine-grained motion tasks.
第一人称视觉微调数据规模较小（inferred）Small scale of the egocentric visual fine-tuning data (inferred)
视觉条件微调阶段依赖采集到的真实场景运动捕捉数据（与第一人称视觉同步），而遥操作采集成本高、规模有限，可能制约视觉感知能力在更广泛任务分布上的泛化性。The vision-conditioned fine-tuning stage relies on collected real-world motion capture data (synchronized with egocentric vision), while teleoperated collection is costly and limited in scale, which may constrain the generalization of the visual perception capability over broader task distributions.
