机器人 · Humanoid Whole-Body ControlRobotics · Humanoid Whole-Body Control
HAIC: Humanoid Agile Object Interaction Control via Dynamics-Aware World Model
让人形机器人在“看不见物体”的情况下，靠本体感知预判其高阶动力学，稳稳滑板、推拉小车、跨地形搬箱Enabling a humanoid robot to skateboard, push and pull carts and carry boxes across terrains steadily, anticipating an unseen object's high-order dynamics from proprioception alone
现有 Human-Object Interaction (HOI) 方法大多只处理 fully actuated objects（物体被端到端刚性约束在机器人手上）。而滑板、小车、行李箱这类 underactuated objects 有独立动力学、非完整约束，且常常挡住机器人视线。HAIC 提出一个 dynamics-aware world model (DWM)：仅用本体感知历史推断物体的位置/朝向/线角速度/线角加速度，再显式投影到一个静态几何先验点云上，构造出一个"看不见也能感知"的动态占据表征；配合非对称的两阶段 teacher-student 训练，在 Unitree G1 上实现了包括动态滑板在内的多种敏捷交互。Most existing Human-Object Interaction (HOI) methods handle only fully actuated objects (the object is rigidly constrained end-to-end to the robot's hand). Underactuated objects such as skateboards, carts and suitcases, by contrast, possess independent dynamics and non-holonomic constraints, and frequently block the robot's line of sight. HAIC proposes a dynamics-aware world model (DWM): it infers an object's position / orientation / linear and angular velocity / linear and angular acceleration from proprioceptive history alone, then explicitly projects them onto a static geometric-prior point cloud, constructing a dynamic occupancy representation that is "perceivable even when unseen"; combined with asymmetric two-stage teacher-student training, it achieves a variety of agile interactions on the Unitree G1, dynamic skateboarding included.
论文指出，主流 HOI 方法的一个共同盲区：只研究 fully actuated objects。The paper identifies a blind spot shared by mainstream HOI methods: they study only fully actuated objects.
“prevailing methodologies predominantly address the manipulation of fully actuated objects, where the target is rigidly coupled to the robot's end-effector and its state is strictly constrained by the robot's kinematics. This paradigm neglects the pervasive class of underactuated objects characterized by independent dynamics and non-holonomic constraints, which pose significant control challenges due to complex coupling forces and frequent visual occlusions.”
换句话说，像滑板、手推车这类物体一旦被抓住/踩上，它自己也会加速、转向、产生惯性力，还常常挡住机器人自身的摄像头——机器人既看不见地面，也看不见物体状态。论文类比人类：即使视觉被携带的物体挡住，人也能靠本体感知 + 内在心智模型推断物体状态，“fusion of geometric reasoning (knowing the object's boundaries) and dynamic sensing (feeling the forces to anticipate acceleration)”，从而提前调整重心。HAIC 就是要把这种能力赋予人形机器人：不依赖外部状态估计（如 motion capture、LiDAR、markers），只用机器人自身本体感知 + 一个物体的标准几何先验，来应对从 fully actuated 到 underactuated 的整个物体谱系。In other words, once an object such as a skateboard or a handcart is grasped or stepped on, it accelerates, turns and generates inertial forces of its own, and it often blocks the robot's own cameras — the robot can see neither the ground nor the object's state. The paper draws an analogy with humans: even when vision is occluded by the carried object, a person can infer the object's state from proprioception plus an internal mental model, a "fusion of geometric reasoning (knowing the object's boundaries) and dynamic sensing (feeling the forces to anticipate acceleration)", and shift the center of gravity in advance. HAIC sets out to endow humanoid robots with this ability: without relying on external state estimation (such as motion capture, LiDAR or markers), using only the robot's own proprioception plus a standard geometric prior of the object, so as to cover the whole spectrum of objects from fully actuated to underactuated.
Fig. 1: HAIC enables a robot to perform complex interactions, including (a) Underactuated Object Interaction — 学习滑板、推/拉小车等技能；(b) Sequential Interaction — 先把箱子搬上小车、再推着小车走，单一策略完成多物体接力；(c) Multi-terrain Interaction — 在平台、斜坡、楼梯等复合地形上边搬箱边保持交互。Fig. 1: HAIC enables a robot to perform complex interactions, including (a) Underactuated Object Interaction — learning skills such as skateboarding and pushing/pulling a cart; (b) Sequential Interaction — first loading a box onto a cart and then pushing the cart away, a single policy relaying across multiple objects; (c) Multi-terrain Interaction — carrying a box while sustaining the interaction over composite terrains such as platforms, slopes and stairs.Fig. 2a: 论文实验覆盖的三类物体/地形 —— Fully actuated objects（刚性抓取的箱子）、Underactuated objects（滑板、小车）、Composed terrains（斜坡+楼梯组合地形），后两者正是既有方法薄弱之处。Fig. 2a: The three classes of objects/terrains covered by the paper's experiments — Fully actuated objects (a rigidly grasped box), Underactuated objects (skateboard, cart) and Composed terrains (combined slope-and-stair terrain); the latter two are precisely where existing methods are weak.
100%“blind multi-terrain box carrying” 成功率（论文原话）success rate of "blind multi-terrain box carrying" (the paper's own wording)
02 Method
HAIC 把问题建模为 proprioception-based POMDP：训练时 teacher 能看到完整的 privileged state（含物体真实状态 $s_t^{obj}$），部署时 student 只能拿到本体感知 $s_t^{prop}$、参考动作 $s_t^{ref}$，以及一个“标准尺寸”的几何模板点云 $\mathcal{P}$（不是真实测量，只是语义先验）。核心是一个 Dynamics-aware World Model (DWM)，由 Object Adapter → Explicit Geometric Projection → Privilege Adapter 三步组成。HAIC casts the problem as a proprioception-based POMDP: during training the teacher observes the full privileged state (including the object's true state $s_t^{obj}$), while at deployment the student receives only proprioception $s_t^{prop}$, the reference motion $s_t^{ref}$, and a "standard-sized" geometric template point cloud $\mathcal{P}$ (not a real measurement, merely a semantic prior). At its core lies a Dynamics-aware World Model (DWM) composed of three steps: Object Adapter → Explicit Geometric Projection → Privilege Adapter.
Fig. 3: Dynamics-aware World Model 总览——先由 Object Adapter 从本体感知历史预测物体的相对位姿/速度/加速度 $\hat{s}_t^{obj}$，再把它投影到几何先验点云 $\mathcal{P}$ 得到动态点云 $\mathcal{P}_t^{dyn}$，最后由 Privilege Adapter 融合本体感知重建 privileged 特征 $\hat{z}_t^{priv}$。Fig. 3: Overview of the Dynamics-aware World Model — the Object Adapter first predicts the object's relative pose / velocity / acceleration $\hat{s}_t^{obj}$ from proprioceptive history, this is then projected onto the geometric-prior point cloud $\mathcal{P}$ to yield the dynamic point cloud $\mathcal{P}_t^{dyn}$, and finally the Privilege Adapter fuses it with proprioception to reconstruct the privileged feature $\hat{z}_t^{priv}$.
1. High-Order Dynamics Prediction（Object Adapter）
Object Adapter 是一个实时的 dynamics predictor：输入上一步预测的物体状态、一段本体感知历史窗口 $H_t=\{s_{t-n}^{prop},\dots,s_t^{prop}\}$，以及未来参考动作 $F_t=\{s_t^{ref},\dots,s_{t+m}^{ref}\}$，输出物体的相对位置 $\hat p_t$、朝向 $\hat R_t$、线/角速度 $(\hat v_t^{lin},\hat v_t^{ang})$，以及线/角加速度 $(\hat a_t^{lin},\hat a_t^{ang})$：$\hat s_t^{obj}=f_{OA}(\hat s_{t-1}^{obj},H_t,F_t)$。加速度这一项是关键——消融实验显示，只预测位姿（Vec-Pose）虽然能维持滑板平衡，却无法应付下板瞬间的冲击力；只有显式预测加速度，机器人才能提前做“前馈式”惯性补偿。The Object Adapter is a real-time dynamics predictor: it takes the object state predicted at the previous step, a window of proprioceptive history $H_t=\{s_{t-n}^{prop},\dots,s_t^{prop}\}$ and future reference motions $F_t=\{s_t^{ref},\dots,s_{t+m}^{ref}\}$, and outputs the object's relative position $\hat p_t$, orientation $\hat R_t$, linear/angular velocity $(\hat v_t^{lin},\hat v_t^{ang})$ and linear/angular acceleration $(\hat a_t^{lin},\hat a_t^{ang})$: $\hat s_t^{obj}=f_{OA}(\hat s_{t-1}^{obj},H_t,F_t)$. The acceleration term is the crux — the ablations show that predicting pose only (Vec-Pose) can keep the skateboard balanced yet fails to cope with the impulsive force at the instant of dismounting; only by explicitly predicting acceleration can the robot perform "feed-forward" inertial compensation ahead of time.
2. Explicit Geometric Projection
预测出的动力学状态被显式投影到一个静态几何先验（点云）上：$\mathcal{P}_t^{dyn}=\hat R_t\cdot\mathcal{P}+\hat p_t$。这一步把一个抽象的状态估计问题，转成一个“空间上落地”的动态占据表征（dynamic occupancy），使策略即便在视觉盲区也能感知碰撞边界与接触可达性（contact affordances）。论文强调 $\mathcal{P}$ 只是一个几何先验（geometric prior），不是 ground-truth 测量。随后 Privilege Adapter 融合动态点云 $\mathcal{P}_t^{dyn}$、当前本体感知与参考动作，输出预测的 privileged 特征 $\hat z_t^{priv}$，作为 student actor 的输入，模拟“privileged actor”的信息流。The predicted dynamics state is explicitly projected onto a static geometric prior (a point cloud): $\mathcal{P}_t^{dyn}=\hat R_t\cdot\mathcal{P}+\hat p_t$. This step turns an abstract state-estimation problem into a spatially grounded dynamic occupancy representation, so that the policy can perceive collision boundaries and contact affordances even inside a visual blind spot. The paper stresses that $\mathcal{P}$ is only a geometric prior, not a ground-truth measurement. The Privilege Adapter then fuses the dynamic point cloud $\mathcal{P}_t^{dyn}$, the current proprioception and the reference motion, and outputs the predicted privileged feature $\hat z_t^{priv}$ as the input of the student actor, emulating the information flow of a "privileged actor".
Fig. 4: 两阶段训练框架——Stage 1（Feature Alignment & Warm-up）训练 privileged teacher，同时让 world model 与 student actor 模仿 teacher 的行为和内部表征；Stage 2（Asymmetric Fine-tuning）用非对称 actor-critic 微调 student，critic 仍能访问 privileged state，同时持续优化 world model 以适应 student 自身探索带来的分布偏移（distribution shift）。Fig. 4: The two-stage training framework — Stage 1 (Feature Alignment & Warm-up) trains the privileged teacher while having the world model and the student actor imitate the teacher's behavior and internal representations; Stage 2 (Asymmetric Fine-tuning) fine-tunes the student with an asymmetric actor-critic in which the critic still accesses the privileged state, and keeps optimizing the world model to accommodate the distribution shift induced by the student's own exploration.
3. 两阶段训练与 EMA 稳定化3. Two-Stage Training and EMA Stabilization
统一的 world model loss 为预测误差与特征对齐误差的加权和：$\mathcal{L}_{WM}=\lambda_{obj}\|\hat s_t^{obj}-s_t^{obj}\|^2+\lambda_{priv}\|\hat z_t^{priv}-z_t^{priv}\|^2$。Stage 1 目标 $\mathcal{L}_{Stage1}=\mathcal{L}_{PPO}(\pi_T,V_T)+\mathcal{L}_{WM}+\lambda_{distill}D_{KL}(\pi_S\|\pi_T)$；Stage 2（asymmetric fine-tuning）目标 $\mathcal{L}_{Stage2}=\mathcal{L}_{PPO}(\pi_S,V_T)+\mathcal{L}_{WM}$——关键是 world model 在两个阶段都持续被优化，“ensuring the world model continuously adapts to the evolving trajectory distribution of the student policy, preventing the 'blind' policy from collapsing due to inaccurate state estimation”。此外还维护 world model 权重的 EMA，rollout 时用 EMA 平滑后的预测与环境交互，梯度则通过在线网络反传，以缓解策略与世界模型耦合训练带来的不稳定。The unified world model loss is a weighted sum of the prediction error and the feature-alignment error: $\mathcal{L}_{WM}=\lambda_{obj}\|\hat s_t^{obj}-s_t^{obj}\|^2+\lambda_{priv}\|\hat z_t^{priv}-z_t^{priv}\|^2$. The Stage 1 objective is $\mathcal{L}_{Stage1}=\mathcal{L}_{PPO}(\pi_T,V_T)+\mathcal{L}_{WM}+\lambda_{distill}D_{KL}(\pi_S\|\pi_T)$; the Stage 2 (asymmetric fine-tuning) objective is $\mathcal{L}_{Stage2}=\mathcal{L}_{PPO}(\pi_S,V_T)+\mathcal{L}_{WM}$ — crucially, the world model keeps being optimized in both stages, "ensuring the world model continuously adapts to the evolving trajectory distribution of the student policy, preventing the 'blind' policy from collapsing due to inaccurate state estimation". An EMA of the world model weights is additionally maintained: rollouts interact with the environment through the EMA-smoothed predictions while gradients are back-propagated through the online network, alleviating the instability brought by training the policy and the world model in a coupled manner.
4. Multiple Objects Contact Reward
面对多物体的顺序交互任务（如先搬箱子上车、再推车），HAIC 设计了一个统一 guidance（接触前，最小化末端执行器到语义目标的距离）与 execution（接触后，在保持几何约束的同时施加合适的力）两阶段的 contact reward $r_{contact}$，且接触判定用标准化脚本自动计算，而非人工标注，具备跨任务通用性。For sequential interaction tasks involving multiple objects (e.g. first loading a box onto a cart, then pushing the cart), HAIC designs a contact reward $r_{contact}$ that unifies two phases — guidance (before contact, minimizing the distance from the end-effector to the semantic target) and execution (after contact, applying an appropriate force while preserving the geometric constraint) — with contact determined automatically by a standardized script rather than by manual annotation, which makes it general across tasks.
03 Experiments
评测平台为 Unitree G1 人形机器人，纯本体计算、无外部动捕。训练用 Isaac Sim + PPO，单张 RTX 4090，两阶段训练约 8 小时。Baseline 是 HDMI*——原始 HDMI [Weng et al., 2025] 的 proprioception-only 变体，同样从零训练以保证公平对比；HDMI 原版在仿真中的 sim-to-sim 上界也作为参照。三类任务：Underactuated Object Interaction（滑板、推/拉车）、Sequential Interaction（先装箱再拉/推车）、Multi-terrain Interaction（负重跨越平台/斜坡/楼梯）。The evaluation platform is the Unitree G1 humanoid, computing purely onboard with no external motion capture. Training uses Isaac Sim + PPO on a single RTX 4090, the two stages taking about 8 hours. The baseline is HDMI* — a proprioception-only variant of the original HDMI [Weng et al., 2025], likewise trained from scratch to ensure a fair comparison; the original HDMI's sim-to-sim upper bound in simulation also serves as a reference. Three task families: Underactuated Object Interaction (skateboard, push/pull cart), Sequential Interaction (load the box first, then pull/push the cart) and Multi-terrain Interaction (traversing platforms/slopes/stairs under load).
注：完整表格（含 Empboe/Empjpe/Empbve/Empbae/Empjve 等本体与物体状态指标）见论文 Table I–IV；此处只摘录 Success Rate 与 Empbpe（Root-relative Mean Per Body Position Error）便于对比，数字均逐字取自论文表格。Note: the complete tables (with body- and object-state metrics such as Empboe/Empjpe/Empbve/Empbae/Empjve) are given in Table I–IV of the paper; only Success Rate and Empbpe (Root-relative Mean Per Body Position Error) are excerpted here for ease of comparison, and all numbers are taken verbatim from the paper's tables.
Fig. 6: Real-world performance comparison with the baseline across various tasks. “With the specifically designed dynamics-aware world model, HAIC maintains robust stability throughout the interaction, whereas the baseline suffers from balance failures and trajectory drift.”
Ablations
论文设计了 5 个变体做消融：Proprio（纯本体感知，无物体表征）、Vec-Pose（只加预测的位姿 $\hat p,\hat R$）、Vec-Dyn（在 Vec-Pose 基础上再加速度/速度，但仍是扁平向量输入）、Geo-Pose（把预测位姿投影到几何点云，但不含高阶动力学）、以及完整的 HAIC（位姿+动力学+几何投影三者齐全）。滑板任务上，“Proprio struggles to maintain balance even during the glide phase”；“Vec-Pose achieves perfect gliding success, it fails during the dismount phase, indicating that position perception aids balance but lacks the foresight to handle impulsive forces upon stepping off”；而 HAIC “achieves high success rates across both phases with the lowest tracking errors”。推车任务上，“methods utilizing geometric projection (Geo-Pose and HAIC) achieve significantly tighter global tracking”，即几何投影和高阶动力学预测分别贡献了不同的能力，二者叠加（HAIC）综合最优。另外与 RMA（隐式环境 latent 自适应）相比，论文报告 HAIC “achieves superior or competitive success rates and significantly lower tracking errors across most kinematic metrics”，验证了显式动力学建模优于隐式自适应。泛化性方面，论文测试了训练范围外的物体尺寸/重量（0.1–0.2m、1–2kg 之外，测试 0.25m、3kg）、随机旋转的地形，以及瞬时侧向推力扰动，均维持稳定；并额外在真实家具（桌子、行李箱、带额外偏航自由度的椅子）上测试，达到 100% SR。The paper ablates five variants: Proprio (proprioception only, no object representation), Vec-Pose (adding only the predicted pose $\hat p,\hat R$), Vec-Dyn (adding acceleration/velocity on top of Vec-Pose, still as a flat vector input), Geo-Pose (projecting the predicted pose onto the geometric point cloud but without high-order dynamics), and the complete HAIC (pose + dynamics + geometric projection all present). On the skateboarding task, "Proprio struggles to maintain balance even during the glide phase"; "Vec-Pose achieves perfect gliding success, it fails during the dismount phase, indicating that position perception aids balance but lacks the foresight to handle impulsive forces upon stepping off"; whereas HAIC "achieves high success rates across both phases with the lowest tracking errors". On the cart tasks, "methods utilizing geometric projection (Geo-Pose and HAIC) achieve significantly tighter global tracking" — that is, geometric projection and high-order dynamics prediction each contribute a distinct capability, and stacking the two (HAIC) is best overall. Compared with RMA (implicit latent environment adaptation), the paper further reports that HAIC "achieves superior or competitive success rates and significantly lower tracking errors across most kinematic metrics", confirming that explicit dynamics modeling outperforms implicit adaptation. As for generalization, the paper tests object sizes/weights outside the training range (beyond 0.1–0.2m and 1–2kg, testing 0.25m and 3kg), randomly rotated terrains and instantaneous lateral push disturbances, remaining stable throughout; it additionally tests on real furniture (a table, a suitcase, and a chair with an extra yaw degree of freedom), reaching 100% SR.
04 Limitations
Note:论文正文没有单独的 "Limitations" 小节；下面第 1、2 条是论文在 Failure Case Analysis / 结果讨论中明确写出的（已标 stated），第 3、4 条是基于方法设计与实验设置推断的（已标 inferred）。The body of the paper has no standalone "Limitations" section; items 1 and 2 below are explicitly written out by the paper in its Failure Case Analysis / results discussion (marked stated), while items 3 and 4 are inferred from the method design and the experimental setup (marked inferred).
[stated] 高速/大冲击场景下 DWM 预测会漂移，进而导致失败[stated] DWM predictions drift under high-speed / high-impact conditions, which in turn causes failures
论文的 failure case analysis 明确指出：“for challenging tasks like pushing or pulling cart and skateboarding, the predicted states exhibit drift from the simulated trajectories, tending to revert toward the reference motion state... When skateboard decelerates and lands, divergence from the actual state will lead to failure (60% SR).” 也就是说 skateboarding 的 complete（含下板）成功率只有 60%，明显低于 glide 阶段的 100%，说明对高动态、非线性强的减速/落地时刻，DWM 的预测精度仍是瓶颈。The paper's failure case analysis states explicitly: "for challenging tasks like pushing or pulling cart and skateboarding, the predicted states exhibit drift from the simulated trajectories, tending to revert toward the reference motion state... When skateboard decelerates and lands, divergence from the actual state will lead to failure (60% SR)." That is, the success rate of the complete skateboarding task (including the dismount) is only 60%, clearly below the 100% of the glide phase, indicating that the prediction accuracy of the DWM remains the bottleneck at highly dynamic, strongly nonlinear deceleration/landing moments.
[stated] 序列任务（Pull Cart w/ Box）成功率仍有限[stated] The success rate on the sequential task (Pull Cart w/ Box) is still limited
Table III 显示，在“Pull Cart w/ Box”这一多物体、长时序、模式切换的复合任务上，HAIC 的成功率为 40%（相对 baseline 的 0% 已是明显提升，但远低于其它单物体任务接近 100% 的表现），论文也承认这类任务存在“compounded complexity and error accumulation over a longer horizon”。Table III shows that on "Pull Cart w/ Box" — a composite task with multiple objects, a long horizon and mode switching — HAIC attains a success rate of 40% (already a clear improvement over the baseline's 0%, yet far below the close-to-100% performance on the other single-object tasks); the paper likewise concedes that such tasks involve "compounded complexity and error accumulation over a longer horizon".
[inferred] 依赖“标准尺寸”几何先验模板，对未知类别物体的泛化尚未验证[inferred] Reliance on a "standard-sized" geometric-prior template; generalization to unseen object categories remains unverified
Student 部署时使用的是一个“Nominal geometric template $\mathcal{P}\in\mathbb{R}^{N\times3}$: a standard-sized point cloud acting as a semantic prior”，论文验证了同一类别物体（箱子、滑板、小车）在尺寸/重量上的泛化（如 0.25m、3kg），但方法本质上仍需要为每类物体准备一个几何先验模板，对训练中未出现过的全新物体类别（不同拓扑/形状先验）是否可行，论文未讨论。At deployment the student uses a "Nominal geometric template $\mathcal{P}\in\mathbb{R}^{N\times3}$: a standard-sized point cloud acting as a semantic prior". The paper verifies generalization in size/weight within the same object category (box, skateboard, cart), e.g. 0.25m and 3kg, but the method inherently still requires a geometric-prior template for each object category; whether it is viable for entirely new object categories unseen during training (with different topology/shape priors) is not discussed.
[inferred] 训练与真机验证均限定在单一机型（Unitree G1）与单 GPU 规模[inferred] Both training and real-robot validation are confined to a single platform (Unitree G1) and a single-GPU scale
论文的仿真训练、真机部署实验均基于 Unitree G1 一款人形机器人，训练在单张 RTX 4090 上完成（约 8 小时/两阶段）；未讨论跨本体（不同尺寸/自由度人形机器人）迁移或大规模分布式训练下的表现。The paper's simulation training and real-robot deployment experiments are all based on one humanoid platform, the Unitree G1, with training completed on a single RTX 4090 (about 8 hours for the two stages); transfer across embodiments (humanoids of different size/degrees of freedom) and performance under large-scale distributed training are not discussed.
