Humanoid Everyday：面向开放世界类人机器人操作的综合数据集

01 动机Motivation

通用人形机器人的进步依赖于大规模、多样化的数据，而现有数据集存在明显短板：任务种类少、传感器模态单一、缺乏标准化评测手段。Humanoid Everyday 旨在从根本上弥补这些缺口。Progress toward general-purpose humanoid robots depends on large-scale, diverse data, yet existing datasets have clear shortcomings: few task types, limited sensor modalities, and no standardized evaluation. Humanoid Everyday sets out to close these gaps at their root.

"现有机器人操作数据集往往局限于桌面操作场景，且大多基于非人形平台，难以捕捉类人机器人在真实世界中面临的高自由度、多任务挑战……我们提出 Humanoid Everyday——一个大规模、多模态的人形机器人操作数据集，覆盖 dextrous object manipulation、human-humanoid interaction 与 locomotion-integrated actions。""Existing robotic manipulation datasets are often confined to tabletop settings and are mostly built on non-humanoid platforms, making it hard to capture the high degrees of freedom and multi-task challenges that humanoid robots face in the real world... We present Humanoid Everyday, a large-scale, multi-modal humanoid manipulation dataset covering dextrous object manipulation, human-humanoid interaction and locomotion-integrated actions."

Humanoid Everyday 数据集总览 — **图1：数据集总览。**Humanoid Everyday 覆盖 7 大类人形操作任务，提供丰富的多模态信息，并配备云端评测平台，支持标准化策略部署与协作研究。**Figure 1: Dataset overview.** Humanoid Everyday covers 7 major categories of humanoid manipulation tasks, provides rich multimodal information, and is paired with a cloud-based evaluation platform supporting standardized policy deployment and collaborative research.

10.3k条轨迹（trajectories）trajectories

3M+帧数据（frames）frames

260个独特任务（unique tasks）unique tasks

7大任务类别major task categories

三大核心贡献Three Core Contributions

大规模多模态数据集：真实世界采集，含 RGB、深度图、LiDAR、触觉及自然语言标注，优化后的遥控采集流水线将采集时间减半。Large-scale multimodal dataset: collected in the real world, containing RGB, depth maps, LiDAR, tactile signals and natural language annotations; an optimized teleoperation pipeline halves the collection time.
代表性策略评测分析：对主流 imitation learning 方法（Diffusion Policy、ACT、OpenVLA、π₀ 等）进行系统评测，揭示其优劣势。Benchmarking of representative policies: a systematic evaluation of mainstream imitation learning methods (Diffusion Policy, ACT, OpenVLA, π₀ and others), revealing their strengths and weaknesses.
云端评测平台：支持标准化、可复现、协作式的策略部署与测试，无需研究者自备硬件。Cloud-based evaluation platform: supports standardized, reproducible and collaborative policy deployment and testing, with no hardware required on the researcher side.

02 方法Method

数据集的核心是一套高效的采集系统：基于 Apple Vision Pro 的遥控接口 + 异步多进程流水线，搭载 Unitree G1/H1 两款人形机器人，以 30Hz 频率同步采集多模态传感数据。At the core of the dataset is an efficient collection system: an Apple Vision Pro based teleoperation interface plus an asynchronous multi-process pipeline, running on the two Unitree G1/H1 humanoid robots and synchronously recording multimodal sensor data at 30 Hz.

硬件平台Hardware Platforms

Unitree G1（29-DoF）Unitree G1 (29-DoF)

7-DoF 三指灵巧手（dexterous hands），内置触觉传感器7-DoF three-finger dexterous hands with built-in tactile sensors
Intel RealSense RGB-D 相机 + Livox LiDAR 系统Intel RealSense RGB-D camera + Livox LiDAR system
总动作空间维度：28 DoFsTotal action space dimensionality: 28 DoFs

Unitree H1（27-DoF）Unitree H1 (27-DoF)

6-DoF INSPIRE 手爪6-DoF INSPIRE hands
Intel RealSense RGB-D 相机 + Livox LiDAR 系统Intel RealSense RGB-D camera + Livox LiDAR system
支持 Loco-Manipulation（行走中操作）任务Supports Loco-Manipulation (manipulation while walking) tasks

遥控接口（Teleoperation Interface）Teleoperation Interface

操作员佩戴 Apple Vision Pro，利用其底部摄像头实时捕捉手腕与手指关键点；手指动作经由 dex-retargeting 系统映射到机器人灵巧手，手腕姿态则经基于 Pinocchio 的逆运动学算法（IK）转化为关节控制指令。The operator wears an Apple Vision Pro and uses its bottom-facing cameras to capture wrist and finger keypoints in real time; finger motions are mapped onto the robot dexterous hands through a dex-retargeting system, while wrist poses are turned into joint control commands by a Pinocchio-based inverse kinematics (IK) algorithm.

高效采集流水线Efficient Collection Pipeline

数据采集流水线架构图 — **图2：数据采集流水线。**将 IO 数据流、IK 计算与机器人关节控制解耦到独立进程，并通过共享内存缓冲区实现低延迟通信，配合多进程与异步 IO 确保高频遥控与高质量数据采集。**Figure 2: Data collection pipeline.** IO data streams, IK computation and robot joint control are decoupled into separate processes and communicate through shared-memory buffers with low latency; multi-processing and asynchronous IO together ensure high-frequency teleoperation and high-quality data collection.

流水线性能对比 — **图3：性能对比。**与 Unitree 官方遥控系统相比，本文流水线将控制延迟从 500ms 降至 2ms，数据采集时间减半。**Figure 3: Performance comparison.** Compared with the official Unitree teleoperation system, this pipeline cuts the control delay from 500 ms to 2 ms and halves the data collection time.

"Our pipeline halves data collection time compared to the official Unitree teleoperation system, while the control delay decreases from 500 ms to 2 ms."

任务分类体系Task Taxonomy

任务分布图 — **图4：任务与技能类别分布。**260 个任务按难度和场景分为 7 大类，每类均含约 40 条采集轨迹。**Figure 4: Distribution of tasks and skill categories.** The 260 tasks are grouped by difficulty and scenario into 7 major categories, each containing about 40 collected trajectories.

Basic Manipulation：抓取、放置等基础操作Basic Manipulation: basic operations such as grasping and placing
Deformable Manipulation：软体/柔性物体操作（如折叠布料）Deformable Manipulation: manipulation of soft and deformable objects, such as folding cloth
Articulated Manipulation：铰链物体操作（如开关门、抽屉）Articulated Manipulation: manipulation of articulated objects, such as opening doors and drawers
Tool Use：使用工具完成任务Tool Use: completing tasks with tools
High-Precision Manipulation：高精度精细操作（如将玫瑰插入花瓶）High-Precision Manipulation: fine-grained high-precision operations, such as inserting a rose into a vase
Human-Robot Interaction (HRI)：人机协同任务Human-Robot Interaction (HRI): collaborative human-robot tasks
Loco-Manipulation：行走与操作相结合的复合任务Loco-Manipulation: composite tasks combining locomotion and manipulation

云端评测平台Cloud-Based Evaluation Platform

03 实验Experiments

在 7 类任务的代表性子集上，对 7 种主流 imitation learning 策略进行系统评测，每种策略每个任务执行 10 次试验，成功率（success rate）为核心指标。On a representative subset of the 7 task categories, 7 mainstream imitation learning policies are systematically evaluated, with 10 trials per policy per task; success rate is the core metric.

实验任务设置 — **图6：实验任务示例。**7 大类别中各选取代表性任务；黄色区域为任务执行范围（含轻微位置扰动），箭头示意机器人手臂轨迹。**Figure 6: Example experimental tasks.** A representative task is picked from each of the 7 categories; the yellow region marks the task execution range (including slight positional perturbations), and the arrows sketch the robot arm trajectory.

主要策略性能对比（成功率 %）Policy Performance Comparison (Success Rate %)

策略 / MethodMethod	Articulate	Tool Use	Basic	Deformable	HRI	Loco-Manip	High Prec.	平均Average
Diffusion Policy	100%	0%	30%	0%	40%	30%	0%	29%
DP3	90%	70%	20%	20%	40%	0%	0%	34%
ACT	100%	0%	70%	0%	70%	0%	0%	34%
OpenVLA	70%	30%	30%	40%	60%	30%	10%	39%
π₀-FAST	100%	40%	60%	20%	30%	10%	0%	37%
π₀.₅	100%	40%	30%	40%	40%	0%	0%	36%
GR00T N1.5	100%	0%	80%	50%	100%	30%	0%	51%

GR00T N1.5 在平均成功率上以 51% 领先所有方法，尤其在 Articulated Manipulation 和 Human-Robot Interaction 任务上达到满分（100%）。High-Precision Manipulation 类别对所有策略均构成极高挑战，几乎所有方法均为 0%。GR00T N1.5 leads all methods in average success rate at 51%, and reaches a perfect score (100%) on Articulated Manipulation and Human-Robot Interaction in particular. The High-Precision Manipulation category is extremely challenging for every policy, with almost all methods at 0%.

Humanoid Everyday 作为预训练 Prior 的消融实验Ablation: Humanoid Everyday as a Pretraining Prior

预训练消融实验 — **图7：预训练消融对比。**对比直接任务特定 fine-tuning 与先在 Humanoid Everyday 上预训练再 fine-tuning 的两阶段方案，结果显示以本数据集为 prior 可显著提升目标任务性能。**Figure 7: Pretraining ablation.** Direct task-specific fine-tuning is compared with a two-stage scheme that first pretrains on Humanoid Everyday and then fine-tunes; using this dataset as a prior markedly improves target-task performance.

两阶段 fine-tuning（先在 Humanoid Everyday 预训练，再迁移到目标任务）优于直接 fine-tuning，验证了本数据集作为通用 humanoid prior 的价值。Two-stage fine-tuning (pretraining on Humanoid Everyday first, then transferring to the target task) outperforms direct fine-tuning, confirming the value of this dataset as a general humanoid prior.

评测平台运行情况Evaluation Platform in Operation

评测系统可持续运行超过 100 分钟，直至电池耗尽The evaluation system runs continuously for more than 100 minutes, until the battery is drained
期间仅需 3 次人工干预（原因：电机过热）Only 3 human interventions were needed during that period (cause: motor overheating)
每分钟可完成多步评测，显著提升评测吞吐量Multiple evaluation steps can be completed per minute, substantially raising evaluation throughput

04 局限性Limitations

注：以下局限性均由论文作者明确陈述（Discussion and Conclusion 节），并非推断。Note: all limitations below are explicitly stated by the authors (Discussion and Conclusion section) rather than inferred.

高维动作空间导致端到端策略性能普遍偏低High-dimensional action spaces keep end-to-end policy performance uniformly low

"all the end-to-end imitation policies struggle in humanoid manipulation tasks due to the high-dimensional action space in our dataset"——类人机器人 28 DoFs 的动作空间远超桌面机器人，现有策略架构尚未充分适配。"all the end-to-end imitation policies struggle in humanoid manipulation tasks due to the high-dimensional action space in our dataset" — the 28-DoF action space of a humanoid far exceeds that of tabletop robots, and existing policy architectures are not yet well adapted to it.

Loco-Manipulation 任务中点云输入退化Point cloud input degrades in Loco-Manipulation tasks

在机器人需要移动的 Loco-Manipulation 任务中，点云帧间变化剧烈，3D 输入的可靠性低于 RGB 图像，DP3 等依赖点云的方法在该类任务成功率为 0%。In Loco-Manipulation tasks where the robot has to move, point clouds change drastically between frames, so 3D input is less reliable than RGB images; point-cloud-based methods such as DP3 obtain a 0% success rate on this category.

高精度操作任务对所有策略均构成极高挑战High-precision manipulation is extremely hard for every policy

"nearly all policies achieve a 0% success rate" on high-precision tasks——现有模型缺乏 fine-grained visuospatial perception，无法完成如"将玫瑰插入花瓶"等精密操作。"nearly all policies achieve a 0% success rate" on high-precision tasks — existing models lack fine-grained visuospatial perception and cannot carry out delicate operations such as "inserting a rose into a vase".

OpenVLA 在高频数据上训练效果差OpenVLA trains poorly on high-frequency data

"OpenVLA does not compress the action space and thus when trained on high-frequency 30 Hz data, it often fails to produce meaningful motions"——不压缩动作空间的 VLA 方法对高频率数据适应性不足。"OpenVLA does not compress the action space and thus when trained on high-frequency 30 Hz data, it often fails to produce meaningful motions" — VLA methods that do not compress the action space adapt poorly to high-frequency data.

云端平台暂不支持场景自动复位The cloud platform does not yet support automatic scene resetting

"our cloud-based evaluation system does not yet support automatic scene resetting, as current imitation learning policies are not sufficiently robust for humanoids to recover the environments without human assistance"——每次评测后场景仍需人工重置。"our cloud-based evaluation system does not yet support automatic scene resetting, as current imitation learning policies are not sufficiently robust for humanoids to recover the environments without human assistance" — after each evaluation the scene still has to be reset by hand.

需要更专门化的模型设计More specialized model designs are needed

现有评测仅覆盖已有 imitation learning 架构，"their performance degrades on more challenging tasks due to the high dimensionality of humanoid action spaces, indicating the need for more specialized model designs"。The current evaluation covers only existing imitation learning architectures: "their performance degrades on more challenging tasks due to the high dimensionality of humanoid action spaces, indicating the need for more specialized model designs".