SAM 3D Body：全身人体网格重建

01 动机Motivation

从单张图像估计三维人体姿态与形状，对机器人、生物力学等真实世界应用至关重要。然而，现有方法在复杂姿态、严重遮挡或非常规视角下表现不稳定，且难以在统一框架中精确估计身体、手部和脚部。Estimating 3D human pose and shape from a single image is essential for real-world applications such as robotics and biomechanics. Existing methods, however, are unstable under challenging poses, severe occlusion or uncommon viewpoints, and struggle to accurately estimate body, hands and feet within a unified framework.

"existing approaches still exhibit unsatisfactory robustness when applied to in-the-wild images, which limits their applicability to real-world scenarios such as robotics and biomechanics. In particular, current models often fail on individuals presenting challenging poses, severe occlusion, or captured from uncommon viewpoints. They also struggle to reliably estimate both the overall body pose and the fine details of the hands and feet in a unified full-body framework."

SAM 3D Body 示例结果 — 图 1：SAM 3D Body（3DB）的人体网格恢复结果。模型对复杂姿态、多种视角的估计体现出出色的鲁棒性，并在统一框架中实现精准的全身（含手部）姿态估计。Figure 1: Human mesh recovery results of SAM 3D Body (3DB). The model shows strong robustness on challenging poses and diverse viewpoints, and delivers accurate full-body (including hands) pose estimation within a unified framework.

核心挑战Core challenges

数据层面Data side

大规模多样人体姿态数据集的采集天然困难且计算代价高昂Collecting large-scale, diverse human pose datasets is inherently difficult and computationally expensive
现有数据集要么多样性不足（实验室采集），要么网格质量低（伪标注）Existing datasets either lack diversity (lab captures) or provide low-quality meshes (pseudo-annotations)
现有野外数据集在规模和多样性上仍显不足Existing in-the-wild datasets remain insufficient in scale and diversity

模型层面Model side

现有架构未充分考虑身体与手部姿态估计在优化机制上的本质差异Existing architectures do not adequately account for the intrinsic difference in the optimization of body and hand pose estimation
缺乏有效的训练策略来处理单目图像中的不确定性和模糊性Effective training strategies for handling the uncertainty and ambiguity of monocular images are lacking
SMPL 等传统表示将骨骼结构与体表形状混杂，可解释性受限Conventional representations such as SMPL entangle skeletal structure with body surface shape, which limits interpretability

7M高质量标注训练图像High-quality annotated training images

5:1用户偏好研究胜率（vs. 最强基线 NLF 胜率 83.8%）Win rate in the user preference study (83.8% against the strongest baseline NLF)

7,800人类偏好研究参与人数Participants in the human preference study

SoTA五大标准基准 + 五个新数据集全面领先Leading across five standard benchmarks + five new datasets

02 方法Method

3DB 采用可提示的编码器-解码器架构：共享图像编码器负责提取特征，独立的身体解码器与手部解码器分别估计姿态参数，并基于全新的 Momentum Human Rig（MHR）参数化表示输出高保真全身网格。3DB adopts a promptable encoder-decoder architecture: a shared image encoder extracts features, while separate body and hand decoders estimate pose parameters, and a high-fidelity full-body mesh is produced on top of the new Momentum Human Rig (MHR) parametric representation.

图 2：SAM 3D Body 模型架构。共享图像编码器配合独立的 Body Decoder 和 Hand Decoder，支持 2D 关键点、掩码等多种可选提示输入（图中 * 标注为可选 token）。MHR 参数化表示解耦骨骼结构（Skeleton）与体表形状（Shape）。Figure 2: SAM 3D Body model architecture. A shared image encoder is paired with separate Body Decoder and Hand Decoder, supporting optional prompts such as 2D keypoints and masks (* in the figure marks optional tokens). The MHR parametric representation disentangles skeletal structure (Skeleton) from body surface shape (Shape).

MHR 参数化表示MHR parametric representation

Momentum Human Rig（MHR）是 ATLAS 的增强版，显式解耦骨骼结构与体表形状，提供更丰富的可控性和可解释性。相比 SMPL 系列将两者混杂于形状空间，MHR 能更直观地映射至骨骼长度等物理参数。The Momentum Human Rig (MHR) is an enhanced version of ATLAS that explicitly disentangles skeletal structure from body surface shape, offering richer controllability and interpretability. Compared with the SMPL family, which entangles the two in a single shape space, MHR maps more directly onto physical parameters such as bone lengths.

可提示式推理Promptable inference

受 SAM 家族启发，3DB 支持多种可选提示：2D 关键点提示通过位置编码叠加至学习 embedding 后作为额外 token 输入姿态解码器；掩码提示经卷积编码后与图像 embedding 逐元素相加。提示机制在训练阶段作为交互式引导，天然助力模糊场景下的姿态估计。Inspired by the SAM family, 3DB supports several optional prompts: 2D keypoint prompts are added to a learned embedding via positional encoding and fed to the pose decoder as extra tokens; mask prompts are encoded by convolutions and added element-wise to the image embedding. Used as interactive guidance during training, the prompting mechanism naturally helps pose estimation in ambiguous scenes.

双解码器设计Two-way decoder design

3DB 的核心创新之一是两路解码器（Two-way Decoder）设计。Body Decoder 负责全身姿态，Hand Decoder 专注手部局部细节（输入可包含手部裁剪图），有效缓解了身体与手部在输入分辨率、相机估计和监督目标上的冲突优化问题。One of the core innovations of 3DB is the Two-way Decoder design. The Body Decoder handles full-body pose while the Hand Decoder focuses on local hand detail (its input may include hand crops), effectively alleviating the conflicting optimization of body and hands over input resolution, camera estimation and supervision targets.

数据引擎与标注流程Data engine and annotation pipeline

数据质量（Quality）：标注流程结合几何约束、参数先验与稠密关键点回归，自动生成高质量 3D 人体网格标注。Quality: the annotation pipeline combines geometric constraints, parametric priors and dense keypoint regression to automatically produce high-quality 3D human mesh annotations.
数据规模（Quantity）：从大型授权图库、多视图采集数据集及合成数据中整合，共创建 7 百万张高质量标注图像。Quantity: consolidating large licensed image libraries, multi-view capture datasets and synthetic data yields a total of 7 million high-quality annotated images.
数据多样性（Diversity）：基于 VLM 的数据引擎主动挖掘野外困难图像并路由至标注，确保覆盖稀有姿态、困难视角和多样外观。Diversity: a VLM-based data engine actively mines difficult in-the-wild images and routes them to annotation, ensuring coverage of rare poses, difficult viewpoints and diverse appearances.

03 实验Experiments

在五大标准基准（3DPW, EMDB, RICH, COCO, LSPET）以及五个全新数据集上与业界最优方法（HMR2.0b, CameraHMR, PromptHMR, SMPLer-X, NLF, SMPLest-X 等）进行全面对比，报告 MPJPE、PA-MPJPE、PVE 及 PCK 等标准指标。A comprehensive comparison against state-of-the-art methods (HMR2.0b, CameraHMR, PromptHMR, SMPLer-X, NLF, SMPLest-X and others) on five standard benchmarks (3DPW, EMDB, RICH, COCO, LSPET) and five brand-new datasets, reporting standard metrics such as MPJPE, PA-MPJPE, PVE and PCK.

标准基准对比（Table 2 节选，3DPW & EMDB）Standard benchmarks (excerpt of Table 2, 3DPW & EMDB)

方法Method	3DPW PA-MPJPE↓	3DPW MPJPE↓	EMDB PA-MPJPE↓	EMDB MPJPE↓
HMR2.0b	54.3	81.3	79.2	118.5
CameraHMR	35.1	56.0	43.3	70.3
PromptHMR	36.1	58.7	41.0	71.7
NLF-L+fit*	33.6	54.9	40.9	68.4
3DB-H（Ours）3DB-H (Ours)	33.2	54.8	38.5	62.9
3DB-DINOv3（Ours）3DB-DINOv3 (Ours)	33.8	54.8	38.2	61.7

* NLF 使用 RICH 数据训练；3DB 未使用 RICH。粗体为最优，下划线为次优。* NLF is trained with RICH data; 3DB does not use RICH. Bold marks the best result, underline the second best.

五个新数据集泛化对比（Table 3 节选，Leave-one-out）Generalization to five new datasets (excerpt of Table 3, leave-one-out)

方法Method	EE4D-Phy PVE↓	Harmony4D PVE↓	Goliath PVE↓	SA1B-Hard PVE↓
CameraHMR	71.1	84.6	66.7	102.8
PromptHMR	74.6	91.9	67.2	92.7
NLF	75.9	97.3	66.5	97.6
3DB-H Leave-one-out（Ours）3DB-H leave-one-out (Ours)	49.7	63.5	54.2	85.6
3DB-H Full dataset（Ours）3DB-H full dataset (Ours)	37.0	41.0	34.5	55.2

手部姿态估计（FreiHand，Table 4）Hand pose estimation (FreiHand, Table 4)

方法Method	PA-MPVPE↓	PA-MPJPE↓	F@5↑	F@15↑
WiLoR†（手部专用）WiLoR† (hand-specific)	5.1	5.5	0.825	0.993
3DB-H（Ours，全身模型）3DB-H (Ours, full-body model)	6.3	5.5	0.735	0.988
3DB-DINOv3（Ours）3DB-DINOv3 (Ours)	6.2	5.5	0.737	0.988

† 使用 FreiHand 数据集训练的手部专用方法。3DB 作为全身模型与手部专用模型性能相当。† A hand-specific method trained on the FreiHand dataset. As a full-body model, 3DB performs on par with hand-specific models.

人类偏好研究（Human Preference Study）Human preference study

用户偏好研究结果 — 图 8：3DB 与六种基线方法的用户偏好研究 Win Rate 对比。共 7,800 名参与者，每组 1,300 人。3DB 对最强基线 NLF 的胜率为 83.8%（67/80），对 SMPLest-X 达到 100%（80/80）。Figure 8: win-rate comparison of the user preference study between 3DB and six baseline methods, with 7,800 participants in total and 1,300 per group. The win rate of 3DB over the strongest baseline NLF is 83.8% (67/80), and reaches 100% (80/80) over SMPLest-X.

83.8%vs NLF 胜率（最强基线）Win rate vs NLF (strongest baseline)

100%vs SMPLest-X 胜率Win rate vs SMPLest-X

20,000+总用户评价次数Total user ratings collected

6pairwise 对比方法数量Methods compared pairwise

分类别分析Per-category analysis

分类别定量分析 — 表 5：SA-1B Hard 数据集上 2D 分类别性能分析。3DB 在遮挡（Occlusion）、截断（Truncation）、复杂姿态（Contortion, Splits, Inverted body）、俯视角等多个困难类别上均显著优于 CameraHMR 和 PromptHMR。Table 5: per-category 2D performance analysis on the SA-1B Hard dataset. 3DB clearly outperforms CameraHMR and PromptHMR on many difficult categories such as occlusion, truncation, challenging poses (contortion, splits, inverted body) and top-down viewpoints.

消融实验Ablation studies

论文通过多视图网格拟合与单视图拟合的对比、有无提示输入的对比、以及 Leave-one-out 与全数据集训练的对比，验证了数据引擎、MHR 表示、双解码器设计与可提示架构各自对最终性能的贡献。3DB 在"非常困难"（very hard）姿态类别、severe truncation 及 top-down viewpoint 等挑战性场景下的优势尤为明显。By comparing multi-view against single-view mesh fitting, with against without prompt inputs, and leave-one-out against full-dataset training, the paper verifies the respective contributions of the data engine, the MHR representation, the two-way decoder design and the promptable architecture. The advantage of 3DB is most pronounced in challenging settings such as the "very hard" pose category, severe truncation and top-down viewpoints.

04 局限性Limitations

注：本文正文无单独的"Limitations"章节。以下局限性部分来自论文结论及讨论中明确提及的问题（标注为论文表述），部分为从方法设计中合理推断（标注为推断 inferred）。Note: the paper has no dedicated "Limitations" section. The limitations below are partly issues explicitly mentioned in its conclusion and discussion (marked stated in the paper), and partly reasonable inferences from the method design (marked inferred).

RICH 数据集上性能差距（论文表述）Performance gap on the RICH dataset (stated in the paper)

3DB 在 RICH 数据集上的表现略逊于 NLF（NLF 在 RICH 上的 PA-MPJPE 为 28.7，3DB-H 为 31.9），原因是 NLF 将 RICH 纳入训练数据，而 3DB 未使用该数据集。这体现了训练数据覆盖范围对域内评估指标的直接影响。3DB performs slightly worse than NLF on the RICH dataset (PA-MPJPE of 28.7 for NLF versus 31.9 for 3DB-H), because NLF includes RICH in its training data while 3DB does not use that dataset. This illustrates the direct effect of training-data coverage on in-domain evaluation metrics.

FreiHand 手部精度与手部专用模型有差距（论文表述）Hand accuracy on FreiHand still trails hand-specific models (stated in the paper)

在 FreiHand 手部估计基准上，3DB 的 PA-MPVPE 为 6.3（DINOv3 版为 6.2），而手部专用模型 WiLoR 为 5.1。论文明确指出 3DB 是全身模型，在手部精度上与顶级手部专用方法仍有差距，但已达到"与手部专用方法相当（comparable）"的水平。On the FreiHand hand estimation benchmark, the PA-MPVPE of 3DB is 6.3 (6.2 for the DINOv3 variant), while the hand-specific model WiLoR reaches 5.1. The paper states explicitly that 3DB is a full-body model and still trails top hand-specific methods in hand accuracy, although it already reaches a level "comparable" to hand-specific methods.

高质量 3D 标注依赖多视图/合成数据（推断 inferred）High-quality 3D annotation depends on multi-view / synthetic data (inferred)

数据引擎的高质量标注依赖多视图采集系统（100+ 摄像头）和合成数据，这些资源在部署时难以大规模扩展至任意野外场景。单视图网格拟合"由于深度歧义和自然遮挡，标注质量往往较低"（论文原文），限制了纯依赖野外图像的标注质量上限。The high-quality annotation of the data engine relies on multi-view capture systems (100+ cameras) and synthetic data, resources that are hard to scale to arbitrary in-the-wild scenes at deployment time. Single-view mesh fitting "often has lower annotation quality due to depth ambiguity and natural occlusion" (quoting the paper), which caps the annotation quality attainable from in-the-wild images alone.

面部估计未包含（推断 inferred）Face estimation not covered (inferred)

3DB 估计身体、脚部和手部，但不包含面部（FLAME）表情估计。相比 SMPL-X 系列的完整面部参数化，面部表情的整合留待未来工作。3DB estimates body, feet and hands, but does not include facial (FLAME) expression estimation. Compared with the complete face parameterization of the SMPL-X family, integrating facial expression is left to future work.

单图像输入、无时序建模（推断 inferred）Single-image input, no temporal modeling (inferred)

3DB 为单帧模型，不利用视频时序信息。视频方法（WHAM、TRAM、GENMO）在部分指标上具有优势（如 GENMO 在 EMDB 上的 PA-MPJPE 为 39.1，3DB-H 为 31.9，3DB 已优于视频方法；但视频方法在时序平滑性上有天然优势）。3DB is a single-frame model and does not exploit temporal information in video. Video methods (WHAM, TRAM, GENMO) have an edge on some metrics (for instance the PA-MPJPE of GENMO on EMDB is 39.1 versus 31.9 for 3DB-H, so 3DB is already better than video methods; video methods do, however, retain a natural advantage in temporal smoothness).