SyncHuman: 同步 2D 与 3D 生成模型的单视图人体重建

01 动机Motivation

从单张图像重建穿衣人体的三维模型，是计算机视觉中的核心难题。现有方法面临两难困境：多视图 2D 生成模型（如 PSHuman）能捕捉精细纹理细节，但生成的多视图图像与三维结构缺乏一致性；而 3D 原生生成模型（如 Trellis）能生成连贯的几何形状，却缺乏精细细节，且严重依赖不准确的 SMPL 姿态估计。Reconstructing a 3D model of a clothed human from a single image is a core challenge in computer vision. Existing methods face a dilemma: multi-view 2D generative models (such as PSHuman) capture fine texture details, but the generated multi-view images lack consistency with the 3D structure; whereas 3D native generative models (such as Trellis) produce coherent geometry yet lack fine details and depend heavily on inaccurate SMPL pose estimation.

"我们提出了一种新型框架，联合利用多视图和 3D 原生生成模型的优势，同时克服各自的局限性。""We propose a novel framework that jointly exploits the strengths of multi-view and 3D native generative models while overcoming the limitations of each."

SyncHuman Teaser — 与基线方法几何质量对比 — **图 1：SyncHuman 与基线方法的几何质量对比。**左起依次为 SMPL 拟合结果、PSHuman（多视图 2D 方法）、Trellis（3D 原生方法）以及本文方法 SyncHuman。可见 SyncHuman 在几何精度和纹理细节上均显著优于两类基线，且不依赖 SMPL 姿态先验。**Figure 1: Geometric quality comparison between SyncHuman and baseline methods.**From left to right: the SMPL fitting result, PSHuman (a multi-view 2D method), Trellis (a 3D native method), and our method SyncHuman. SyncHuman is clearly superior to both classes of baselines in geometric accuracy and texture detail, and does not rely on an SMPL pose prior.

0.8353Chamfer Distance ↓
（X-Humans，越低越好）Chamfer Distance ↓
(X-Humans, lower is better)

21.84PSNR ↑
（X-Humans，越高越好）PSNR ↑
(X-Humans, higher is better)

0.8741SSIM ↑
（X-Humans）SSIM ↑
(X-Humans)

0.0786LPIPS ↓
（X-Humans，越低越好）LPIPS ↓
(X-Humans, lower is better)

多视图 2D 方法的问题Problems with Multi-view 2D Methods

多视图图像间三维结构不一致Multi-view images are inconsistent in 3D structure
从不一致视图重建导致几何误差累积Reconstruction from inconsistent views leads to accumulated geometric errors
依赖扩散模型的随机性，无法保证跨视图一致性Reliance on the stochasticity of diffusion models gives no guarantee of cross-view consistency

3D 原生方法的问题Problems with 3D Native Methods

严重依赖 SMPL 姿态估计，误差传播明显Heavy reliance on SMPL pose estimation, with pronounced error propagation
生成形状细节粗糙，纹理质量差Generated shapes have coarse details and poor texture quality
复杂服装结构难以保真重建Complex garment structures are hard to reconstruct faithfully

02 方法Method

SyncHuman 由两个核心模块构成：2D-3D 跨空间生成模型（通过双向同步注意力联合微调多视图与 3D 生成器）和多视图引导解码器（MVGD）（将 DINOv2 特征从生成的多视图图像注入 3D 解码过程），二者共同实现从单张图像到高保真三维人体的重建。SyncHuman consists of two core modules: a 2D-3D cross-space generative model (which jointly fine-tunes the multi-view and 3D generators through bidirectional synchronization attention) and a multi-view guided decoder (MVGD) (which injects DINOv2 features from the generated multi-view images into the 3D decoding process); together they reconstruct a high-fidelity 3D human from a single image.

**图 2：SyncHuman 整体框架。**输入单张图像，首先通过 2D-3D 跨空间生成模型联合生成多视图图像和 3D 表示；随后，多视图引导解码器（MVGD）提取 DINOv2 视觉特征，通过特征注入机制增强三维几何细节与纹理质量，最终输出高保真着装人体网格。**Figure 2: Overall framework of SyncHuman.**Given a single input image, the 2D-3D cross-space generative model first jointly generates multi-view images and a 3D representation; the multi-view guided decoder (MVGD) then extracts DINOv2 visual features and, through a feature injection mechanism, enhances 3D geometric detail and texture quality, finally outputting a high-fidelity clothed human mesh.

核心组件一：2D-3D 跨空间同步注意力Key Component 1: 2D-3D Cross-space Synchronization Attention

在扩散去噪过程中，本文设计了双向的像素对齐同步注意力，使 2D 多视图分支与 3D 体素分支在每个时间步互相感知：During diffusion denoising, the paper designs a bidirectional pixel-aligned synchronization attention that lets the 2D multi-view branch and the 3D voxel branch perceive each other at every timestep:

2D→3D 方向：每个 3D 体素向量在四个正交视图的对应 2D 特征图上查询像素对齐特征，获取外观细节指导。2D→3D direction: Each 3D voxel vector queries pixel-aligned features on the corresponding 2D feature maps of the four orthogonal views to obtain appearance detail guidance.
3D→2D 方向：每个 2D 像素沿投影光线查询对应的 3D 体素列特征，获取几何结构约束。3D→2D direction: Each 2D pixel queries the corresponding column of 3D voxel features along its projection ray to obtain geometric structure constraints.

训练采用结合 2D 和 3D 分支的 Flow Matching 损失。整体框架无需 SMPL 作为条件，仅以输入图像的 DINOv2 特征为条件进行生成。Training uses a Flow Matching loss that combines the 2D and 3D branches. The overall framework needs no SMPL condition and generates conditioned only on the DINOv2 features of the input image.

双向同步注意力机制示意图 — **图 3：像素对齐双向同步注意力机制。**左：2D→3D 注意力，3D 体素查询对应视图的 2D 特征；右：3D→2D 注意力，2D 像素查询沿投影光线的 3D 体素特征。双向交互使两个分支在扩散过程中实时对齐，确保多视图图像与三维形状的一致性。**Figure 3: Pixel-aligned bidirectional synchronization attention.**Left: 2D→3D attention, where 3D voxels query the 2D features of the corresponding views; right: 3D→2D attention, where 2D pixels query the 3D voxel features along the projection ray. The bidirectional interaction aligns the two branches in real time during diffusion, ensuring consistency between the multi-view images and the 3D shape.

核心组件二：多视图引导解码器（MVGD）Key Component 2: Multi-view Guided Decoder (MVGD)

在生成阶段结束后，MVGD 从生成的多视图彩色图和法线图中提取 DINOv2 特征，通过拼接（concatenation）和 MLP 层注入 FlexiCubes 解码器，将精细的视觉细节从 2D 图像域提升到三维几何域，显著提升表面法线一致性和纹理保真度。After the generation stage, MVGD extracts DINOv2 features from the generated multi-view color and normal maps and injects them into the FlexiCubes decoder through concatenation and MLP layers, lifting fine visual details from the 2D image domain into the 3D geometry domain and markedly improving surface normal consistency and texture fidelity.

训练设置Training Setup

训练数据集：THuman2.1、CustomHumans、THuman3.0、2K2K（约 5,000 人体扫描）Training datasets: THuman2.1, CustomHumans, THuman3.0 and 2K2K (about 5,000 human scans)
2D-3D 模型：8× H800 GPU，batch=32，共 50,000 次迭代2D-3D model: 8× H800 GPU, batch=32, 50,000 iterations in total
解码器：1× H800 GPU，batch=4，共 14,000 次迭代Decoder: 1× H800 GPU, batch=4, 14,000 iterations in total
多视图分支：微调自 SD 2.1，batch=32，共 30,000 次迭代Multi-view branch: fine-tuned from SD 2.1, batch=32, 30,000 iterations in total

推理速度Inference Speed

SyncHuman：38.57 秒SyncHuman: 38.57 s
PSHuman：52.98 秒PSHuman: 52.98 s
Trellis：15.68 秒Trellis: 15.68 s
在单张 H800 GPU 上测量；SyncHuman 比 PSHuman 快约 27%，但比 Trellis 慢约 2.5×。Measured on a single H800 GPU; SyncHuman is about 27% faster than PSHuman, but about 2.5× slower than Trellis.

03 实验Experiments

在 CAPE-NFP、CAPE-FP 和 X-Humans 三个测试集上（共 250 个扫描样本）评估几何精度（Chamfer Distance、P2S、Normal Consistency）和渲染质量（PSNR、SSIM、LPIPS）指标，与 ICON、ECON、GTA、SIFU、SiTH、Human3Diff、PSHuman、Trellis 等基线比较。Geometric accuracy (Chamfer Distance, P2S, Normal Consistency) and rendering quality (PSNR, SSIM, LPIPS) are evaluated on the three test sets CAPE-NFP, CAPE-FP and X-Humans (250 scan samples in total), and compared with baselines such as ICON, ECON, GTA, SIFU, SiTH, Human3Diff, PSHuman and Trellis.

Table 1：定量对比结果（X-Humans 测试集）Table 1: Quantitative comparison (X-Humans test set)

方法Method	Cham. Dist ↓	P2S ↓	NC ↑	PSNR ↑	SSIM ↑	LPIPS ↓
ICON	1.4971	1.3920	0.8133	—	—	—
ECON	1.6425	1.4398	0.8054	—	—	—
GTA	1.5050	1.4662	0.8044	20.0084	0.8502	0.1129
SIFU	1.5391	1.4331	0.8093	20.6747	0.8455	0.1104
SiTH	1.5104	1.4345	0.7972	19.8245	0.8204	0.1182
Human3Diff	1.5034	1.4219	0.7468	19.7181	0.8065	0.1334
PSHuman	1.4377	1.1385	0.8393	20.8405	0.8523	0.0980
Trellis	2.0043	1.5053	0.7718	17.0786	0.7238	0.1529
SyncHuman（Ours）SyncHuman (Ours)	0.8353	0.7593	0.8872	21.8385	0.8741	0.0786

**图 4：几何质量定性对比。**SyncHuman 重建的人体网格在服装细节（褶皱、领口、袖口）和整体形状准确性上均优于 PSHuman 和 Trellis，特别是在复杂姿态下表现尤为突出。**Figure 4: Qualitative comparison of geometric quality.**The human meshes reconstructed by SyncHuman outperform PSHuman and Trellis in garment details (folds, collars, cuffs) and in overall shape accuracy, and are particularly strong under complex poses.

外观质量定性对比 — **图 5：外观（纹理）质量定性对比。**SyncHuman 生成的渲染图像在色彩保真度、纹理清晰度方面均超越基线方法，LPIPS 指标提升最为明显（0.0786 vs PSHuman 0.0980）。**Figure 5: Qualitative comparison of appearance (texture) quality.**The renderings generated by SyncHuman surpass the baseline methods in color fidelity and texture sharpness, with the most pronounced gain on the LPIPS metric (0.0786 vs PSHuman 0.0980).

消融实验Ablation Studies

Table 2 验证了 2D-3D 跨空间同步模型的有效性：单独使用 Trellis（PSNR 17.079）或 PSHuman（PSNR 20.840），加入双向同步注意力后提升至 PSNR 21.838，Chamfer Distance 从 1.438 降至 0.835。Table 3 验证 MVGD 的贡献：相比原始解码器（微调后 PSNR 21.362），MVGD 将 PSNR 进一步提升至 21.838，并将 Chamfer Distance 从 0.887 降至 0.835。Table 2 verifies the effectiveness of the 2D-3D cross-space synchronization model: starting from Trellis alone (PSNR 17.079) or PSHuman alone (PSNR 20.840), adding bidirectional synchronization attention raises PSNR to 21.838 and lowers Chamfer Distance from 1.438 to 0.835. Table 3 verifies the contribution of MVGD: compared with the original decoder (PSNR 21.362 after fine-tuning), MVGD further raises PSNR to 21.838 and lowers Chamfer Distance from 0.887 to 0.835.

消融配置Ablation Setting	PSNR ↑	SSIM ↑	LPIPS ↓	Cham. ↓	P2S ↓
Trellis（原始）Trellis (original)	17.079	0.724	0.153	2.004	1.505
Trellis（微调）Trellis (fine-tuned)	20.344	0.844	0.101	1.135	1.041
PSHuman	20.840	0.852	0.098	1.438	1.138
原始解码器（微调）Original decoder (fine-tuned)	21.362	0.866	0.090	0.887	0.810
完整 SyncHumanFull SyncHuman	21.838	0.874	0.079	0.835	0.759

04 局限性Limitations

说明：前两条局限性为论文 Limitations 章节的明确陈述（stated）；后两条为根据方法设计推断（inferred）。Note: The first two limitations are explicitly stated in the paper's Limitations section; the latter two are inferred from the method design.

极端光照条件下的伪影 — **图 6（局限性示例）：**训练数据使用均匀光照渲染，导致在极端光照条件下重建纹理出现伪影。**Figure 6 (limitation example):** The training data is rendered under uniform lighting, so the reconstructed texture shows artifacts under extreme lighting conditions.

光照偏差（stated）：训练数据均匀光照导致极端光照下出现伪影Lighting bias (stated): uniform lighting in the training data causes artifacts under extreme lighting

论文明确指出："由于训练数据集使用均匀光照渲染，重建纹理在极端光照条件下可能出现伪影。"该问题源于训练集（THuman2.1 等）的渲染设置，导致模型对非均匀光照场景泛化能力有限。The paper states explicitly: "Since the training dataset is rendered with uniform lighting, the reconstructed texture may show artifacts under extreme lighting conditions." The problem originates in the rendering setup of the training sets (THuman2.1 and others), which limits the model's generalization to non-uniformly lit scenes.

训练数据规模有限（stated）：约 5,000 人体扫描，限制生成质量Limited training data scale (stated): about 5,000 human scans, which caps generation quality

论文明确指出："多视图生成模型仅从约 5,000 个人体扫描中微调自 SD 2.1，因此生成质量仍受到限制。"与通用扩散模型的数十亿图像训练相比，人体领域特定数据的稀缺制约了模型的泛化上限。The paper states explicitly: "The multi-view generative model is fine-tuned from SD 2.1 on only about 5,000 human scans, so the generation quality is still limited." Compared with the billions of images used to train general diffusion models, the scarcity of human-domain-specific data constrains the model's generalization ceiling.

网格表面孔洞（inferred）：FlexiCubes 无水密约束导致网格不完整Holes in the mesh surface (inferred): FlexiCubes enforces no watertight constraint, so meshes are incomplete

方法使用 FlexiCubes 进行网格提取，该方法不强制水密（watertight）约束，可能导致生成网格出现表面孔洞，影响下游应用（如物理模拟、3D 打印）的适用性。The method uses FlexiCubes for mesh extraction, which does not enforce a watertight constraint, so the generated mesh may contain surface holes, affecting its suitability for downstream applications such as physical simulation and 3D printing.

推理速度劣于纯 3D 方法（inferred）：38.57 秒 vs Trellis 15.68 秒Inference slower than pure 3D methods (inferred): 38.57 s vs Trellis 15.68 s

由于需要同时运行 2D 多视图分支和 3D 体素分支，并加入双向注意力交互，SyncHuman 的推理时间（38.57 秒/张）约为 Trellis（15.68 秒/张）的 2.5 倍，在实时或高吞吐量场景下存在明显瓶颈。Because the 2D multi-view branch and the 3D voxel branch must run at the same time with bidirectional attention interaction, the inference time of SyncHuman (38.57 s per image) is about 2.5 times that of Trellis (15.68 s per image), a clear bottleneck in real-time or high-throughput settings.