SAM 3D: 3Dfy Anything in Images — 海报（中文）

01 动机Motivation

计算机视觉长期依赖多视角几何来恢复 3D 结构，而人类凭借阴影、纹理乃至"熟悉物体"等图像线索便可从单张图片感知深度与形状。现有单视角重建方法（如 Trellis、HunyuanD3-2.1）在隔离合成对象上表现尚可，但在自然场景中面临严重遮挡与杂乱背景时却力不从心——根本原因在于大规模真实图像配对 3D 数据的匮乏。Computer vision has long relied on multi-view geometry to recover 3D structure, whereas humans can perceive depth and shape from a single picture using image cues such as shading, texture, and even recognition of familiar objects. Existing single-view reconstruction methods (e.g. Trellis, HunyuanD3-2.1) do reasonably well on isolated synthetic objects, but fall short in natural scenes with severe occlusion and cluttered backgrounds — the root cause being the scarcity of large-scale real images paired with 3D ground truth.

"A fundamental challenge for learning such models is the lack of data: specifically, natural images paired with 3D ground truth are difficult to obtain at scale."

SAM 3D 输入与输出示例 — **图 1**：SAM 3D 将单张图像转化为可拆分的 3D 场景。上方：输入自然图像；下方：逐对象恢复的高质量 3D 资产，包含完整几何、纹理与相对相机的布局信息，可从任意视角重新渲染。**Figure 1**: SAM 3D turns a single image into a decomposable 3D scene. Top: the input natural image. Bottom: the high-quality 3D assets recovered per object, carrying complete geometry, texture and layout relative to the camera, and re-renderable from any viewpoint.

5:1真实物体人工偏好胜率（vs. 最佳基线）Human preference win rate on real objects (vs. the best baseline)

6:1场景重建人工偏好胜率（vs. 最佳基线）Human preference win rate on scene reconstruction (vs. the best baseline)

1,000SA-3DAO 基准：艺术家手工制作的 3D meshSA-3DAO benchmark: artist-crafted 3D meshes

~3.14MMITL 数据引擎标注的无纹理 mesh 总量Total untextured meshes annotated by the MITL data engine

02 方法Method

SAM 3D 采用双阶段生成架构：Geometry 模型预测粗粒度形状与布局，Texture & Refinement 模型在此基础上补充几何细节与纹理。整体训练遵循"合成预训练 → 半合成中训练 → 真实世界后训练"的 LLM 式多阶段流程，通过人工与模型协作的数据引擎（MITL）突破 3D 数据瓶颈。SAM 3D adopts a two-stage generative architecture: the Geometry model predicts coarse shape and layout, and the Texture & Refinement model then adds geometric detail and texture on top of it. Training as a whole follows an LLM-style multi-stage pipeline of "synthetic pre-training → semi-synthetic mid-training → real-world post-training", breaking through the 3D data bottleneck by means of a human-and-model-in-the-loop data engine (MITL).

SAM 3D 架构图 — **图 2**：SAM 3D 架构。（上）Geometry 模型以 DINOv2 提取裁剪目标及全图特征，通过 1.2B 参数的 Mixture-of-Transformers (MoT) 流匹配 transformer 联合预测粗形状 O 与布局 (R, t, s)；（右）双流信息共享的多模态自注意力机制；（下）Texture & Refinement 模型（600M 参数稀疏 latent flow transformer）在粗体素基础上细化几何并合成纹理。**Figure 2**: the SAM 3D architecture. (Top) The Geometry model uses DINOv2 to extract features from the cropped target and from the full image, and jointly predicts the coarse shape O and the layout (R, t, s) with a 1.2B-parameter Mixture-of-Transformers (MoT) flow matching transformer; (right) the multimodal self-attention that shares information across the two streams; (bottom) the Texture & Refinement model (a 600M-parameter sparse latent flow transformer) refines the geometry and synthesizes texture on top of the coarse voxels.

双流输入编码Dual-stream input encoding

使用 DINOv2 对 裁剪目标图（高分辨率局部细节）与完整场景图（全局上下文与识别线索）分别编码，各自搭配对应二值 mask，产生 4 组 conditioning tokens。可选地接入点云图（LiDAR 或单目深度估计），实现与外部流水线的无缝对接。DINOv2 separately encodes the cropped target image (high-resolution local detail) and the full scene image (global context and recognition cues), each paired with its corresponding binary mask, producing 4 sets of conditioning tokens. A point map (from LiDAR or monocular depth estimation) can optionally be fed in, allowing seamless integration with external pipelines.

多阶段训练流程Multi-stage training pipeline

预训练：在 Iso-3DO（2.7M 合成网格，Objaverse-XL）上训练，共 2.5 万亿 token，建立形状与纹理的基础能力。Pre-training: trained on Iso-3DO (2.7M synthetic meshes, Objaverse-XL) for a total of 2.5 trillion tokens, establishing the basic capability for shape and texture.
中训练：在 RP-3DO（6100 万半合成样本，render-paste 方法）上继续训练 2.7 万亿 token，注入遮挡鲁棒性、mask 跟随与布局估计能力。Mid-training: training continues on RP-3DO (61000000 semi-synthetic samples produced by the render-paste method) for another 2.7 trillion tokens, injecting occlusion robustness, mask following and layout estimation.
后训练 SFT：先在 MITL-3DO 标注数据上微调，再在 Art-3DO（3D 艺术家高质量 mesh）上精调，消除 floater、无底 mesh 等常见缺陷。Post-training SFT: first fine-tuned on the MITL-3DO annotations, then further fine-tuned on Art-3DO (high-quality meshes from 3D artists), removing common defects such as floaters and bottomless meshes.
偏好对齐 DPO：基于人工偏好数据 D+/D− 进行 Direct Preference Optimization，对齐人类审美偏好（对称性、封闭性等）。Preference alignment DPO: Direct Preference Optimization on the human preference data D+/D−, aligning the model with human aesthetic preferences such as symmetry and closedness.
蒸馏：将 Geometry 模型推理所需函数评估次数（NFE）从 25 降至 4，实现亚秒级推理。Distillation: the number of function evaluations (NFE) required for Geometry model inference is reduced from 25 to 4, enabling sub-second inference.

MITL 数据引擎（Model-in-the-Loop）The MITL data engine (Model-in-the-Loop)

数据引擎将标注任务分解为三个子任务：Stage 1 识别目标对象并获取 mask；Stage 2 由标注员从 N=8 个候选 3D mesh 中选出最优者并评分（低质量样本路由至 3D 艺术家）；Stage 3 标注员在点云参考下手动调整物体的平移、旋转与缩放。随着训练迭代，模型自身最终贡献约 80% 的标注数据，形成正向飞轮效应。The data engine decomposes annotation into three subtasks: Stage 1 identifies the target object and obtains its mask; Stage 2 has annotators select and score the best of N=8 candidate 3D meshes (low-quality samples are routed to 3D artists); Stage 3 has annotators manually adjust the object's translation, rotation and scale against a point-cloud reference. As training iterates, the model itself eventually contributes about 80% of the annotation data, creating a virtuous flywheel.

SAM 3D 数据引擎流程 — **图 3 / 图 5（来自论文）**：数据引擎各阶段示例。绿色轮廓标出目标对象，右下角为对应 GT mesh。数据集按来源分为四个象限：Iso-3DO（合成隔离对象）、RP-3DO（render-paste 半合成）、MITL-3DO（真实图像 MITL 标注）和 Art-3DO（3D 艺术家精标）。**Figures 3 / 5 (from the paper)**: examples from each stage of the data engine. Green outlines mark the target object; the bottom right shows the corresponding GT mesh. The dataset is split by source into four quadrants: Iso-3DO (isolated synthetic objects), RP-3DO (render-paste semi-synthetic), MITL-3DO (MITL annotation on real images) and Art-3DO (fine annotation by 3D artists).

03 实验Experiments

评测基准包括：SA-3DAO（1K 艺术家 3D mesh，真实世界场景）；ISO3D（来自 3D Arena，无 GT 几何，使用感知相似度指标）；Aria Digital Twin (ADT)（布局评测）；以及大规模人工偏好测试集（Pref Set，来自 MetaCLIP、SA-1B、LVIS）。对比方法包括 Trellis、HunyuanD3-2.1/2.0、Direct3D-S2、TripoSG、Hi3DGen、MIDI。The evaluation benchmarks are: SA-3DAO (1K artist 3D meshes, real-world scenes); ISO3D (from 3D Arena, no GT geometry, scored with perceptual similarity metrics); Aria Digital Twin (ADT) (layout evaluation); and a large-scale human preference test set (Pref Set, drawn from MetaCLIP, SA-1B and LVIS). The methods compared against include Trellis, HunyuanD3-2.1/2.0, Direct3D-S2, TripoSG, Hi3DGen and MIDI.

3D 形状质量（SA-3DAO 基准，数值越高越好，Chamfer/EMD 越低越好）3D shape quality (SA-3DAO benchmark; higher is better, while lower Chamfer/EMD is better)

方法Method	F1@0.01 ↑	vIoU ↑	Chamfer ↓	EMD ↓
Trellis	0.1475	0.1392	0.0902	0.2131
HY3D-2.1	0.1399	0.1266	0.1126	0.2432
HY3D-2.0	0.1574	0.1504	0.0866	0.2049
Direct3D-S2	0.1513	0.1465	0.0962	0.2160
TripoSG	0.1533	0.1445	0.0844	0.2057
Hi3DGen	0.1629	0.1531	0.0937	0.2134
SAM 3D（本文）SAM 3D (ours)	0.2344	0.2311	0.0400	0.1211

3D 布局（SA-3DAO & Aria Digital Twin，联合生成）3D layout (SA-3DAO & Aria Digital Twin, joint generation)

方法Method	SA-3DAO 3D IoU ↑	SA-3DAO ADD-S@0.1 ↑	ADT 3D IoU ↑	ADT ADD-S@0.1 ↑
SAM 3D + FoundationPose（流水线）SAM 3D + FoundationPose (pipeline)	0.2837	0.5079	0.3661	0.6495
MIDI（联合生成）MIDI (joint generation)	—	—	0.0336	0.0175
SAM 3D（联合生成）SAM 3D (joint generation)	0.4254	0.7232	0.4970	0.7673

单物体质量对比 — **图 6**：与主流 image-to-3D 方法的定性对比（SA-3DAO 评测集）。从左至右：Trellis、HunyuanD3-2.1、Direct3D-S2、Hi3DGen、SAM 3D（本文），最右为 3D 艺术家 GT mesh。SAM 3D 在严重遮挡与复杂纹理下展现出更强的泛化能力。**Figure 6**: qualitative comparison with mainstream image-to-3D methods (SA-3DAO evaluation set). From left to right: Trellis, HunyuanD3-2.1, Direct3D-S2, Hi3DGen, SAM 3D (ours), with the 3D artist GT mesh on the far right. SAM 3D shows stronger generalization under severe occlusion and complex texture.

场景重建对比与偏好结果 — **图 7 / 图 8**：场景重建定性对比（上），以及对象级与场景级人工偏好测试结果（下）。SAM 3D 在所有维度上显著优于对比方法：场景重建胜率 6:1，物体级胜率 5:1。**Figures 7 / 8**: qualitative comparison of scene reconstruction (top), together with object-level and scene-level human preference results (bottom). SAM 3D significantly outperforms the compared methods along every dimension: a 6:1 win rate on scene reconstruction and 5:1 at the object level.

消融分析：多阶段训练的累积增益Ablation: the cumulative gains of multi-stage training

训练阶段Training stage	F1@0.01 ↑	vIoU ↑	Chamfer ↓	EMD ↓
预训练（Iso-3DO）Pre-training (Iso-3DO)	0.1349	0.1202	0.1036	0.2396
+ 中训练（RP-3DO）+ mid-training (RP-3DO)	0.1705	0.1683	0.0760	0.1821
+ SFT（MITL-3DO）+ SFT (MITL-3DO)	0.2027	0.2025	0.0578	0.1510
+ DPO（MITL-3DO）+ DPO (MITL-3DO)	0.2156	0.2156	0.0498	0.1367
+ SFT（Art-3DO）+ SFT (Art-3DO)	0.2331	0.2337	0.0445	0.1257
最终模型（+ DPO Art-3DO）Final model (+ DPO Art-3DO)	0.2344	0.2311	0.0400	0.1211

消融实验表明，每个训练阶段均带来近单调的 3D 形状改进，充分验证了多阶段训练设计的有效性。数据引擎迭代运行越久，Elo 分数近线性提升（每 3 周一个 checkpoint，Elo 差 400 分对应 10:1 胜率）。The ablation shows that every training stage brings a near-monotonic improvement in 3D shape, fully validating the effectiveness of the multi-stage training design. The longer the data engine keeps iterating, the more nearly linearly the Elo score rises (one checkpoint every 3 weeks; an Elo gap of 400 points corresponds to a 10:1 win rate).

纹理偏好比较 — **图 9**：纹理质量人工偏好对比。固定使用 SAM 3D 输出的几何形状，仅对比各方法生成的纹理。SAM 3D 的纹理质量同样显著优于其他方法。**Figure 9**: human preference comparison on texture quality. The geometry is fixed to the output of SAM 3D and only the textures generated by each method are compared. The texture quality of SAM 3D is likewise significantly better than that of the other methods.

04 局限性Limitations

说明：以下局限性均来自论文附录 F"Limitations"章节，为作者明确陈述（stated）。Note: all the limitations below come from Appendix F "Limitations" of the paper and are explicitly stated by the authors.

分辨率上限导致细节失真The resolution ceiling distorts fine detail

Geometry 模型使用粗形状分辨率 O ∈ ℝ^64³，Gaussian splat 解码器最多 32 splats/voxel。对于复杂形状（如人体的手部、面部），整体尺度所能分配的 voxel/splat 数量有限，而人类视觉对此类局部特征极为敏感，因此会出现可感知的形变或细节丢失。论文指出：当单独重建手部或头部时，SAM 3D 表现明显更好。解决方向包括提升输出分辨率、超分辨率模型、基于部件的生成，或切换到隐式 3D 表示。The Geometry model uses a coarse shape resolution of O ∈ ℝ^64³, and the Gaussian splat decoder allows at most 32 splats/voxel. For complex shapes (such as human hands and faces), the overall scale leaves only a limited number of voxels/splats for those regions, while human vision is extremely sensitive to exactly such local features, so perceptible deformation or loss of detail appears. The paper notes that SAM 3D performs markedly better when a hand or a head is reconstructed on its own. Directions for a fix include raising the output resolution, super-resolution models, part-based generation, or switching to an implicit 3D representation.

对象布局的独立预测缺乏物理推理Independent per-object layout prediction lacks physical reasoning

SAM 3D 一次预测一个对象，未经训练以推理多物体间的物理交互，如接触关系、物理稳定性、穿透检测或共面对齐（同一地平面）。多物体联合预测并加入相应约束损失将是下一步工作。SAM 3D predicts one object at a time and is not trained to reason about physical interactions between multiple objects, such as contact relations, physical stability, interpenetration detection or coplanar alignment (a shared ground plane). Predicting multiple objects jointly with the corresponding constraint losses would be the next step.

纹理预测与姿态解耦导致旋转对称歧义Decoupling texture prediction from pose causes rotational-symmetry ambiguity

纹理预测在不知晓预测对象姿态的情况下进行。对于具有旋转对称性的物体，模型偶尔会预测出实际上将物体旋转到错误朝向的纹理。Texture prediction is carried out without knowledge of the predicted object pose. For objects with rotational symmetry, the model occasionally predicts a texture that effectively rotates the object into the wrong orientation.