SceneComplete: 面向机器人操作的开放世界杂乱场景3D补全SceneComplete: Open-World 3D Scene Completion in Cluttered Real World Environments for Robot Manipulation

01 动机Motivation

机器人在日常杂乱环境中操作，需要对三维场景进行精确理解，才能稳定可靠地抓取和放置物体，并避免碰撞。然而现实中往往只能获得单张RGB-D图像，场景中物体相互遮挡，且多为训练数据之外的新颖物体——这是当前三维场景重建方法面临的核心挑战。Robots operating in everyday cluttered environments need an accurate understanding of the 3D scene in order to grasp and place objects stably and reliably and to avoid collisions. In practice, however, often only a single RGB-D image is available, the objects in the scene occlude one another, and most of them are novel objects outside the training data — this is the core challenge facing current 3D scene reconstruction methods.

"Careful robot manipulation in every-day cluttered environments requires an accurate understanding of the 3D scene, in order to grasp and place objects stably and reliably and to avoid colliding with other objects. In general, we must construct such a 3D interpretation of a complex scene based on limited input, such as a single RGB-D image."

先前方法的不足Shortcomings of Prior Methods

现有方法（如 PartialDecomp、OctMAE、ZeroGrasp）在开放世界场景下存在明显局限：要么仅能预测场景级占据值而无法给出精确物体网格，要么依赖有限的物体类别，难以泛化到真实杂乱环境中的新颖物体。SceneComplete 的核心思路是组合（composing）已有的大型预训练视觉模型，而非端到端训练一个新模型，从而天然具备开放词汇泛化能力，并能随基础模型的改进而不断提升。Existing methods (such as PartialDecomp, OctMAE and ZeroGrasp) have clear limitations in open-world settings: they either predict only scene-level occupancy without yielding precise object meshes, or depend on a limited set of object categories and generalize poorly to novel objects in real cluttered environments. The core idea of SceneComplete is to compose existing large pre-trained vision models rather than train a new model end-to-end, which gives open-vocabulary generalization for free and lets the system keep improving as its foundation models improve.

02 方法Method

SceneComplete 流水线由六个顺序模块组成，每个模块均调用独立的预训练大型视觉模型。从单张RGB-D输入出发，逐步完成：物体识别 → 分割 → 图像修复 → Image-to-3D → 尺度估计 → 6D位姿配准，最终输出完整场景的物体网格集合。The SceneComplete pipeline consists of six sequential modules, each invoking an independent pre-trained large vision model. Starting from a single RGB-D input, it proceeds step by step: object recognition → segmentation → image inpainting → Image-to-3D → scale estimation → 6D pose registration, finally outputting the set of object meshes for the complete scene.

03 实验Experiments

在 GraspNet-1B 大型基准数据集上进行定量评估（场景重建质量 + 抓取指标），并在 YCB-V 数据集和真实机器人平台上验证抓取成功率。基线包括 PartialDecomp、OctMAE 和 ZeroGrasp。Quantitative evaluation is carried out on the large-scale GraspNet-1B benchmark (scene reconstruction quality + grasp metrics), and grasp success rates are validated on the YCB-V dataset and on a real robot platform. Baselines include PartialDecomp, OctMAE and ZeroGrasp.

表I：GraspNet-1B 场景重建与抓取质量对比Table I: Scene reconstruction and grasp quality on GraspNet-1B

MIoU: Mean Intersection over Union；CD: Chamfer Distance；MMD-EMD: Modified Maximum Distance using Earth Mover's Distance；GC: Grasp Collision率。SceneComplete 在所有四项指标上均优于所有基线。MIoU: Mean Intersection over Union; CD: Chamfer Distance; MMD-EMD: Modified Maximum Distance using Earth Mover's Distance; GC: Grasp Collision rate. SceneComplete outperforms all baselines on all four metrics.

表II：YCB-V 数据集抓取成功率（Grasp Success Rate）Table II: Grasp success rate on the YCB-V dataset

表III：真实机器人抓取放置成功率Table III: Real-robot pick-and-place success rate

消融实验Ablation Study

图4 展示了修复（inpainting）模块的关键作用：去掉修复步骤后，Image-to-3D 模型只能生成残缺网格；未适配的 BrushNet 会引入伪影，而经过 LoRA 适配的版本能正确恢复遮挡区域，生成完整可见物体图像。消融验证了流水线中每个模块（尤其是修复和尺度估计）对最终重建质量的不可替代性。Figure 4 illustrates the critical role of the inpainting module: with the inpainting step removed, the Image-to-3D model can only produce incomplete meshes; the unadapted BrushNet introduces artifacts, whereas the LoRA-adapted version correctly recovers the occluded regions and produces images of fully visible objects. The ablation confirms that every module in the pipeline — inpainting and scale estimation in particular — is indispensable to the final reconstruction quality.

04 局限性Limitations