计算机视觉 · 3D Panoptic SegmentationComputer Vision · 3D Panoptic Segmentation
UP-Fuse: Uncertainty-guided LiDAR-Camera Fusion for 3D Panoptic Segmentation
用预测的不确定性动态调制跨模态融合，在相机失效、标定漂移下仍不"退化到比纯 LiDAR 还差"Dynamically modulating cross-modal fusion with predicted uncertainty, so that under camera failure and calibration drift the model no longer "degrades below a pure LiDAR model"
Rohit Mohan, Florian Drews, Yakov Miron, Daniele Cattaneo, Abhinav Valada · University of Freiburg & Robert Bosch GmbH
LiDAR-camera 融合通常能提升 3D 全景分割精度，但一旦相机出现遮挡、过曝、标定漂移等失效，融合模型的表现反而可能跌破纯 LiDAR baseline。UP-Fuse 在 range-view 空间引入一个不确定性引导的融合模块：先学习量化相机特征在各种视觉退化下的"表征漂移量"，再用这个不确定性动态削弱不可靠的视觉线索，同时用 hybrid 2D-3D transformer 解码器直接输出 3D 全景分割结果。LiDAR-camera fusion usually improves 3D panoptic segmentation accuracy, but once the camera suffers occlusion, overexposure, calibration drift or other failures, the fused model may instead fall below the LiDAR-only baseline. UP-Fuse introduces an uncertainty-guided fusion module in the range-view space: it first learns to quantify the "representation drift" of camera features under various visual degradations, then uses this uncertainty to dynamically attenuate unreliable visual cues, while a hybrid 2D-3D transformer decoder directly outputs the 3D panoptic segmentation result.
Panoptic nuScenes / SemanticKITTI / Panoptic Waymo（自建）Panoptic nuScenes / SemanticKITTI / Panoptic Waymo (self-built)PQ 80.7（nuScenes val, LiDAR+Camera）PQ 80.7 (nuScenes val, LiDAR+Camera)相机失效场景仅降 1.2 PQ（对比 baseline 降 4.2–5.0）Only 1.2 PQ drop under camera failure (vs. 4.2–5.0 for baselines)📄 arXiv:2602.19349Project page
LiDAR 提供精确几何但稀疏、缺纹理；相机提供稠密纹理但在恶劣条件下不可靠。现有融合方法只学"该融合哪些特征"（relevance），却不学"这些特征此刻是否可信"（reliability），因此一旦相机传感器退化或失效，融合模型会"sharply degrade"，甚至比纯 LiDAR 基线还差。LiDAR provides precise geometry but is sparse and lacks texture; cameras provide dense texture but are unreliable under adverse conditions. Existing fusion methods only learn "which features should be fused" (relevance), but not "whether those features are trustworthy at this moment" (reliability); therefore, once the camera sensor degrades or fails, the fused model will "sharply degrade", ending up even worse than the LiDAR-only baseline.
"A crucial gap lies in developing a fusion paradigm that can discern not only which features are relevant but also whether they are valid, enabling adaptive and context-dependent integration across modalities."
Figure 1：绿色为正确预测、红色为错误。LiDAR-camera 融合相对 LiDAR-only 显著提升分割效果，能检测出被漏检的车辆；但在相机传感器失效场景下，融合方法的性能反而跌破 LiDAR-only 基线，且漏检此前已识别出的物体——这凸显了多模态感知中"relevance"与"reliability"缺一不可。Figure 1: green marks correct predictions, red marks errors. LiDAR-camera fusion markedly improves segmentation over LiDAR-only and detects vehicles that were missed; but under camera sensor failure the fusion method instead falls below the LiDAR-only baseline and misses objects it had previously recognized — highlighting that in multimodal perception "relevance" and "reliability" are both indispensable.
80.7Panoptic nuScenes val PQ（LC，本文方法）Panoptic nuScenes val PQ (LC, ours)
-1.2相机失效（L*）下 UP-Fuse 的 ΔPQ，对比 baseline -4.2 ~ -5.0ΔPQ of UP-Fuse under camera failure (L*), vs. -4.2 ~ -5.0 for baselines
5.7FPS（LC），高于对比的 fusion baselines 0.9–2.5 FPSFPS (LC), higher than the compared fusion baselines at 0.9–2.5 FPS
+0.1day→night 域偏移下 ΔPQ（唯一不掉点的融合方法）ΔPQ under the day→night domain shift (the only fusion method that does not drop)
02 Method
UP-Fuse 先把 LiDAR 点云和多视角相机图像投影到统一的 2D range-view 空间做像素对齐的特征提取；随后由 Uncertainty-Aware Fusion Module 根据"跨模态相关性"与"视觉可靠性"两个维度自适应融合；最后由 Hybrid 2D-3D Panoptic Decoder 直接在原始点云上生成 3D 全景分割结果，同时缓解 2D 投影带来的空间歧义与 360° 环绕不连续问题。UP-Fuse first projects the LiDAR point cloud and the multi-view camera images into a unified 2D range-view space for pixel-aligned feature extraction; the Uncertainty-Aware Fusion Module then fuses them adaptively along two dimensions, "cross-modal relevance" and "visual reliability"; finally the Hybrid 2D-3D Panoptic Decoder generates the 3D panoptic segmentation directly on the raw point cloud, while mitigating the spatial ambiguity caused by 2D projection and the 360° wrap-around discontinuity.
Figure 2：LiDAR 和多视角相机图像被融合到统一的 range-view 特征空间。Uncertainty-Aware Fusion Module 通过不确定性加权的可变形跨模态交互（deformable cross-modal interaction）自适应整合两种模态、抑制不可靠视觉线索；随后 Hybrid 2D-3D Panoptic Decoder 生成 3D 预测（图中棕色路径/模块仅在训练阶段使用）。Figure 2: LiDAR and multi-view camera images are fused into a unified range-view feature space. The Uncertainty-Aware Fusion Module adaptively integrates the two modalities through uncertainty-weighted deformable cross-modal interaction and suppresses unreliable visual cues; the Hybrid 2D-3D Panoptic Decoder then generates the 3D prediction (the brown paths/modules in the figure are used only during training).
不确定性量化（Uncertainty Quantification of Camera Features）Uncertainty Quantification of Camera Features
核心假设是：把相机特征的 aleatoric uncertainty 建模为"在输入退化下的不稳定性"。训练时对每张原图 Iorig 采样一个退化版本 Iaug（亮度/对比度扰动、传感器 dropout、以及与 Cityscapes/COCO/Dark Zurich 做 histogram matching 引入的跨域光度变化），以 0.5 概率保留 Iaug=Iorig 作为零不确定性样本。用两者在 frozen encoder 下的特征 L2 距离作为 ground-truth 不稳定性 dgt，训练一个轻量 3 层 MLP 用 Huber loss 从退化特征回归预测这个距离 dpred，再通过 U = 1 − exp(−dpred) 映射成 [0,1] 的概率化不确定性分数。The core assumption is to model the aleatoric uncertainty of camera features as "instability under input degradation". During training, for every original image Iorig a degraded version Iaug is sampled (brightness/contrast perturbation, sensor dropout, and cross-domain photometric variation introduced by histogram matching with Cityscapes/COCO/Dark Zurich), and with probability 0.5 Iaug=Iorig is kept as a zero-uncertainty sample. The L2 distance between the two feature maps under a frozen encoder serves as the ground-truth instability dgt; a lightweight 3-layer MLP is trained with a Huber loss to regress this distance dpred from the degraded features, and U = 1 − exp(−dpred) then maps it to a probabilistic uncertainty score in [0,1].
在每个尺度 s∈{4,8,16,32}，融合通过以 LiDAR query 为条件的可变形注意力（deformable attention）聚合空间对齐的视觉特征，预测的不确定性分数动态调制该跨模态交互的权重，从而在训练和推理阶段都能让网络"衰减不可靠线索、保留可靠信息"，而不是对所有相机特征一视同仁地融合。At every scale s∈{4,8,16,32}, fusion aggregates spatially aligned visual features through deformable attention conditioned on LiDAR queries, and the predicted uncertainty score dynamically modulates the weight of this cross-modal interaction, so that at both training and inference time the network can "attenuate unreliable cues and retain reliable information" instead of fusing all camera features indiscriminately.
03 Experiments
在 Panoptic nuScenes、SemanticKITTI 上评测，并因 Waymo Open Dataset 原生不提供全景标注，作者基于其公开的语义分割与 3D bounding box 标签自建了 Panoptic Waymo 新基准（含多个 baseline）。主指标为 PQ / PQ† / SQ / RQ（Panoptic Quality 及其变体）。Evaluated on Panoptic nuScenes and SemanticKITTI; and since the Waymo Open Dataset does not natively provide panoptic annotations, the authors built a new Panoptic Waymo benchmark (with several baselines) from its public semantic segmentation and 3D bounding box labels. The main metrics are PQ / PQ† / SQ / RQ (Panoptic Quality and its variants).
Benchmark
Modality
最强对比方法Strongest baseline
UP-Fuse
Panoptic nuScenes val
LC
IAL-PieAug 82.3 PQ
80.7 PQ（FPS 5.7 vs 0.9）80.7 PQ (FPS 5.7 vs 0.9)
Panoptic nuScenes val
L
IAL 77.0 PQ
74.9 PQ
相机失效（L*）Camera failure (L*)
LC→L*
IAL ΔPQ −4.6
ΔPQ −1.2（最小掉点）ΔPQ −1.2 (smallest drop)
day→night 域偏移day→night domain shift
LC
IAL ΔPQ −2.1
ΔPQ +0.1（唯一提升）ΔPQ +0.1 (the only gain)
需要如实指出：在 Panoptic nuScenes val 标准（无干扰）条件下，IAL-PieAug（82.3 PQ）和 IAL（80.3 PQ）在纯精度上优于 UP-Fuse（80.7 PQ 时略高于 IAL，但低于 IAL-PieAug）——UP-Fuse 的核心优势不在"干净条件下的极限精度"，而在鲁棒性与效率的综合表现：在标定漂移（0°–5° 旋转误差）下，论文报告 "UP-Fuse (red) outperforms all baselines, dropping only 4.4% in PQ compared to >8% for s..."（其余基线掉点普遍高于 UP-Fuse 的两倍）。To state it faithfully: under the standard (undisturbed) Panoptic nuScenes val setting, IAL-PieAug (82.3 PQ) and IAL (80.3 PQ) beat UP-Fuse on raw accuracy (at 80.7 PQ it is slightly above IAL but below IAL-PieAug) — the core advantage of UP-Fuse is not "peak accuracy under clean conditions" but its combined robustness and efficiency: under calibration drift (0°–5° rotation error) the paper reports "UP-Fuse (red) outperforms all baselines, dropping only 4.4% in PQ compared to >8% for s..." (the remaining baselines generally drop more than twice as much as UP-Fuse).
Figure 4：在 0°–5° 的 LiDAR-camera 标定旋转误差范围内，UP-Fuse（红色曲线）始终优于所有基线，PQ 仅下降 4.4%，而其余方法下降幅度超过 8%。Figure 4: across the 0°–5° range of LiDAR-camera calibration rotation error, UP-Fuse (red curve) consistently outperforms all baselines, with PQ dropping only 4.4%, while the other methods drop by more than 8%.Figure 10：在自建的 Panoptic Waymo val 集上，UP-Fuse 与 baseline LCPS 架构的定性对比（红色为错误预测，蓝色为正确预测）——UP-Fuse 在这一新基准上同样展现出更少的错误区域。Figure 10: qualitative comparison between UP-Fuse and the baseline LCPS architecture on the self-built Panoptic Waymo val set (red marks wrong predictions, blue marks correct ones) — UP-Fuse likewise exhibits fewer error regions on this new benchmark.
Ablations
消融实验（Table VII，nuScenes val）逐步验证各组件的增益：从纯 LiDAR 的 Hybrid 2D-3D Decoder（M2，74.9 PQ）加上简单的 concat 融合先提升到 77.2 PQ；换成 Deformable Cross-Modal Interaction（D-CMI）后进一步提升到 79.6 PQ；单独加入传感器退化增强（Sensor Degradation Augs.）反而短暂掉到 78.9 PQ；只有把不确定性预测和 D-CMI 结合成 Uncertainty-Aware D-CMI 后才达到完整的 80.7 PQ——说明"知道该融合什么"和"知道该信任什么"两者缺一不可，必须联合建模才能兑现最终增益。The ablation study (Table VII, nuScenes val) validates the gain of each component step by step: starting from the LiDAR-only Hybrid 2D-3D Decoder (M2, 74.9 PQ), adding plain concat fusion first raises it to 77.2 PQ; replacing that with Deformable Cross-Modal Interaction (D-CMI) further raises it to 79.6 PQ; adding the sensor degradation augmentation (Sensor Degradation Augs.) on its own instead drops it briefly to 78.9 PQ; only after combining uncertainty prediction and D-CMI into Uncertainty-Aware D-CMI does it reach the full 80.7 PQ — showing that "knowing what to fuse" and "knowing what to trust" are both indispensable and must be modeled jointly for the final gain to materialize.
04 Limitations
Note:论文正文 Conclusion 部分明确给出了 "Limitations and Future Work" 段落，以下第一条为作者原文陈述（stated）；第二、三条为根据实验结果与设计推断（inferred）。the Conclusion of the paper explicitly contains a "Limitations and Future Work" paragraph; the first item below is stated verbatim by the authors (stated), while the second and third are inferred from the experimental results and the design (inferred).
依赖固定相机内外参，不显式校正标定误差（作者陈述）Relies on fixed camera intrinsics/extrinsics and does not explicitly correct calibration error (stated by the authors)
原文："A limitation of our framework is its reliance on fixed camera parameters for view transformation. While our uncertainty module mitigates calibration drift by suppressing the visual stream, it does not explicitly correct the extrinsics. Consequently, in scenarios with severe misalignment, performance is bounded by the LiDAR-only baseline." 也就是说，不确定性模块只能"关闭"不可靠的视觉分支，无法主动修正标定漂移本身；严重失准时融合收益的上限就是纯 LiDAR 基线。作者将联合外参精化列为未来工作方向。From the paper: "A limitation of our framework is its reliance on fixed camera parameters for view transformation. While our uncertainty module mitigates calibration drift by suppressing the visual stream, it does not explicitly correct the extrinsics. Consequently, in scenarios with severe misalignment, performance is bounded by the LiDAR-only baseline." In other words, the uncertainty module can only "switch off" the unreliable visual branch and cannot actively correct the calibration drift itself; under severe misalignment the upper bound of the fusion gain is exactly the LiDAR-only baseline. The authors list joint extrinsics refinement as future work.
在标准（无干扰）条件下并非全面 SOTA（推断）Not uniformly SOTA under standard (undisturbed) conditions (inferred)
Table I 显示在 Panoptic nuScenes val 无干扰设置下，IAL-PieAug（82.3 PQ）和 IAL（80.3 PQ）的纯精度均高于 UP-Fuse（80.7 PQ 高于 IAL 但低于 IAL-PieAug 的对比略有出入，需以 Table I 数字为准）；UP-Fuse 的主要卖点是鲁棒性与效率（更高 FPS），而非在干净数据上刷新绝对精度上限。Table I shows that under the undisturbed Panoptic nuScenes val setting both IAL-PieAug (82.3 PQ) and IAL (80.3 PQ) exceed UP-Fuse on raw accuracy (the claim that 80.7 PQ is above IAL but below IAL-PieAug is slightly inconsistent, so the numbers in Table I should be taken as authoritative); the main selling point of UP-Fuse is robustness and efficiency (higher FPS), not pushing the absolute accuracy ceiling on clean data.
不确定性模块依赖训练时的退化增强分布（推断）The uncertainty module depends on the degradation-augmentation distribution seen during training (inferred)
不确定性预测 MLP 是通过 brightness/contrast/dropout/跨域 histogram matching 等特定合成退化学习"表征不稳定性"的；论文未在正文中明确讨论该模块对训练分布之外、更极端或未见过的退化类型（如物理传感器损伤、极端天气叠加）的泛化边界。The uncertainty-prediction MLP learns "representation instability" from specific synthetic degradations such as brightness/contrast/dropout and cross-domain histogram matching; the paper does not explicitly discuss in the main text the generalization boundary of this module to degradation types outside the training distribution that are more extreme or entirely unseen (e.g. physical sensor damage or compounded extreme weather).
