D4RT — 用「按需查询」高效重建动态四维场景

01 研究动机Motivation

「从视频中理解和重建动态场景复杂的几何与运动，依然是计算机视觉的一个巨大挑战。」尽管大家都需要统一的 4D 理解，「主流方法往往把问题拆成一个个离散、任务专用的组件」。“Understanding and reconstructing the intricate geometry and motion of dynamic scenes from video remains a grand challenge in computer vision.” Although unified 4D understanding is needed everywhere, “prevailing approaches often decompose the problem into discrete, task-specific components”.

MegaSaM

「依赖一堆现成模型，分别估计单目深度、度量深度和运动分割。」融合这些信号「需要计算上昂贵的测试时优化来强制几何一致」。“Relies on a suite of off-the-shelf models to separately estimate monocular depth, metric depth and motion segmentation.” Fusing these signals “requires computationally expensive test-time optimization to enforce geometric consistency”.

VGGT

前馈方法，但「为不同模态使用各自专用的解码器」。关键是，它「无法为场景的动态部分建立对应」。A feed-forward approach, but one that “uses dedicated decoders for the different modalities”. Crucially, it “cannot establish correspondence for the dynamic parts of a scene”.

SpatialTrackerV2

「引入了动态」，但「仍缺乏统一的单阶段表述，反而依赖昂贵的迭代细化」。“Introduces dynamics”, yet “still lacks a unified single-stage formulation and instead relies on expensive iterative refinement”.

共性问题Shared shortcomings

许多方法不支持视频内变化的内参、只能用第一帧作参考、或为稀疏/稠密输出准备不同的头——管线臃肿。Many methods do not support intrinsics that vary within a video, can only use the first frame as reference, or need separate heads for sparse/dense outputs —— a bloated pipeline.

「我们提出范式转变：从碎片化的、帧级解码，转向高效的、按需查询（on-demand querying）。」“We propose a paradigm shift: from fragmented, frame-level decoding to efficient, on-demand querying.”

teaser — **核心思想。**「我们的模型首先编码输入视频，生成一个潜在的场景表征，然后据此独立地解码任意数量的时空点查询。这个简单、新颖的设计把所有 4D 重建任务统一到单一接口，并解锁了高效的训练与推理。」**Core idea.** “Our model first encodes the input video into a latent scene representation, from which it then independently decodes an arbitrary number of spatio-temporal point queries. This simple, novel design unifies all 4D reconstruction tasks under a single interface and unlocks efficient training and inference.”

贡献Contributions

「一种对视频中点级 4D 场景信息进行高效前馈查询的新方法。」“A new method for efficient feed-forward querying of point-level 4D scene information in video.”
「通过单一接口，为静态与动态场景解锁 4D 对应、点云、深度图和相机参数。」“Unlocking 4D correspondence, point clouds, depth maps and camera parameters for static and dynamic scenes through a single interface.”
「在动态 4D 重建与跟踪上刷新 SOTA，同时在速度与精度上都超越现有方法。」“New SOTA on dynamic 4D reconstruction and tracking, while surpassing existing methods in both speed and accuracy.”
「一个灵活的解码器，解锁了高效跟踪视频中所有像素的算法，实现稠密、整体的场景重建。」“A flexible decoder that unlocks an algorithm for efficiently tracking every pixel in a video, enabling dense, holistic scene reconstruction.”

02 方法Method

D4RT 是「一个受 Scene Representation Transformer 启发的简洁编码器-解码器架构」。编码器产出全局场景表征 F；一个轻量解码器随后「通过一个简单的低层接口查询 F」。D4RT is “a simple encoder-decoder architecture inspired by the Scene Representation Transformer”. The encoder produces a global scene representation F; a lightweight decoder then “queries F through a simple, low-level interface”.

method overview — **两阶段架构。**编码器捕捉「关于整个环境的信息，识别跨所有视频帧的稠密对应，并理解时间的流动及其对场景的影响」。F 一旦算出「就在第二阶段保持固定，解码器从任意数量的查询 cross-attend 进 F」。**Two-stage architecture.** The encoder captures “information about the entire environment, identifies dense correspondence across all video frames, and understands the flow of time and its effect on the scene”. Once computed, F “stays fixed in the second stage, and the decoder cross-attends into F from an arbitrary number of queries”.

查询的定义：把空间与时间解耦Defining a query: decoupling space from time

一个查询定义为 q = (u, v, t_src, t_tgt, t_cam)：A query is defined as q = (u, v, t_src, t_tgt, t_cam):

源参数 (u, v, t_src) —— 源帧 t_src 中某个关注点的归一化 2D 坐标 (u,v)∈[0,1]²。Source parameters (u, v, t_src) —— the normalized 2D coordinates (u,v)∈[0,1]² of a point of interest in source frame t_src.
目标参数 (t_tgt, t_cam) —— 「目标时刻与参考相机坐标系的时间索引」。Target parameters (t_tgt, t_cam) —— “the time indices of the target moment and of the reference camera frame”.

每个查询「被完全独立地处理」，输出其 3D 点位置 P = Dec(q, F) ∈ ℝ³。本文强调三个优良性质：Every query is “processed entirely independently”, outputting its 3D point position P = Dec(q, F) ∈ ℝ³. The paper highlights three desirable properties:

「这些索引不必相同，允许空间与时间的完全解耦。」“These indices need not be identical, allowing a complete decoupling of space and time.”
「每个查询独立解码，既支持高效的训练与推理，也支持灵活的解码（稀疏或稠密）。」“Each query is decoded independently, supporting efficient training and inference as well as flexible decoding (sparse or dense).”
「这个接口以统一、一致的方式解锁了一整套下游应用。」“This interface unlocks a whole suite of downstream applications in a unified, consistent manner.”

架构Architecture

编码器Encoder

一个「带交替的局部逐帧与全局自注意力层」的 ViT。视频「在 tokenize 前 resize 到固定的正方形分辨率」；原始长宽比被嵌入「单独的一个 token」。默认：ViT-g、40 层、时空 patch 2×16×16，约 1B 参数。A ViT “with alternating local per-frame and global self-attention layers”. The video is “resized to a fixed square resolution before tokenization”; the original aspect ratio is embedded in “a single separate token”. Default: ViT-g, 40 layers, spatio-temporal patches of 2×16×16, about 1B parameters.

点级解码器Point-level decoder

「一个小型 cross-attention Transformer」（8 层、144M）。查询 token = (u,v) 的 Fourier 嵌入 + 可学习时间步嵌入。关键技巧：「用一个以 (u,v) 为中心的 9×9 局部 RGB patch 嵌入来增强查询，能极大地提升性能」。“A small cross-attention Transformer” (8 layers, 144M). The query token = a Fourier embedding of (u,v) + a learned time-step embedding. Key trick: “augmenting the query with an embedding of a local 9×9 RGB patch centred at (u,v) improves performance dramatically”.

为什么禁止查询间交互？「独立解码查询是一个有重大优势的刻意设计决策。」它支持高效训练（少量查询即可监督）、推理时自由选取查询、平凡的并行——且作者「在早期实验中经验性地观察到：一旦启用查询间的 self-attention，性能会大幅下降」。Why forbid interaction between queries? “Decoding queries independently is a deliberate design decision with major advantages.” It enables efficient training (a handful of queries suffices for supervision), free choice of queries at inference and trivial parallelism —— and the authors “empirically observed in early experiments that performance drops substantially once self-attention between queries is enabled”.

训练Training

端到端训练。主监督是「对归一化 3D 点位置 P 的 L1 损失」（按均值深度归一化，再经 sign(x)·log(1+|x|)「抑制远点的影响」）。辅助损失：2D 坐标 L1、3D 表面法向余弦相似度、目标点可见性 BCE、点运动 L1，以及置信度惩罚。
配置：ViT-g 编码器 + 8 层解码器；48 帧片段、256×256；每步 2048 个随机查询；500K 步、64 块 TPU，「刚过 2 天」。Trained end-to-end. The main supervision is “an L1 loss on the normalized 3D point position P” (normalized by mean depth, then passed through sign(x)·log(1+|x|) to “suppress the influence of distant points”). Auxiliary losses: L1 on 2D coordinates, cosine similarity of 3D surface normals, BCE on target-point visibility, L1 on point motion, and a confidence penalty.
Setup: ViT-g encoder + 8-layer decoder; 48-frame clips at 256×256; 2048 random queries per step; 500K steps on 64 TPUs, “just over 2 days”.

高效的稠密动态对应Efficient dense dynamic correspondence

朴素地跟踪所有像素「需要 O(T²HW) 次查询，其中大多数并不需要」。本文引入一个用占据栅格的算法，它「只从未访问的像素发起新轨迹」，每条全视频轨迹会把它「可见经过」的所有时空像素标记为已访问。「经验上，根据视频运动复杂度，这带来 5–15× 的自适应加速。」之所以可行，「正是因为我们的解码器既稀疏又轻量」。Naively tracking every pixel “requires O(T²HW) queries, most of which are unnecessary”. The paper introduces an algorithm based on an occupancy grid that “only starts new trajectories from unvisited pixels”; each full-video trajectory marks every spatio-temporal pixel it “visibly passes through” as visited. “Empirically, this yields an adaptive speed-up of 5–15×, depending on the motion complexity of the video.” This is feasible “precisely because our decoder is both sparse and lightweight”.

03 一套查询接口，覆盖所有 4D 任务One query interface, all 4D tasks

「通过简单地改变查询的取法，我们的框架就能处理一大类 4D 任务」——无需任务专用的头。“By simply changing how queries are drawn, our framework can handle a broad family of 4D tasks” —— with no task-specific heads.

任务Task	查询取法How queries are drawn
点轨迹（3D）Point trajectories (3D)	固定 (u,v,t_src)，让 t_tgt = t_cam = {1…T} 遍历 →「它的点轨迹，即对应点在整段视频中的 3D 轨迹」。Fix (u,v,t_src) and sweep t_tgt = t_cam = {1…T} → “its point trajectory, i.e. the 3D trajectory of the corresponding point throughout the video”.
点云Point clouds	「视频中所有像素的 3D 位置都能在一个共享参考帧 t_cam 下直接预测」，无需用噪声相机做坐标变换。“The 3D positions of all pixels in the video can be predicted directly in a shared reference frame t_cam”, with no coordinate transform through noisy cameras.
深度图Depth maps	「令 t_src = t_tgt = t_cam 查询任意像素，只取输出 P 的 Z 分量」。“Query any pixel with t_src = t_tgt = t_cam and take only the Z component of the output P”.
相机外参Camera extrinsics	在两帧网格采样源点 → 同一批 3D 点在不同参考系下的表达 → 用「Umeyama 算法（求解 3×3 SVD 分解）」求刚性变换。Sample source points on a grid in two frames → the same set of 3D points expressed in different reference frames → recover the rigid transform with “the Umeyama algorithm (solving a 3×3 SVD decomposition)”.
相机内参Camera intrinsics	解码网格点，假设主点 (0.5, 0.5) 的针孔模型，按 f = p_z·(u−0.5)/p_x 求焦距，「对 k 个估计取中位数以稳健化」。Decode grid points, assume a pinhole model with principal point (0.5, 0.5), and solve for the focal length as f = p_z·(u−0.5)/p_x, “taking the median over k estimates for robustness”.

point cloud comparison — **整体重建。**纯重建方法（MegaSaM、π³）「无法理解场景中的动态，要么在实体移动时产生重复的影像，要么完全无法重建它们」。基于跟踪的模型「通常只从一帧跟点，在第一帧中被遮挡的所有区域留下空洞，如此处的 SpatialTrackerV2」。相比之下，「D4RT 能在统一参考系下跟踪视频里所有动态像素」。**Holistic reconstruction.** Pure reconstruction methods (MegaSaM, π³) “fail to understand the dynamics in the scene, either producing duplicated imagery as entities move or failing to reconstruct them at all”. Tracking-based models “typically track points from a single frame, leaving holes in every region occluded in the first frame, as with SpatialTrackerV2 here”. By contrast, “D4RT tracks all dynamic pixels in the video in a unified reference frame”.

04 实验Experiments

D4RT「刷新了 SOTA，在广泛的 4D 重建任务上超越了既有方法」——速度与精度兼得。D4RT “sets a new SOTA, outperforming established methods on a wide range of 4D reconstruction tasks” —— in both speed and accuracy.

18–300×3D 跟踪比前作更快faster 3D tracking than prior work

~100×相机位姿吞吐量超过 MegaSaMcamera-pose throughput over MegaSaM

5–15×跟踪所有像素的自适应加速adaptive speed-up for tracking all pixels

SOTA4D 跟踪、深度、点云、相机4D tracking, depth, point clouds, cameras

4D 重建与跟踪（TAPVid-3D）4D reconstruction and tracking (TAPVid-3D)

在 TAPVid-3D（「三个真实世界、富有挑战的子集」）上评测。对局部坐标系 3D 跟踪，报告 APD₃D、遮挡精度 OA 与 3D Average Jaccard（AJ），「D4RT 达到 SOTA……无论是否给定真值内参」。对世界坐标系 3D 跟踪——它「也衡量模型隐式切换参考系的能力」——「我们的模型在这项任务上同样出色，各项指标都有强力提升」。在效率上，「我们的模型比前作快 18–300×」。Evaluated on TAPVid-3D (“three challenging real-world subsets”). For 3D tracking in the local frame, the paper reports APD₃D, occlusion accuracy OA and 3D Average Jaccard (AJ): “D4RT achieves SOTA … both with and without ground-truth intrinsics”. For 3D tracking in the world frame —— which “also measures the ability of a model to implicitly switch reference frames” —— “our model likewise excels at this task, with strong gains across all metrics”. On efficiency, “our model is 18–300× faster than prior work”.

speed vs accuracy — **效率 vs 精度（相机位姿）。**「为公平起见，我们去掉了基线中与相机估计无关的解码头，使它们更快。即便如此，我们发现 D4RT 在精度与效率上都超越所有现有方法，吞吐量比 MegaSaM 高出两个数量级。」**Efficiency vs accuracy (camera pose).** “For fairness, we removed from the baselines the decoding heads unrelated to camera estimation, making them faster. Even so, we find that D4RT surpasses all existing methods in both accuracy and efficiency, with a throughput two orders of magnitude higher than MegaSaM.”

3D 重建：点云 / 深度 / 相机位姿3D reconstruction: point clouds / depth / camera pose

点云（Sintel 动态 + ScanNet 静态）：mean-shift 对齐后取 mean L1，「我们的模型在各数据集上都优于近期的 SOTA 模型」。Point clouds (dynamic Sintel + static ScanNet): mean L1 after mean-shift alignment — “our model outperforms recent SOTA models on every dataset”.
深度（Sintel / ScanNet / KITTI / Bonn），scale-only 与 scale-and-shift 对齐：「在所有 benchmark 上都是顶尖水平，在最难、高动态的 Sintel 上尤为突出」。Depth (Sintel / ScanNet / KITTI / Bonn), under scale-only and scale-and-shift alignment: “top-tier on all benchmarks, and especially pronounced on the hardest, highly dynamic Sintel”.
相机位姿（ATE / RPE-trans / RPE-rot / AUC@30）：「我们的模型在室内（ScanNet、Re10K）和模拟室外（Sintel）场景上都强力超越竞争基线」。Camera pose (ATE / RPE-trans / RPE-rot / AUC@30): “our model strongly outperforms competing baselines on both indoor (ScanNet, Re10K) and simulated outdoor (Sintel) scenes”.

in the wild results — 「我们进一步在多样的真实世界视频上评测我们的方法……展示了它在静态与动态场景上的表现。」“We further evaluate our method on a diverse set of real-world videos … demonstrating its behaviour on both static and dynamic scenes.”

消融（默认 ViT-L + 6 层解码器，Sintel 上评测）Ablations (default ViT-L + 6-layer decoder, evaluated on Sintel)

消融项Ablation	发现Finding
局部 RGB patchLocal RGB patch	「带来巨大的性能提升。」它帮助查询「建立更可靠的对应」，并提供「帮助从背景中分割物体、产生细粒度预测的低层线索」——比 DPT 的 skip 连接「简单得多」。“Brings a huge performance gain.” It helps queries “establish more reliable correspondence” and supplies “low-level cues that help segment objects from the background and produce fine-grained predictions” —— and is “far simpler” than DPT-style skip connections.
编码器规模Encoder scale	从 ViT-B(90M) 到 ViT-g(1B)：「显著的性能提升，深度估计与相机位姿的 RPE-R 尤其明显」。From ViT-B(90M) to ViT-g(1B): “a significant performance gain, particularly pronounced for depth estimation and camera-pose RPE-R”.
辅助损失Auxiliary losses	「有些显著提升深度（如 2D 位置、法向），有些对更好的相机位姿至关重要（如置信度）。」“Some markedly improve depth (e.g. 2D position, normals), while others are crucial for better camera pose (e.g. confidence).”
连续查询 → 子像素Continuous queries → sub-pixel	由于 (u,v) 在连续 [0,1]²，「我们的解码器能在任意分辨率探测场景」。配合原分辨率 RGB patch，可恢复发丝、物体边界等精细细节。Because (u,v) live in the continuous [0,1]², “our decoder can probe the scene at arbitrary resolution”. Combined with native-resolution RGB patches, this recovers fine detail such as hair strands and object boundaries.

long video kitti — **长视频泛化。**视频被切成「重叠片段」，再「用 Umeyama 算法估计 Sim(3) 变换对齐，基于重叠区置信度最高的 85% 的点」。在 KITTI 上，「我们的模型给出最佳 ATE……大幅超越 VGG-T 与 π³」。**Long-video generalization.** Videos are split into “overlapping chunks”, then aligned by “estimating a Sim(3) transform with the Umeyama algorithm, based on the 85% most confident points in the overlap”. On KITTI, “our model gives the best ATE … substantially outperforming VGG-T and π³”.

05 局限性Limitations

说明：正文没有设置专门的「Limitations」章节。以下要点部分来自论文（附录/消融）中明确报告的现象，部分是根据方法设计推断的约束，已尽量标注来源。Note: the paper has no dedicated “Limitations” section. The points below partly report phenomena stated explicitly in the paper (appendix/ablations) and partly are constraints inferred from the method design; sources are flagged wherever possible.

高频细节恢复仍不完美（附录明确）High-frequency detail is still recovered imperfectly (explicit in the appendix)

子像素解码消融指出：朴素的稠密查询「恢复了更平滑的边缘，但仍然无法恢复高频细节」。要恢复精细细节（发丝、物体边界），需要额外机制——「在原始分辨率下从源帧提取局部 RGB patch 进行解码」。换言之，编码器原生分辨率的输出「本质上是粗糙的」，锐度高度依赖 RGB patch 技巧。The sub-pixel decoding ablation notes that naive dense querying “recovers smoother edges but still fails to recover high-frequency detail”. Recovering fine detail (hair strands, object boundaries) requires an extra mechanism —— “decoding with local RGB patches extracted from the source frame at native resolution”. In other words, output at the encoder's native resolution is “inherently coarse”, and sharpness depends heavily on the RGB-patch trick.

训练/推理分辨率低且需固定正方形（256×256）Low training/inference resolution, fixed to a square (256×256)

模型「在 256×256 分辨率的 48 帧片段上训练」，编码器把输入「resize 到固定的正方形分辨率再 tokenize」（长宽比只能通过一个单独 token 间接编码）。这对高分辨率或极端长宽比场景是结构性约束；精细几何主要靠解码端的连续查询 + 原分辨率 patch 来补偿。The model is “trained on 48-frame clips at 256×256 resolution”, and the encoder “resizes the input to a fixed square resolution before tokenization” (aspect ratio can only be encoded indirectly through a single separate token). This is a structural constraint for high-resolution or extreme-aspect-ratio scenes; fine geometry is mainly compensated for on the decoding side by continuous queries plus native-resolution patches.

长视频需切片 + Sim(3) 拼接（非端到端）Long videos need chunking + Sim(3) stitching (not end-to-end)

处理长序列时并非一次前馈：视频被切成「重叠片段」，再用 Umeyama Sim(3) 估计对齐（类似 VGGT-Long 的第一阶段，但省略回环检测与全局优化）。超长视频的全局一致性因此依赖后处理拼接，而非模型内生的全局推理。Long sequences are not handled in a single forward pass: the video is split into “overlapping chunks”, then aligned by Umeyama Sim(3) estimation (similar to the first stage of VGGT-Long, but omitting loop closure and global optimization). Global consistency for very long videos therefore relies on post-hoc stitching rather than on global reasoning intrinsic to the model.

查询被刻意禁止相互交互Queries are deliberately forbidden to interact

独立解码是效率的关键，但也是一种约束：作者「经验性地观察到，启用查询间 self-attention 会带来大幅性能下降」。每个查询孤立解码，无法显式利用与邻近查询的关系——所有跨点一致性都必须由共享的场景表征 F 隐式承载。Independent decoding is the key to efficiency, but it is also a constraint: the authors “empirically observed that enabling self-attention between queries causes a substantial performance drop”. Each query is decoded in isolation and cannot explicitly exploit its relation to neighbouring queries —— all cross-point consistency must be carried implicitly by the shared scene representation F.

训练严重依赖合成数据（推断）Training depends heavily on synthetic data (inferred)

训练混合偏重合成/半合成数据集（Kubric、PointOdyssey、Dynamic Replica、VirtualKITTI、TartanAir……），因为动态对应的真值在真实世界极难获取。这带来对合成分布的潜在依赖与 sim-to-real 泛化压力——是这类动态对应方法共同的约束。The training mixture leans on synthetic/semi-synthetic datasets (Kubric, PointOdyssey, Dynamic Replica, VirtualKITTI, TartanAir …), because ground truth for dynamic correspondence is extremely hard to obtain in the real world. This brings a potential dependence on the synthetic distribution and sim-to-real generalization pressure —— a constraint shared across this family of dynamic-correspondence methods.

内参估计依赖简化假设（推断）Intrinsics estimation rests on simplifying assumptions (inferred)

内参恢复「假设主点在 (0.5, 0.5) 的针孔相机模型」。带畸变的相机（如鱼眼）需要「在初始估计之上加一个非线性细化步骤」，并非由主模型直接产出。Intrinsics recovery “assumes a pinhole camera model with principal point at (0.5, 0.5)”. Cameras with distortion (e.g. fisheye) require “a non-linear refinement step on top of the initial estimate”, which the main model does not produce directly.

总体而言，D4RT 的设计哲学是「用一个统一、轻量、可自由查询的接口换取强大的效率与灵活性」，其代价主要落在分辨率/高频细节、长视频拼接，以及对合成数据的依赖上。Overall, the design philosophy of D4RT is to “trade a unified, lightweight, freely queryable interface for strong efficiency and flexibility”, and the price falls mainly on resolution/high-frequency detail, long-video stitching, and reliance on synthetic data.