PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation

01 动机Motivation

机器人操作长期依赖任务专用模型或精确感知管线，难以泛化到"野外"（in-the-wild）环境。作者指出，人类能仅凭一眼和对动作的构想就预测三维世界如何响应——这种能力对机器人操作至关重要。现有方法要么依赖对象先验、要么局限于二维外观，无法捕获精细接触动力学。Robotic manipulation has long relied on task-specific models or precise perception pipelines, and generalizes poorly to "in-the-wild" environments. The authors observe that humans can predict how the 3D world will respond from a single glance and a contemplated action—a capability equally vital for robotic manipulation. Existing methods either depend on objectness priors or are confined to 2D appearance, and cannot capture fine-grained contact dynamics.

"Humans anticipate, from a glance and a contemplated action of their bodies, how the 3D world will respond, a capability equally vital for robotic manipulation."

"Unification for scaling: represent state and action in the same modality of 3D physical space."

**图1：PointWorld 总览。**系统展示跨不同具身形态（单臂 Franka、双臂人形机器人）和多样化任务类型（刚体推拉、可变形物体、关节体操作、工具使用）的操作能力，均由同一预训练三维世界模型驱动。**Figure 1: PointWorld overview.**The system demonstrates manipulation across different embodiments (single-arm Franka, bimanual humanoid) and diverse task families (rigid-body pushing and pulling, deformable objects, articulated-object manipulation, tool use), all driven by the same pretrained 3D world model.

~2M预训练轨迹数pretraining trajectories

~500h覆盖数据时长hours of data covered

0.1s单步推理时延single-step inference latency

1B最大模型参数量largest model size

核心洞察：将场景点云（scene point flow）与机器人点流（robot point flow）统一于同一三维物理空间，摆脱对象级标注和具身形态假设，实现跨机器人、跨任务的规模化学习——类比语言模型中的 next-token prediction，但面向三维空间与时间上的交互。Key insight: unifying scene point flow and robot point flow in the same 3D physical space removes object-level annotation and embodiment assumptions, enabling scalable learning across robots and tasks—analogous to next-token prediction in language models, but directed at interaction in 3D space and time.

02 方法Method

PointWorld 将状态和动作均表示为三维点流：场景状态由 RGB-D 反投影得到的点云描述，机器人动作由基于 URDF 正向运动学采样的机器人表面点流描述。模型在单次前向传播中以 chunk 形式（H=10 步）预测未来帧的逐点三维位移。PointWorld represents both state and action as 3D point flow: the scene state is described by a point cloud back-projected from RGB-D, and the robot action by a surface point flow sampled from the robot URDF with forward kinematics. In a single forward pass the model predicts per-point 3D displacements of future frames in chunks (H=10 steps).

**图2：系统架构。**输入 RGB-D 图像经遮罩（去除机器人像素）并反投影为场景点云；机器人动作通过 URDF 正向运动学生成机器人点流。两者拼接后送入 PTv3 backbone（场景点用冻结 DINOv2 特征，机器人点用时间嵌入），MLP 头预测逐点三维位移，应用于场景点得到下一帧预测。**Figure 2: System architecture.**The input RGB-D image is masked (robot pixels removed) and back-projected into a scene point cloud; the robot action is turned into robot point flow through URDF forward kinematics. The two are concatenated and fed into a PTv3 backbone (scene points carry frozen DINOv2 features, robot points carry temporal embeddings); an MLP head predicts per-point 3D displacements, which are applied to the scene points to obtain the next-frame prediction.

状态表示：Scene Point FlowState representation: Scene Point Flow

从 RGB-D 图像中遮罩掉机器人区域后反投影剩余像素，得到场景点云。与现有方法不同，PointWorld 不需要对象先验（objectness prior），仅使用原始几何与外观。场景点用冻结的 DINOv2 特征编码外观信息。逐帧点对应关系仅在模型"想象"推理阶段维护。After masking out the robot region in the RGB-D image, the remaining pixels are back-projected into a scene point cloud. Unlike existing methods, PointWorld requires no objectness prior, using only raw geometry and appearance. Scene points encode appearance with frozen DINOv2 features. Per-frame point correspondences are maintained only during the model's "imagination" rollout.

动作表示：Robot Point FlowAction representation: Robot Point Flow

通过 URDF 和正向运动学在机器人表面各链接上采样点，生成具身无关（embodiment-agnostic）的机器人点流表示。这种方式是"fully, rather than partially, observable"——完整暴露机器人几何，而非仅用末端执行器位姿或关节角。实验中每个 gripper 采样 300–500 点以平衡效率与接触推理能力。Points are sampled on every link of the robot surface through URDF and forward kinematics, yielding an embodiment-agnostic robot point flow representation. This makes the action "fully, rather than partially, observable"—the whole robot geometry is exposed, instead of only an end-effector pose or joint angles. In the experiments 300–500 points are sampled per gripper to balance efficiency against contact reasoning.

动力学预测与训练目标Dynamics prediction and training objective

将场景点与机器人点拼接成单一点云后由 PTv3 backbone 处理。训练面临两大挑战：(i) 稀疏训练信号（仅约 1–5% 的点在运动），(ii) 真实世界深度噪声。解决方案：Scene points and robot points are concatenated into a single point cloud and processed by the PTv3 backbone. Training faces two challenges: (i) a sparse training signal (only about 1–5% of the points move), and (ii) real-world depth noise. The solutions:

Movement weighting：m_k,i = σ(κ(δ_k,i − τ))，聚焦动态点，抑制静态点梯度。Movement weighting: m_k,i = σ(κ(δ_k,i − τ)), focusing on dynamic points and suppressing gradients from static ones.
Uncertainty regularization：预测对数方差 s_k,i，让模型自动下调噪声点的损失权重。Uncertainty regularization: a predicted log-variance s_k,i lets the model automatically down-weight the loss on noisy points.
Huber loss：对三维残差施加 robust 损失，配合 visibility-aware 监督。Huber loss: a robust loss on the 3D residuals, paired with visibility-aware supervision.

监督信号丰富度 — **图3：丰富的监督信号。**以布料操作为例，展示 movement weighting 和 uncertainty regularization 对动态区域的聚焦效果——仅有少数高运动点得到强监督，噪声点被不确定性头自动降权。**Figure 3: Rich supervision signal.**Taking cloth manipulation as an example, the figure shows how movement weighting and uncertainty regularization focus on dynamic regions—only a few high-motion points receive strong supervision, while noisy points are automatically down-weighted by the uncertainty head.

03 实验Experiments

PointWorld 在三个维度验证：(1) backbone 架构对比；(2) 数据与模型规模的 scaling law；(3) 跨域泛化与真实世界操作。数据集涵盖 DROID（D，大规模真实操作）、BridgeV2（B，家庭场景）及私有人形机器人数据（H），评测指标为 ℓ₂ mover error（动态点）和 ℓ₂ static error（静态点）。PointWorld is validated along three axes: (1) backbone architecture comparison; (2) scaling laws of data and model size; (3) cross-domain generalization and real-world manipulation. The datasets cover DROID (D, large-scale real manipulation), BridgeV2 (B, household scenes) and an in-house humanoid dataset (H); the metrics are the ℓ₂ mover error (dynamic points) and the ℓ₂ static error (static points).

Backbone 架构对比（Table 1）Backbone architecture comparison (Table 1)

Backbone	Params (相对)Params (relative)	ℓ₂ mover ↓	ℓ₂ static ↓	Latency (ms)
GBND (基线)GBND (baseline)	1.00×	0.0390	0.0066	13.46
PointNet	1.03×	0.0369	0.0084	5.93
SparseConv	33.31×	0.0396	0.0076	17.70
Transformer	41.06×	0.0339	0.0071	30.43
PTv3-50M	49.14×	0.0331	0.0067	59.60
PTv3-411M	398.67×	0.0315	0.0059	102.47
PTv3-1B	957.71×	0.0312	0.0056	123.65

PTv3-1B 在动态点误差上以 0.0312 达到最优，比 GBND 基线降低约 20%。延迟 123.65 ms 仍满足实时操作需求（约 8 Hz）。PTv3-1B reaches the best dynamic-point error at 0.0312, about 20% lower than the GBND baseline. Its latency of 123.65 ms still satisfies real-time manipulation (about 8 Hz).

Scaling Law（图9）Scaling Law (Figure 9)

"Scaling model size from 50M to 1B parameters yields smooth, log-linear gains" — consistent with "scaling-law observations in vision and language modeling."

跨域泛化（Table 2）Cross-domain generalization (Table 2)

设置Setting	ℓ₂ mover（Zero-Shot）ℓ₂ mover (Zero-Shot)	ℓ₂ mover（Finetuned）ℓ₂ mover (Finetuned)
D→D（域内）D→D (in-domain)	0.0315	—
B→B（域内）B→B (in-domain)	0.0087	—
D→B（跨域）D→B (cross-domain)	0.1460	0.0107
B→D（跨域）B→D (cross-domain)	0.0558	0.0378
D→H（held-out 真实）D→H (held-out real)	0.0305	0.0271
D+B→H（联合）D+B→H (joint)	0.0300	0.0272
Specialist（从零训练）Specialist (trained from scratch)	0.0293	—

零样本跨域迁移（D→H）误差 0.0305 与从零训练 Specialist（0.0293）接近；少量微调后进一步降至 0.0271。体现预训练世界模型的强泛化能力。The zero-shot cross-domain transfer (D→H) error of 0.0305 is close to the Specialist trained from scratch (0.0293); light finetuning lowers it further to 0.0271, showing the strong generalization of the pretrained world model.

真实世界操作Real-world manipulation

**图5：真实操作任务展示（图8）。**PointWorld 在 Franka 上完成刚体推拿（纸巾盒、书本）、可变形物体（围巾折叠、枕头放置）、关节体操作（微波炉开关、抽屉关合）和工具使用（扫帚清扫）等任务，均无需任务专用示范。**Figure 5: Real-world manipulation tasks (Figure 8).**On a Franka, PointWorld completes rigid-body pushing and pulling (tissue box, book), deformable objects (folding a scarf, placing a pillow), articulated-object manipulation (opening and closing a microwave, closing a drawer) and tool use (sweeping with a broom), none of which requires task-specific demonstrations.

消融实验Ablations

Movement weighting 单独使用：过度强调噪声点；需配合 uncertainty head + Huber loss 才能稳定训练。Movement weighting alone: over-emphasizes noisy points; it needs the uncertainty head plus the Huber loss to train stably.
Chunked vs. autoregressive：训练与测试均使用 chunked 策略时误差最小，且单次前向传播相比 autoregressive 快 2–10×。Chunked vs. autoregressive: the error is lowest when the chunked strategy is used in both training and testing, and the single forward pass is 2–10× faster than autoregressive rollout.
Partial observability：随机视角训练在不同相机数量下最鲁棒。Partial observability: training with randomized viewpoints is the most robust across different numbers of cameras.
动作表示对比：Robot point flow 优于全身点云、6-DoF 末端执行器位姿和关节角表示。Action representation comparison: robot point flow outperforms whole-body point clouds, 6-DoF end-effector poses and joint-angle representations.

04 局限性Limitations

说明：以下局限性均为作者在论文附录中明确陈述（stated by authors）。Note: All the limitations below are explicitly stated by the authors in the appendix of the paper.

静态初始状态假设Static initial-state assumption

模型假设任务开始时场景处于静态；无法处理动态初始状态（如运动中的物体）。The model assumes the scene is static when a task starts; it cannot handle dynamic initial states (such as objects already in motion).

奖励 / 代价函数需人工指定Reward / cost function must be specified manually

当前需通过 GUI 或 VLM 手动指定任务目标，缺乏自动奖励推断能力，限制了系统的自主性。Task goals currently have to be specified by hand through a GUI or a VLM; the absence of automatic reward inference limits the autonomy of the system.

小物体与标定噪声Small objects and calibration noise

对细粒度小物体操作表现欠佳；对深度传感器噪声和相机标定误差较为敏感，影响点云质量。Performance on fine-grained small-object manipulation is weak, and the method is fairly sensitive to depth-sensor noise and camera calibration error, which degrade point-cloud quality.

相关性而非因果性Correlation rather than causality

模型从交互数据中学习统计相关性，无法区分因果机制，可能在分布外场景产生错误预测。The model learns statistical correlations from interaction data and cannot distinguish causal mechanisms, so it may produce wrong predictions in out-of-distribution scenes.

缺乏光度（photometric）动力学建模No photometric dynamics modeling

仅预测几何位移，不对外观变化（光照、颜色、纹理）建模，限制了在需要感知外观变化任务上的应用。Only geometric displacement is predicted; appearance changes (lighting, color, texture) are not modeled, which limits application to tasks that require perceiving appearance change.

刚体机器人结构假设Rigid robot structure assumption

假设机器人具有刚体结构；无法建模可变形末端执行器（如软体夹爪）。The robot is assumed to have a rigid structure; deformable end-effectors (such as soft grippers) cannot be modeled.

精确执行与点追踪假设Assumed exact execution and point tracking

假设动作被精确执行且点追踪准确；真实场景中追踪失败可能导致预测误差累积。Actions are assumed to be executed exactly and point tracking to be accurate; in real scenes tracking failures can make prediction error accumulate.

缺乏显式物理先验No explicit physical prior

隐式学习物理规律，未编码守恒定律或物理约束，可能在极端或罕见物理场景下泛化不佳。Physical laws are learned implicitly, with no conservation laws or physical constraints encoded, so generalization may be poor in extreme or rare physical scenarios.