NeuROK: 生成式 4D 神经物体运动学

01 动机Motivation

现有的 4D 动态生成方法大多依赖预定义的物理模型，通过系统辨识估计参数——这将方法局限于特定类别，且难以扩展到大规模数据。NeuROK 的核心洞见是：通过学习"数据驱动的运动学状态参数化"，可以突破这些限制。Most existing 4D dynamics generation methods rely on predefined physical models and estimate parameters through system identification — which confines them to specific categories and makes scaling to large data difficult. NeuROK's key insight is that learning a "data-driven kinematic state parameterization" can break through these limits.

"Given a 3D geometric snapshot of a dynamic object, humans can intuitively imagine how the object would react under different physical conditions, even without precise knowledge of the governing physical equations."

NeuROK teaser figure — **图 1：NeuROK 总览。** 给定静态 3D 物体，在不同物理条件（力、动作、初始速度）驱动下，NeuROK 生成多样的逼真 4D 动态序列，涵盖弹性体、布料、铰接体、刚体等多种物体类型。**Figure 1: NeuROK overview.** Given a static 3D object and driven by different physical conditions (forces, actions, initial velocities), NeuROK generates diverse, realistic 4D dynamic sequences covering elastic bodies, cloth, articulated bodies, rigid bodies and other object types.

为什么现有方法不够？Why are existing methods insufficient?

类别受限： 现有方法（如 PhysDreamer、OmniPhysGS）通常假设特定物理模型（如弹性体或高斯溅射），无法跨类别泛化。Category-limited: Existing methods (such as PhysDreamer and OmniPhysGS) usually assume a specific physical model (elasticity or Gaussian splatting, for instance) and cannot generalize across categories.
需要物理标注： 基于"系统辨识"的方法需要对每类物体估计质量、刚度等物理参数，难以大规模部署。Physical annotations required: Methods based on "system identification" must estimate physical parameters such as mass and stiffness for every object category, which is hard to deploy at scale.
缺乏通用性： 铰接体、布料、连续体等物体的物理行为差异极大，统一建模是一大挑战。Lack of generality: Articulated bodies, cloth and continuum bodies differ enormously in physical behavior, so modeling them uniformly is a major challenge.

81.4%用户研究偏好率（动作对齐，105人）User study preference rate (action alignment, 105 participants)

83.3%用户研究偏好率（运动真实性，105人）User study preference rate (motion realism, 105 participants)

0.764逆运动学 IoU（超越次优方法 +34%）Inverse kinematics IoU (+34% over the second best)

2.343WorldScore MM（次优方法为 0.889）WorldScore MM (second best: 0.889)

02 方法Method

NeuROK 的核心思路是：先学习一个低维潜空间（即"神经运动学状态参数化"），再在此潜空间中用 Lagrangian 力学方程驱动动力学演化——将高维顶点空间的问题降维到低维广义坐标空间。NeuROK's core idea is to first learn a low-dimensional latent space (the "neural kinematic state parameterization"), and then drive the dynamic evolution inside this latent space with Lagrangian mechanics equations — reducing a problem posed in the high-dimensional vertex space to a low-dimensional space of generalized coordinates.

NeuROK architecture overview — **图 3：框架总览。** 给定静态 3D 形状，NeuROK 使用基于 Transformer 的编码器预测该实例特定的潜空间，以表示物体的不同运动学状态；再通过 Lagrangian 力学在此潜空间中进行动力学求解，最终解码为变形 mesh。**Figure 3: Framework overview.** Given a static 3D shape, NeuROK uses a Transformer-based encoder to predict an instance-specific latent space that represents the different kinematic states of the object; dynamics are then solved within this latent space via Lagrangian mechanics and finally decoded into a deformed mesh.

运动学先验编码器 ℰ_condKinematic prior encoder ℰ_cond

以静态 3D mesh 为输入，输出该实例的运动学先验分布。采用 Perceiver 架构（可学习 token + 交叉注意力 + 自注意力），位置编码跟随 3DShape2Vecset，天然支持可变数量的顶点，具备良好的可扩展性。Takes a static 3D mesh as input and outputs the kinematic prior distribution of that instance. It adopts a Perceiver architecture (learnable tokens + cross-attention + self-attention) with positional encoding following 3DShape2Vecset, natively supporting a variable number of vertices and offering good scalability.

变分变形编码器 ℰ_VAEVariational deformation encoder ℰ_VAE

在条件 mesh 基础上，对变形场编码为后验分布参数。顶点变形通过 dual quaternions（双四元数） 参数化，再经重心插值映射至采样点，有效捕捉旋转主导的变形（如铰接体关节转动）。Conditioned on the input mesh, the deformation field is encoded into posterior distribution parameters. Vertex deformations are parameterized by dual quaternions and then mapped to sample points through barycentric interpolation, effectively capturing rotation-dominated deformations such as joint rotation in articulated bodies.

变形解码器 𝒟Deformation decoder 𝒟

从采样的潜向量解码为变形 mesh：在输入 mesh 表面采样查询点，经自注意力和交叉注意力处理潜 token，MLP 预测变形向量，最终通过加权平均驱动 mesh 顶点。Decodes a sampled latent vector into a deformed mesh: query points are sampled on the input mesh surface, latent tokens are processed by self-attention and cross-attention, an MLP predicts the deformation vectors, and mesh vertices are finally driven by weighted averaging.

Lagrangian 动力学求解Lagrangian dynamics solving

学到的潜空间充当广义坐标（generalized coordinates），在此低维空间中求解 Euler-Lagrange 方程：The learned latent space serves as generalized coordinates, and the Euler-Lagrange equation is solved within this low-dimensional space:

m G(z)z̈ + C(z, ż) + ∇_z V = 0

其中 G(z) 为广义质量矩阵，C(z,ż) 为科里奥利项，∇_z V 为势能梯度。该公式维持能量守恒等物理一致性，同时无需为每类物体手动定义物理约束。Here G(z) is the generalized mass matrix, C(z,ż) the Coriolis term and ∇_z V the potential energy gradient. The formulation preserves physical consistency such as energy conservation while requiring no manually defined physical constraints for each object category.

训练目标Training objective

采用条件 VAE 目标函数，在大规模 4D 数据集上联合训练三个模型：A conditional VAE objective is used to jointly train the three models on a large-scale 4D dataset:

ℒ = ||δ_sample − δ_pred||²₂ + λ · D_KL(q ‖ p)， λ = 0.01

训练仅需 4D 几何轨迹（变形 mesh 序列），无需物理参数标注或动作标注。训练数据涵盖从现有工作和物理仿真中整合的大规模 4D 数据集，覆盖弹性体、布料、铰接体、刚体、连续体等多类别。Training requires only 4D geometric trajectories (sequences of deformed meshes), with no physical parameter or action annotations. The training data is a large-scale 4D dataset assembled from existing work and physics simulation, covering elastic bodies, cloth, articulated bodies, rigid bodies, continuum bodies and more.

03 实验Experiments

实验从两个维度评估 NeuROK：（1）逆运动学——给定目标形状，找到潜编码使重建最优；（2）生成式 4D 仿真——在不同物理条件下生成时序动态序列。用户研究共邀请 105 名参与者。The experiments evaluate NeuROK along two axes: (1) inverse kinematics — given a target shape, find the latent code that best reconstructs it; (2) generative 4D simulation — generating temporal dynamic sequences under different physical conditions. The user study involved 105 participants in total.

逆运动学性能（Table 1）Inverse kinematics performance (Table 1)

方法Method	Chamfer (L1) ↓	Chamfer (L2) ↓	IoU ↑
KeyPointDeformer	0.067	0.067	0.570
CANOR	0.082	0.067	0.568
NeuROK（本文）NeuROK (ours)	0.028	0.028	0.764

**图 5：逆运动学定性比较。** NeuROK 在重建各类物体的目标变形状态时，相比 KeyPointDeformer 和 CANOR 均有明显优势，形状边界更清晰，IoU 提升显著。**Figure 5: Qualitative comparison on inverse kinematics.** When reconstructing the target deformation states of various objects, NeuROK holds a clear advantage over KeyPointDeformer and CANOR, with sharper shape boundaries and a marked IoU gain.

生成式 4D 仿真（Table 2）Generative 4D simulation (Table 2)

与三类基线方法对比：PhysDreamer（弹性体物理仿真）、OmniPhysGS（高斯溅射物理）、AnimateAnyMesh（mesh 动画）。评估指标包括用户研究（动作对齐度、运动真实性）、VBench AQ（视频质量）、WorldScore MM（多模态世界分数）。Comparison against three families of baselines: PhysDreamer (elastic-body physics simulation), OmniPhysGS (Gaussian splatting physics) and AnimateAnyMesh (mesh animation). Evaluation metrics include the user study (action alignment, motion realism), VBench AQ (video quality) and WorldScore MM (multimodal world score).

方法Method	对齐度 ↑Alignment ↑	真实性 ↑Realism ↑	VBench AQ ↑	WorldScore MM ↑
PhysDreamer	5.95%	5.36%	0.362	0.783
OmniPhysGS	1.67%	0.48%	0.380	0.544
AnimateAnyMesh	5.83%	6.67%	0.450	0.889
NeuROK（本文）NeuROK (ours)	81.43%	83.33%	0.483	2.343

**图 6：4D 生成定性比较。** 在相同物理条件输入下，NeuROK 生成的运动序列在真实性和对齐度上均大幅优于 PhysDreamer、OmniPhysGS 和 AnimateAnyMesh。**Figure 6: Qualitative comparison on 4D generation.** Under the same physical-condition inputs, the motion sequences generated by NeuROK are far superior to PhysDreamer, OmniPhysGS and AnimateAnyMesh in both realism and alignment.

消融实验（Table 1 ablation）Ablation study (Table 1 ablation)

对三个关键设计组件进行消融，均在逆运动学任务上评估：Three key design components are ablated, all evaluated on the inverse kinematics task:

配置Configuration	Chamfer L1 ↓	Chamfer L2 ↓	IoU ↑
NeuROK 完整版NeuROK, full model	0.028	0.028	0.764
去除模型降维（w/o Model Reduction）Without model reduction (w/o Model Reduction)	0.045	0.059	0.711
去除数据增强（w/o Data Aug）Without data augmentation (w/o Data Aug)	0.036	0.041	0.724
去除双四元数（w/o Dual-Quat）Without dual quaternions (w/o Dual-Quat)	0.033	0.037	0.728

泛化性：未见类别与真实扫描物体Generalization: unseen categories and real scanned objects

**图 9：未见类别的泛化能力。** NeuROK 可直接泛化到训练集中完全未出现的物体类别，无需针对新类别重新训练或微调。**Figure 9: Generalization to unseen categories.** NeuROK generalizes directly to object categories that never appear in the training set, with no retraining or fine-tuning for the new categories.

此外，NeuROK 可直接对真实世界扫描获得的 3D 物体进行仿真（Figure 7），无需针对扫描物体进行特殊适配，体现了框架的实用性。In addition, NeuROK can directly simulate 3D objects acquired by real-world scanning (Figure 7) without any special adaptation for scanned objects, which demonstrates the practicality of the framework.

04 局限性Limitations

注：论文结论部分未包含明确的局限性段落，仅表示"开创了有前景的未来研究方向"。以下各点：标注"作者表述"者来自论文原文；标注"推断（inferred）"者为基于方法设计合理推断的潜在局限。Note: The conclusion of the paper contains no explicit limitations paragraph and only states that it "opens up promising future research directions". Among the points below, those marked "stated" come from the original text of the paper, while those marked "inferred" are potential limitations reasonably inferred from the design of the method.

依赖训练数据覆盖的物体类型 （推断 inferred）Dependence on the object types covered by the training data (inferred)

NeuROK 从大规模 4D 数据集学习运动学先验。对于与训练分布差异极大的物体（如极端拓扑形变、流体类物体），潜空间的表达能力可能受限。虽然论文展示了对未见类别的泛化（Figure 9），但泛化边界尚未被系统性验证。NeuROK learns kinematic priors from a large-scale 4D dataset. For objects that differ greatly from the training distribution (extreme topological deformation or fluid-like objects, for instance), the expressive power of the latent space may be limited. Although the paper demonstrates generalization to unseen categories (Figure 9), the boundary of that generalization has not been systematically verified.

物理输入精度要求 （推断 inferred）Accuracy requirements on the physical inputs (inferred)

将外部物理条件（力、初始速度、动作）接入系统的方式是通过优化初始运动学状态 (z₀, ż₀) 以匹配指定粒子的位置和速度。这一接口在面对高度不确定性的物理输入或缺乏明确锚点的场景时，行为尚不清晰。External physical conditions (forces, initial velocities, actions) are fed into the system by optimizing the initial kinematic state (z₀, ż₀) so that it matches the position and velocity of specified particles. How this interface behaves under highly uncertain physical inputs, or in scenes that lack clear anchor points, remains unclear.

仅限 mesh 表示，不直接支持高斯溅射或 NeRF （推断 inferred）Restricted to mesh representations; no direct support for Gaussian splatting or NeRF (inferred)

NeuROK 以 3D mesh 为输入输出，对于以高斯溅射（3DGS）或神经辐射场（NeRF）表示的场景，需要额外的格式转换步骤，与直接操作渲染表示的方法（如 PhysDreamer、OmniPhysGS）相比，端到端流程稍显复杂。NeuROK takes 3D meshes as its input and output. Scenes represented by Gaussian splatting (3DGS) or neural radiance fields (NeRF) require an extra format-conversion step, which makes the end-to-end pipeline slightly more involved than methods operating directly on rendering representations (such as PhysDreamer and OmniPhysGS).

"开创新的研究范式" （作者表述 stated）"Introducing a new research paradigm" (stated)

论文结论原文："Our work opens up promising future research directions and introduces a new research paradigm in 4D visual generation." 这意味着若干开放问题（如更精细的物理接口、更大规模训练数据、与渲染管线的深度集成）有待后续工作探索。From the conclusion of the paper: "Our work opens up promising future research directions and introduces a new research paradigm in 4D visual generation." This implies that a number of open problems — finer-grained physical interfaces, larger-scale training data, deeper integration with rendering pipelines — remain for future work.