PhyCo：面向生成式运动的可控物理先验学习PhyCo: Learning Controllable Physical Priors for Generative Motion
PhyCo: Learning Controllable Physical Priors for Generative Motion
Sriram Narayanan¹²，Ziyu Jiang²，Srinivasa G. Narasimhan¹，Manmohan Chandraker²³ | ¹Carnegie Mellon University ²NEC Labs America ³UC San DiegoSriram Narayanan¹², Ziyu Jiang², Srinivasa G. Narasimhan¹, Manmohan Chandraker²³ | ¹Carnegie Mellon University ²NEC Labs America ³UC San Diego
PhyCo 是首个将大规模物理仿真数据集、ControlNet 物理属性条件调节，以及 VLM 奖励优化三者融合的视频生成框架。该框架在不依赖推理期仿真器的前提下，实现了对摩擦力、弹性系数、形变参数和外力方向等物理属性的连续可控生成，显著提升了生成视频的物理真实性。PhyCo is the first video generation framework that unifies a large-scale physics simulation dataset, ControlNet-based conditioning on physical properties, and VLM reward optimization. Without relying on a simulator at inference time, it achieves continuously controllable generation of physical properties such as friction, restitution, deformation parameters and external force direction, substantially improving the physical realism of the generated videos.
当前视频扩散模型虽然在外观合成上取得了显著进展，但在物理一致性方面仍存在严重缺陷：物体运动不符合物理规律、碰撞缺乏真实弹性响应、材质属性无法在动态行为中体现。现有方案或依赖显式物理求解器（限制灵活性），或缺乏连续的属性控制（难以精确调节）。Current video diffusion models have made striking progress in appearance synthesis, yet they remain seriously deficient in physical consistency: object motion violates physical laws, collisions lack realistic elastic responses, and material properties are not reflected in dynamic behavior. Existing solutions either rely on explicit physics solvers (limiting flexibility) or lack continuous property control (making precise tuning difficult).
"物体发生漂移，碰撞缺乏真实弹性，材质响应往往与其底层属性不匹配。"——原文"Objects drift, collisions lack realistic elasticity, and material responses often do not match their underlying properties." — from the paper
图1：PhyCo 可控物理生成示例。同一场景下，通过调节摩擦系数（friction μf）、弹性系数（restitution e）、形变参数（deformation）以及外力方向和大小（force magnitude & direction），生成的视频呈现出截然不同的物理行为。所有控制均通过像素对齐的物理属性图（property map）输入，无需在推理时调用仿真器。Figure 1: Examples of PhyCo controllable physics generation. Within the same scene, adjusting the friction coefficient (friction μf), the restitution coefficient (restitution e), the deformation parameters (deformation) and the direction and magnitude of the external force (force magnitude & direction) yields videos with markedly different physical behavior. All controls are supplied as pixel-aligned physical property maps, with no simulator call required at inference time.
现有方法的不足Shortcomings of existing methods
显式物理方法Explicit physics methods
PhysGen、PhysDreamer 依赖物理求解器PhysGen and PhysDreamer rely on physics solvers
推理时需重建几何结构，计算代价高Geometry must be reconstructed at inference time, which is computationally expensive
难以泛化到多样化场景Hard to generalize to diverse scenes
隐式/提示词方法Implicit / prompt-based methods
PhysCtrl、VLIPP 缺乏连续属性控制PhysCtrl and VLIPP lack continuous property control
Force Prompting 仅限单一外力属性Force Prompting is limited to a single force property
无法同时控制摩擦、弹性、形变等多种属性Cannot control friction, restitution, deformation and other properties simultaneously
100K+高保真物理仿真训练视频High-fidelity physics-simulated training videos
36.3Physics-IQ Score（测试外推，新SOTA）Physics-IQ Score (test-time extrapolation, new SOTA)
15.2°真实视频力方向误差（vs. 40.5° 基线）Force-direction error on real videos (vs. 40.5° baseline)
02方法Method
PhyCo 采用两阶段训练范式：（1）基于 ControlNet 的物理监督微调（Physics-Supervised Fine-Tuning），通过像素对齐的物理属性图条件化预训练扩散模型；（2）基于视觉语言模型（VLM）的奖励优化，进一步提升物理属性的可控性与准确度。PhyCo adopts a two-stage training paradigm: (1) ControlNet-based Physics-Supervised Fine-Tuning, which conditions a pretrained diffusion model on pixel-aligned physical property maps; (2) reward optimization based on a vision-language model (VLM), which further improves the controllability and accuracy of physical properties.
图2：PhyCo 两阶段训练流程。第一阶段：基于 Kubric + PyBullet + Blender 构建物理仿真数据集，通过 ControlNet 架构将物理属性图作为条件输入，对冻结的 Cosmos-Predict2 基础模型进行微调。第二阶段：对 Qwen2.5-VL-3B 进行物理问答微调（VLM fine-tuning），再将其作为奖励模型通过端到端反向传播优化 ControlNet 分支的参数，提高属性控制精度。Figure 2: The two-stage training pipeline of PhyCo. Stage one: a physics simulation dataset is built with Kubric + PyBullet + Blender, and physical property maps are fed as conditioning input through a ControlNet architecture to fine-tune the frozen Cosmos-Predict2 base model. Stage two: Qwen2.5-VL-3B is fine-tuned on physics question answering (VLM fine-tuning) and then serves as a reward model, optimizing the parameters of the ControlNet branches through end-to-end backpropagation to improve property control accuracy.
数据集基于 Kubric 框架，使用 PyBullet 物理引擎与 Blender 渲染器构建，包含六类控制场景：砖块在平面滑动（摩擦）、球弹离墙壁（弹性）、球垂直弹跳、软球自由下落、物体撞击可形变体，以及台球多球碰撞。每类场景对物理属性、物体颜色、表面材质、相机位姿和 50 种 HDRI 光照环境进行系统性随机化，最终生成超过 100K 条视频。The dataset is built on the Kubric framework with the PyBullet physics engine and the Blender renderer, and covers six categories of controlled scenarios: a brick sliding on a plane (friction), a ball bouncing off a wall (restitution), a ball bouncing vertically, a soft ball in free fall, an object striking a deformable body, and multi-ball billiard collisions. For each category, physical properties, object colors, surface materials, camera poses and 50 HDRI lighting environments are systematically randomized, ultimately yielding more than 100K videos.
物理属性被编码为像素空间内的圆形 blob 表示，归一化至 [-1, 1]。属性分为三组：Physical properties are encoded as circular blobs in pixel space, normalized to [-1, 1]. They fall into three groups:
接触属性：摩擦系数（μf）和弹性系数（e）Contact properties: friction coefficient (μf) and restitution coefficient (e)
外力属性：力的大小（F）及方向（cos φ、sin φ）Force properties: force magnitude (F) and direction (cos φ, sin φ)
每组属性对应一个独立的 ControlNet 分支，仅更新 ControlNet 层权重，基础扩散模型保持冻结。训练序列为 57 帧 @ 24 FPS，使用标准扩散 score-matching 目标函数。Each property group has its own ControlNet branch; only the ControlNet layer weights are updated, while the base diffusion model stays frozen. Training sequences are 57 frames @ 24 FPS, using the standard diffusion score-matching objective.图3：PhyCo 仿真数据集示例。六类场景分别对应不同的物理动力学模式，系统性覆盖摩擦、弹性、形变和外力等属性的多样化取值范围，确保训练数据的物理多样性与视觉可观测性。Figure 3: Examples from the PhyCo simulation dataset. The six scenario categories correspond to different physical dynamics modes and systematically cover a diverse range of values for friction, restitution, deformation and external force, ensuring both physical diversity and visual observability of the training data.
阶段二：VLM 奖励优化Stage 2: VLM reward optimization
单步重建优化因其全局轨迹编码特性，往往产生模糊的细节表现，不足以精确学习物理属性控制。PhyCo 引入 NN 步去噪 rollout，生成与推理阶段一致的预测潜变量，再通过 VLM 奖励对其进行反馈优化。Because single-step reconstruction optimization encodes the trajectory globally, it tends to produce blurry details and is insufficient for precisely learning physical property control. PhyCo introduces an NN-step denoising rollout that yields predicted latents consistent with the inference stage, which are then refined through VLM reward feedback.
VLM 奖励模型基于 Qwen2.5-VL-3B，在 PhyCo 数据集上进行物理问答（physics-related queries）微调（LoRA rank=64，200 步，4×H100），在 100 次迭代内达到约 85% 的预测准确率。奖励函数采用正确/错误答案 logit 差的 binary cross-entropy：The VLM reward model is based on Qwen2.5-VL-3B and fine-tuned on physics-related queries over the PhyCo dataset (LoRA rank=64, 200 steps, 4×H100), reaching about 85% prediction accuracy within 100 iterations. The reward function is a binary cross-entropy on the logit gap between correct and incorrect answers:
ℒVLM = −∑i log σ(ζ₊(i) − ζ₋(i))
VLM 奖励优化阶段使用 8×H200 GPU 训练 100 次迭代（约 70 分钟），峰值显存 115 GB VRAM，通过端到端反向传播更新 DiT backbone 和 ControlNet 分支。The VLM reward optimization stage runs 100 iterations on 8×H200 GPUs (about 70 minutes) with a peak memory footprint of 115 GB VRAM, updating the DiT backbone and the ControlNet branches through end-to-end backpropagation.
图4：PhyCo 真实场景生成结果。在真实图像输入条件下，PhyCo 能够根据指定的物理属性生成符合物理规律的动态视频。不同行对应不同的属性控制条件，可观察到运动轨迹、碰撞弹性与形变行为均与输入属性高度一致。Figure 4: PhyCo generation results on real scenes. Given real image inputs, PhyCo generates physically plausible dynamic videos according to the specified physical properties. Different rows correspond to different property control conditions; motion trajectories, collision elasticity and deformation behavior all closely match the input properties.
03实验Experiments
PhyCo 在 Physics-IQ 基准、用户研究（2AFC）、合成数据可控性消融以及跨架构泛化四个维度进行了系统评估。基线方法包括 SVD-XT、VLIPP、Cosmos-Predict2（基础模型）、CogVideoX-I2V-5B 和 Force Prompting。PhyCo is systematically evaluated along four dimensions: the Physics-IQ benchmark, a user study (2AFC), controllability ablations on synthetic data, and cross-architecture generalization. Baselines include SVD-XT, VLIPP, Cosmos-Predict2 (base model), CogVideoX-I2V-5B and Force Prompting.
Physics-IQ 评估模型在固体力学、流体力学、光学、磁学和热力学五个物理领域的理解能力。下表为测试外推条件（120 帧，超出训练的 57 帧）下的结果：Physics-IQ evaluates a model's understanding across five physical domains: solid mechanics, fluid mechanics, optics, magnetism and thermodynamics. The table below reports results under test-time extrapolation (120 frames, beyond the 57 frames used in training):
方法Method
固体力学Solid mechanics
流体力学Fluid mechanics
光学Optics
磁学Magnetism
热力学Thermodynamics
IQ Score
SVD-XT
21.9
20.5
6.8
8.4
17.1
19.1
VLIPP
42.3
34.1
16.9
13.4
8.8
34.6
Ours（仅文本）Ours (text only)
36.5
28.9
18.9
12.6
32.0
30.9
Ours（ControlNet）Ours (ControlNet)
42.3
30.7
19.3
12.6
40.1
35.3
Ours（ControlNet + VLM）Ours (ControlNet + VLM)
44.1
31.2
20.1
17.2
33.1
36.3
训练条件下 Physics-IQ（57 帧 @ 24 FPS）Physics-IQ under the training condition (57 frames @ 24 FPS)
方法Method
固体力学Solid mechanics
流体力学Fluid mechanics
光学Optics
磁学Magnetism
热力学Thermodynamics
平均Average
Ours（ControlNet + VLM）Ours (ControlNet + VLM)
53.1
44.3
20.3
20.8
35.9
43.6
用户研究（2AFC，16 名参与者，39 组视频对）User study (2AFC, 16 participants, 39 video pairs)
用户被要求在两段视频中选出物理行为更真实的一段（2AFC 设计）。PhyCo 在所有属性维度均显著优于基线：Participants were asked to pick the more physically realistic of two videos (2AFC design). PhyCo significantly outperforms the baselines on every property dimension:
对比方案Comparison
摩擦（Friction）Friction
弹性（Restitution）Restitution
形变（Deformation）Deformation
外力（Force）Force
PhyCo vs. CogVideoX-I2V-5B
95.5%
100.0%
82.2%
91.1%
PhyCo vs. Cosmos-Predict2
100.0%
93.2%
91.3%
86.4%
PhyCo vs. Force Prompting
—
—
—
71.7%
合成数据可控性消融Controllability ablation on synthetic data
评估各方法对输入物理属性的量化遵从误差（越低越好）：Quantitative compliance error of each method with respect to the input physical properties (lower is better):
方法Method
力大小误差Force magnitude error
摩擦误差Friction error
力方向误差（°）Force direction error (°)
弹性误差Restitution error
形变误差Deformation error
基础模型（零样本）Base model (zero-shot)
0.38
0.33
91.87
0.40
0.45
仅文本微调Text-only fine-tuning
0.31
0.30
40.35
0.31
0.14
ControlNet（−VLM）ControlNet (−VLM)
0.33
0.24
38.05
0.28
0.14
ControlNet（+VLM）ControlNet (+VLM)
0.28
0.20
22.53
0.16
0.10
真实视频力方向遵从性Force-direction compliance on real videos
在 25 段真实世界视频上测试力方向控制精度：PhyCo 平均方向误差为 15.2°，Force Prompting 为 40.5°，PhyCo 表现"显著更低的平均方向误差"。Force-direction control accuracy was tested on 25 real-world videos: PhyCo attains a mean direction error of 15.2°, versus 40.5° for Force Prompting, showing a "significantly lower average direction error".
跨架构泛化（Wan2.2 + 文本条件）Cross-architecture generalization (Wan2.2 + text conditioning)
方法Method
固体力学Solid mechanics
流体力学Fluid mechanics
光学Optics
磁学Magnetism
热力学Thermodynamics
平均Average
Wan2.2（零样本）Wan2.2 (zero-shot)
34.3
35.2
18.1
10.7
36.0
30.5
PhyCo 数据集微调Fine-tuned on the PhyCo dataset
42.1
37.6
21.9
12.2
22.1
35.1
跨架构泛化平均提升 4.6%，验证了 PhyCo 数据集在不同模型架构上的有效性。Cross-architecture generalization raises the average by 4.6%, confirming the effectiveness of the PhyCo dataset across different model architectures.
图6：与基线方法的定性对比。在相同的物理属性输入条件下，PhyCo（最右列）生成的视频在碰撞弹性、摩擦减速和形变响应等方面均更符合物理规律，而基线方法（CogVideoX-I2V-5B、Cosmos-Predict2）的动态行为与目标物理属性存在明显偏差。Figure 6: Qualitative comparison with baseline methods. Under identical physical property inputs, the videos generated by PhyCo (rightmost column) follow physical laws more closely in collision elasticity, friction-induced deceleration and deformation response, whereas the dynamics of the baselines (CogVideoX-I2V-5B, Cosmos-Predict2) deviate noticeably from the target physical properties.
消融分析小结Summary of the ablation analysis
消融实验表明：（1）ControlNet 物理属性条件相比纯文本提示在可控性上有显著提升，尤其体现在力方向误差从 40.35° 降至 38.05°；（2）VLM 奖励优化进一步大幅降低所有属性误差，力方向误差进一步降至 22.53°，弹性误差从 0.28 降至 0.16，充分验证了两阶段设计的必要性。The ablations show that: (1) conditioning on ControlNet physical properties markedly improves controllability over pure text prompts, most visibly in the force-direction error dropping from 40.35° to 38.05°; (2) VLM reward optimization further reduces every property error by a large margin, lowering the force-direction error to 22.53° and the restitution error from 0.28 to 0.16, which fully validates the necessity of the two-stage design.
04局限性Limitations
说明：以下局限性均来自论文原文明确陈述（stated by authors），并非推断。Note: All limitations below are explicitly stated by the authors in the paper, not inferred.
物理近似而非精确复现Physical approximation rather than exact reproduction
生成的动态效果是对真实物理的近似，而非精确再现。论文原文指出："生成的动态是对真实物理的近似，而非精确的物理再现"（"an approximation of real physics rather than an accurate reproduction"）。The generated dynamics approximate real physics rather than reproduce it exactly. The paper states verbatim: "an approximation of real physics rather than an accurate reproduction".
物理先验仅覆盖简化的刚体与软体行为Physical priors cover only simplified rigid-body and soft-body behavior
PhyCo 所学习的物理先验主要针对受控场景中的刚体和软体运动，对复杂交互——"关节运动（articulated motion）、流固耦合（fluid-structure coupling）或多接触动力学（multi-contact dynamics）"——的建模仍然不完整。The physical priors learned by PhyCo mainly target rigid-body and soft-body motion in controlled scenes; the modeling of complex interactions — "articulated motion, fluid-structure coupling or multi-contact dynamics" — remains incomplete.
空间属性图不强制守恒律Spatial property maps do not enforce conservation laws
像素对齐的属性图无法严格保证动量守恒和形变能量守恒，在某些情况下会产生"细微但可观测的物理偏差"（"subtle but noticeable physical deviations"）。Pixel-aligned property maps cannot strictly guarantee conservation of momentum or of deformation energy, and in some cases produce "subtle but noticeable physical deviations".
高频结构区域存在闪烁伪影Flickering artifacts in high-frequency structure regions
在运动剧烈的区域，尤其是细薄或高频纹理结构（thin or high-frequency structures）附近，生成视频可能出现闪烁伪影。论文指出该问题可通过更高训练帧率和更强主干网络加以缓解，但尚未完全解决。In regions with strong motion, especially near thin or high-frequency textured structures (thin or high-frequency structures), the generated videos may exhibit flickering artifacts. The paper notes that this can be mitigated by a higher training frame rate and a stronger backbone, but it has not been fully resolved.
计算资源需求较高High compute requirements
ControlNet 微调需要 4×H100 GPU（约半天，约 45 GB/GPU）；VLM 奖励优化需要 8×H200 GPU（约 70 分钟，峰值 115 GB VRAM）。对普通研究者的复现构成一定门槛。ControlNet fine-tuning requires 4×H100 GPUs (about half a day, roughly 45 GB per GPU); VLM reward optimization requires 8×H200 GPUs (about 70 minutes, peak 115 GB VRAM). This poses a real barrier to reproduction for ordinary researchers.
