Puffin: Thinking with Camera

01 动机Motivation

相机几何理解（从图像估计 roll/pitch/FoV 等参数）与可控图像生成（按指定视角合成场景）是空间智能的两大基石，但长期以来被作为独立任务研究。现有方法将相机参数当作辅助数字标签，忽视了其本身作为模态的语义价值，导致模型在追求语义对齐时忽略精确的空间约束，性能次优。Camera geometry understanding (estimating parameters such as roll/pitch/FoV from an image) and controllable image generation (synthesizing a scene from a specified viewpoint) are two cornerstones of spatial intelligence, yet they have long been studied as separate tasks. Existing methods treat camera parameters as auxiliary numerical labels and overlook their semantic value as a modality in their own right, so models chasing semantic alignment neglect precise spatial constraints and remain suboptimal.

"Camera-centric understanding and generation are two cornerstones of spatial intelligence, yet they are typically studied in isolation."

核心挑战在于模态鸿沟：相机参数是抽象的数值（如 FoV=72°），缺乏视觉语义，而大型多模态模型（LMM）的训练目标是语义对齐。现有做法要么独立训练专用的几何估计器，要么将参数硬编码为辅助标签，均未能充分利用 LMM 的视觉-语言推理能力。Puffin 提出将相机参数视为一种语言，用摄影专业术语（如 "tilt-up"、"Dutch angle"、"close-up"）作为中间表示，弥合数值几何与高层语义之间的差距。The core challenge is the modality gap: camera parameters are abstract numbers (e.g. FoV=72°) that carry no visual semantics, whereas large multimodal models (LMMs) are trained for semantic alignment. Existing practice either trains a dedicated geometry estimator in isolation or hard-codes the parameters as auxiliary labels, and neither fully exploits the vision-language reasoning ability of LMMs. Puffin instead treats camera parameters as a kind of language, using professional photographic terms (such as "tilt-up", "Dutch angle", "close-up") as an intermediate representation that bridges numerical geometry and high-level semantics.

02 方法Method

Puffin 以自回归语言回归（camera understanding）和扩散模型生成（camera-controllable generation）为双主干，通过共享的"Thinking with Camera"链式推理将二者统一。整个系统基于四阶段渐进式训练策略，在 Puffin-4M 数据集上训练，共耗时约 4 天（64 A100 GPU）。Puffin rests on two backbones — autoregressive language regression (camera understanding) and diffusion-based generation (camera-controllable generation) — and unifies them through a shared "Thinking with Camera" chain of reasoning. The whole system follows a four-stage progressive training strategy and is trained on the Puffin-4M dataset, taking about 4 days in total (64 A100 GPUs).

Thinking with Camera 机制Mechanism

该机制的核心思想是将三类空间先验编码进 chain-of-thought 推理链：The core idea of this mechanism is to encode three kinds of spatial prior into the chain-of-thought reasoning chain:

• Spatially Grounded Visual Cues（空间视觉线索）：天空、天花板、地面等无纹理区域虽缺乏局部特征，却蕴含垂直方向的强先验，对 pitch 估计尤为关键。Puffin 将这类隐含先验显式化，嵌入"thinking caption"。
Spatially Grounded Visual Cues: textureless regions such as sky, ceiling and ground lack local features, yet they carry a strong prior on the vertical direction and are especially critical for pitch estimation. Puffin makes such implicit priors explicit and embeds them into the "thinking caption".
• Professional Photographic Terms（摄影专业术语）：用 "close-up"、"tilt-up"、"Dutch angle" 等摄影师常用词汇替代裸数值，与 LMM 训练语料中的知识高度对齐，充当几何与语义之间的桥梁。
Professional Photographic Terms: replacing bare numbers with vocabulary commonly used by photographers, such as "close-up", "tilt-up" and "Dutch angle", aligns closely with the knowledge in LMM training corpora and serves as a bridge between geometry and semantics.
• Geometric Context（几何上下文）：将 roll/pitch/FoV 三个维度解耦，每个参数锚定到语义描述符上，最终以连续 latent 形式送入扩散模型，提供细粒度空间先验。
Geometric Context: the roll/pitch/FoV dimensions are decoupled, each parameter is anchored to a semantic descriptor, and the result is finally fed into the diffusion model as a continuous latent that supplies a fine-grained spatial prior.

模型架构Model Architecture

Puffin 由四个核心模块构成：Puffin comprises four core modules:

• Geometry-aligned Vision Encoder：采用 C-RADIOv3-H，专为几何特征保真设计，避免标准视觉编码器压缩几何信息导致的性能下降。直接 fine-tune InternVL3 或 Qwen2.5-VL 等通用 VLM 的实验表明，其 "underperforms specialized vision-only baselines due to compressed features lacking geometric fidelity"。
Geometry-aligned Vision Encoder: C-RADIOv3-H is adopted, designed for geometric feature fidelity, avoiding the performance drop caused by standard vision encoders that compress geometric information. Experiments that directly fine-tune general-purpose VLMs such as InternVL3 or Qwen2.5-VL show that such a model "underperforms specialized vision-only baselines due to compressed features lacking geometric fidelity".
• LLM Backbone：Qwen2.5-1.5B，负责语言推理与参数回归；通过 64 个可学习 query 的 connector 与视觉编码器对接。
LLM Backbone: Qwen2.5-1.5B, responsible for language reasoning and parameter regression; it interfaces with the vision encoder through a connector of 64 learnable queries.
• Diffusion Backbone：SD3-Medium，负责相机可控图像生成；接受文本条件与 pixel-wise camera map（perspective field）作为空间先验。
Diffusion Backbone: SD3-Medium, responsible for camera-controllable image generation; it accepts text conditions and a pixel-wise camera map (perspective field) as the spatial prior.
• 四阶段训练：Stage I（Alignment，冻结基础模型，训练 connector）→ Stage II（SFT，解冻全部模块）→ Stage III（Thinking SFT，引入空间推理 caption）→ Stage IV（Instruction Tuning，跨视角任务）。
Four-stage training: Stage I (Alignment, freeze the base models and train the connector) → Stage II (SFT, unfreeze all modules) → Stage III (Thinking SFT, introduce spatial reasoning captions) → Stage IV (Instruction Tuning, cross-view tasks).

Puffin-4M 数据集Dataset

从约 20 万张全景图（panoramic images）出发，通过透视投影裁切构造 400 万个 vision-language-camera 三元组，覆盖室内/室外多样场景，参数范围为 roll/pitch ∈ [−45°, 45°]、FoV ∈ [20°, 105°]、yaw ∈ [0°, 360°]。每个样本包含：精确相机参数、场景文字描述、pixel-wise perspective field map、以及空间推理 caption（用于 Thinking SFT）。Starting from about 200000 panoramic images, perspective-projection cropping is used to construct 4000000 vision-language-camera triplets covering diverse indoor and outdoor scenes, with parameter ranges roll/pitch ∈ [−45°, 45°], FoV ∈ [20°, 105°] and yaw ∈ [0°, 360°]. Each sample contains precise camera parameters, a text description of the scene, a pixel-wise perspective field map, and a spatial reasoning caption (used for Thinking SFT).

03 实验Experiments

评估分两大任务：（1）相机几何理解，在 MegaDepth、TartanAir、LaMAR 三个公开数据集及自建 Puffin-Und 基准上评估 roll/pitch/FoV 误差；（2）相机可控图像生成，在自建 Puffin-Gen 基准（650 个 caption-camera 对）上评估空间精度（Up Vector/Latitude/Gravity 误差）与视觉质量（FID）。The evaluation covers two tasks: (1) camera geometry understanding, where roll/pitch/FoV errors are measured on three public datasets — MegaDepth, TartanAir and LaMAR — as well as on the in-house Puffin-Und benchmark; (2) camera-controllable image generation, where spatial accuracy (Up Vector/Latitude/Gravity error) and visual quality (FID) are measured on the in-house Puffin-Gen benchmark (650 caption-camera pairs).

相机理解：与专用方法对比（Table 3）Camera understanding: comparison with specialized methods (Table 3)

相机可控生成：与生成方法对比（Table 4）Camera-controllable generation: comparison with generative methods (Table 4)

消融实验（Table 5）Ablation study (Table 5)

消融实验在自建 Puffin-Und 基准上进行，验证 Thinking with Camera 机制及架构选择的重要性：The ablation study is conducted on the in-house Puffin-Und benchmark to verify the importance of the Thinking with Camera mechanism and of the architectural choices:

消融结论：（1）直接 fine-tune 通用 VLM（InternVL3、Qwen2.5-VL）由于视觉特征经过语义压缩，反而不如专用视觉编码器；（2）Thinking with Camera 机制在所有三个指标上均带来稳定提升；（3）统一训练（理解+生成）相比各自独立训练存在正向互促。Ablation conclusions: (1) directly fine-tuning general-purpose VLMs (InternVL3, Qwen2.5-VL) is in fact worse than a dedicated vision encoder, because their visual features have been semantically compressed; (2) the Thinking with Camera mechanism brings consistent gains on all three metrics; (3) unified training (understanding + generation) is positively mutually reinforcing compared with training each task separately.

04 局限性Limitations