Efficient Artistic Mesh Generation via MeshVAE and Flow-based Diffusion Transformer
Weiyu Li · Antoine Toisoul · Tom Monnier · Roman Shapovalov · Rakesh Ranjan · Ping Tan · Andrea Vedaldi | Meta AI, HKUST
MeshFlow 提出了一套新的 3D 网格生成框架：用 MeshVAE 将顶点位置、法线与拓扑关系压缩进紧凑的连续隐空间，再用 Rectified Flow Diffusion Transformer (DiT) 并行去噪所有 latent token，最终解码出显式的顶点与边，得到适合下游 3D 工作流的 artist-like 网格。整套流程彻底绕开了自回归（AR）方法的逐 token 预测，推理时间约 1.2 秒，比最快的 AR 方法快 18×。MeshFlow proposes a new 3D mesh generation framework: MeshVAE compresses vertex positions, normals and topological relations into a compact continuous latent space, and a Rectified Flow Diffusion Transformer (DiT) then denoises all latent tokens in parallel, finally decoding explicit vertices and edges to obtain artist-like meshes suited to downstream 3D workflows. The whole pipeline entirely bypasses the per-token prediction of auto-regressive (AR) methods, with an inference time of about 1.2 s, 18× faster than the fastest AR method.
现有的 artist-like 网格生成方法大多采用 Auto-Regressive (AR) next-token prediction。AR 方法在两个维度上存在根本性瓶颈：推理开销随网格规模呈二次方增长，且必须对顶点坐标做离散化量化，引入不可避免的量化误差。MeshFlow 从这两个根本问题出发，提出了一套连续隐空间 + 并行生成的替代方案。Most existing artist-like mesh generation methods adopt Auto-Regressive (AR) next-token prediction. AR methods face fundamental bottlenecks along two dimensions: inference cost grows quadratically with mesh size, and vertex coordinates must be discretized and quantized, introducing unavoidable quantization error. Starting from these two root problems, MeshFlow proposes an alternative built on a continuous latent space plus parallel generation.
"Current mesh generators often adopt Auto-Regressive (AR) next-token prediction, a natural choice given the discrete nature of mesh topology. However, AR methods scale poorly because the inference cost is quadratic in mesh size. They also require discretizing the vertex coordinates, which introduces quantization errors."
Figure 1 (Teaser).MeshFlow 从噪声 latent 出发，经 flow-based transformer 并行去噪，约 1.2 秒内解码出高质量 artist-like 三角网格。生成的网格保留显式顶点与边，直接适用于下游 3D 工作流（材质、渲染、动画等）。Starting from noise latents, MeshFlow denoises in parallel with a flow-based transformer and decodes a high-quality artist-like triangle mesh in about 1.2 s. The generated mesh retains explicit vertices and edges, and is directly usable in downstream 3D workflows (materials, rendering, animation, etc.).
~1.2s单个网格推理时间（采样 + 提取）Inference time for a single mesh (sampling + extraction)
18×比最快 AR 方法（FastMesh-V1K）快Faster than the fastest AR method (FastMesh-V1K)
512MeshVAE latent codes 数量Number of MeshVAE latent codes
0.014MeshVAE Compression Ratio（vs AR 方法 0.22–0.47）MeshVAE Compression Ratio (vs 0.22–0.47 for AR methods)
与 AR 方法相比，MeshFlow 的核心优势体现在三点：Compared with AR methods, MeshFlow offers three core advantages:
无量化（No Quantization）：顶点坐标以连续浮点数表示，保留精细几何细节。No Quantization: vertex coordinates are represented as continuous floating-point numbers, preserving fine geometric detail.
线性扩展（Linear Scaling）：DiT 对所有 latent token 并行去噪，推理开销随网格规模线性增长，而非 AR 的二次方增长。Linear Scaling: the DiT denoises all latent tokens in parallel, so inference cost grows linearly with mesh size instead of quadratically as in AR.
Artist-like 输出：解码后保留显式顶点与边，适合下游 3D 工作流。Artist-like output: explicit vertices and edges are retained after decoding, suiting downstream 3D workflows.
02方法 MethodMethod
MeshFlow 由两个核心模块串联而成：MeshVAE（将离散网格压缩为连续隐向量）和 Flow-based Diffusion Transformer（在隐空间并行去噪生成）。MeshFlow chains two core modules: MeshVAE (compressing a discrete mesh into continuous latent vectors) and a Flow-based Diffusion Transformer (generating by parallel denoising in the latent space).
Figure 2 (方法总览). MeshFlow 将顶点位置、法线以及离散的 adjacency 关系一同编码进 MeshVAE 的连续隐空间；Flow-based DiT 对所有 latent token 并行去噪；最后由 Mesh Decoder 恢复出顶点、法线与完整的网格连通性。Figure 2 (Method Overview). MeshFlow encodes vertex positions, normals and discrete adjacency relations together into the continuous latent space of MeshVAE; the Flow-based DiT denoises all latent tokens in parallel; finally the Mesh Decoder recovers vertices, normals and the full mesh connectivity.
网格表示（Mesh Representation）Mesh Representation
一个三角网格 M 可以表示为顶点集 V = (v₁, …, vₙ) 和面集 F。MeshFlow 不直接对面进行编码，而是对边（edges）编码。若排除三角边环退化情形，面 F 可由边集 E 完整恢复。核心创新在于：借鉴 SpaceMesh 的思路，用每个顶点的连续 edge embedding eᵢ ∈ ℝᴰ 来隐式表达 adjacency：两顶点间距离 d(eᵢ, eⱼ) ≤ τ 则判定存在边。此外，每个顶点还携带一个 outward normal nᵢ 用以恢复面的方向。因此，整个网格被表示为三元组 (v, n, e)，全部为连续量，彻底避免了面 token 的离散化。A triangle mesh M can be written as a vertex set V = (v₁, …, vₙ) and a face set F. MeshFlow does not encode faces directly but encodes edges instead. Excluding degenerate triangle edge-cycle cases, the face set F can be fully recovered from the edge set E. The core innovation follows SpaceMesh: a continuous edge embedding eᵢ ∈ ℝᴰ per vertex implicitly expresses adjacency — an edge is deemed to exist when the distance between two vertices satisfies d(eᵢ, eⱼ) ≤ τ. In addition, each vertex carries an outward normal nᵢ to recover face orientation. The whole mesh is therefore represented as a triple (v, n, e), entirely continuous, completely avoiding discretization of face tokens.
顶点比面更紧凑：网格通常拥有 2–3 倍于顶点数的面，因此顶点级别的表示天然比面级别的 tokenizer 更短，压缩比显著更优。Vertices are more compact than faces: a mesh typically has two to three times as many faces as vertices, so a vertex-level representation is inherently shorter than a face-level tokenizer and yields a markedly better compression ratio.
MeshVAE — 连续隐空间压缩MeshVAE — Continuous Latent Space Compression
MeshVAE 的目标是将网格的三元组 (v, n, A) 压缩至低维连续隐向量 z，再从 z 解码出 (v̂, n̂, ê, m̂)。The goal of MeshVAE is to compress the mesh triple (v, n, A) into a low-dimensional continuous latent vector z, and then decode (v̂, n̂, ê, m̂) back from z.
Figure 3 (MeshVAE 详细结构). Encoder 先对顶点与法线做 Fourier Positional Encoding，再通过 TokenMerge（类似 pixel-shuffle 操作）将 N 个顶点 token 下采样为更少的 latent token n < N，随后经 Cross-Attention 和多层 Self-Attention 得到 latent z。Decoder 对称设计：SplitToken 将 n 个 latent 映射回 N 个位置，输出顶点、法线、edge embedding 与 mask。Figure 3 (MeshVAE Architecture). The encoder first applies Fourier Positional Encoding to vertices and normals, then uses TokenMerge (a pixel-shuffle-like operation) to downsample the N vertex tokens into fewer latent tokens n < N, followed by Cross-Attention and stacked Self-Attention layers to obtain the latent z. The decoder is symmetric: SplitToken maps the n latents back to N positions and outputs vertices, normals, edge embeddings and masks.
Encoder：对每个顶点拼接位置、法线的 Fourier PE 及其邻居坐标；TokenMerge 下采样；Cross-Attention + Self-Attention 得到 latent z。Encoder: for each vertex, concatenate the Fourier PE of its position and normal with its neighbour coordinates; downsample with TokenMerge; Cross-Attention + Self-Attention give the latent z.
Decoder：SplitToken 将 latent 映射回 N 个 token；Cross-Attention + learnable Positional Embedding 输出 (v̂, n̂, ê, m̂)。Decoder: SplitToken maps the latents back to N tokens; Cross-Attention + learnable Positional Embedding output (v̂, n̂, ê, m̂).
训练损失：顶点与法线 MSE + 顶点 mask BCE + 对比学习 adjacency loss（正边拉近 embedding 距离，负边推开）+ KL 正则。Training losses: MSE on vertices and normals + BCE on the vertex mask + a contrastive adjacency loss (pulling embeddings of positive edges closer and pushing negative ones apart) + KL regularization.
TokenMerge 关键性：对比 Q-Former（随机初始化可学习 query）和 FPS（最远点采样），TokenMerge 才能保证训练收敛和高保真重建（edges F1 = 99.78 vs Q-Former 的 49.47）。TokenMerge is critical: compared with Q-Former (randomly initialized learnable queries) and FPS (farthest point sampling), only TokenMerge guarantees training convergence and high-fidelity reconstruction (edges F1 = 99.78 vs 49.47 for Q-Former).
Figure 4 (VAE 对比). AR 方法必须对坐标做量化（128 级），导致精细几何细节丢失。MeshVAE 在连续空间重建，忠实保留了输入网格的精细纹理与拓扑。Figure 4 (VAE Comparison). AR methods must quantize coordinates (128 levels), which loses fine geometric detail. MeshVAE reconstructs in a continuous space and faithfully preserves the fine texture and topology of the input mesh.
生成阶段采用 Rectified Flow (RF)，其直线 ODE 公式避免了路径交叉、最小化时间步离散误差。模型训练目标为 Conditional Flow Matching (CFM)：The generation stage adopts Rectified Flow (RF), whose straight-line ODE formulation avoids path crossing and minimizes timestep discretization error. The training objective is Conditional Flow Matching (CFM):
"v_θ(RoPE3D(x_t, c_vox), t) → (ε − x₀)"
条件生成中，输入点云先做体素化，再通过 3D RoPE 位置编码与噪声 latent 融合，同时将顶点数量拼入时间 embedding 作为全局条件。DiT 采用 18 个 Transformer block、1024 维隐层，共 427M 参数；MeshVAE 的 encoder/decoder 各 8 层、1024 维，共 233M 参数。推理时还采用 Flash Attention + BF16 混合精度加速，并使用 EMA 提升稳定性与泛化。For conditional generation, the input point cloud is first voxelized and then fused with the noisy latents via 3D RoPE positional encoding, while the vertex count is concatenated into the time embedding as a global condition. The DiT uses 18 Transformer blocks with a 1024-dim hidden size, 427M parameters in total; the MeshVAE encoder/decoder each have 8 layers of width 1024, 233M parameters in total. Inference additionally uses Flash Attention + BF16 mixed precision for speed, and EMA to improve stability and generalization.
训练末期引入 logit-normal timestep 采样（借鉴 SD3），推理时采用 timestep shifting 3.0，促使模型在生成阶段更关注精细几何细节。Late in training, logit-normal timestep sampling is introduced (following SD3), and timestep shifting 3.0 is used at inference, pushing the model to attend more to fine geometric detail during generation.
对于生成结果的后处理：检测 boundary edge（仅属于一个三角面的边），若 k < 5 的边环则自动三角化修补，增强生成结果的鲁棒性。Post-processing of the generated result: boundary edges (edges belonging to only one triangle) are detected, and edge cycles with k < 5 are automatically triangulated and patched, improving the robustness of the output.
03实验 ExperimentsExperiments
评估在 Toys4K 数据集（所有对比模型均未在此训练，保证公平泛化测试）上进行，使用 Chamfer Distance (CD) 和 Hausdorff Distance (HD)（均乘以 100 缩放），以及推理时间。Evaluation is carried out on the Toys4K dataset (none of the compared models were trained on it, ensuring a fair generalization test), using Chamfer Distance (CD) and Hausdorff Distance (HD) (both scaled by 100), together with inference time.
注：Inf. Time 参照 FastMesh 的计算方式（多个对象的平均推理时间）。AR 方法在处理单个对象时往往需要约 6× 的报告时间，而 MeshFlow 保持恒定运行时间。*所有 baseline 数值引自 FastMesh 论文。Note: Inf. Time follows the protocol of FastMesh (average inference time over multiple objects). AR methods often need about 6× the reported time when processing a single object, whereas MeshFlow keeps a constant runtime. *All baseline numbers are quoted from the FastMesh paper.
单个网格推理时间对比Single-mesh Inference Time Comparison
方法Method
BPT
TreeMeshGPT
DeepMesh
FastMesh-V1K
FastMesh-V4K
MeshFlow (Ours)
推理时间Inference time
~8 min
~4 min
~50 min
~21 s
~50 s
~1.2 s
MeshVAE 重建对比MeshVAE Reconstruction Comparison
类型Type
方法Method
CD ↓ (×100)
Compression Ratio ↓
AR
TreeMeshGPT
1.63
0.22
ArAE
EdgeRunner
1.21
0.47
Diffusion
MeshVAE (Ours)
1.29
0.014
MeshVAE 的 CD 接近最优（1.29 vs 1.21），而 Compression Ratio 达到 0.014，远低于 AR 方法的 0.22–0.47，证明其表示极其紧凑。The CD of MeshVAE is near-optimal (1.29 vs 1.21), while its Compression Ratio reaches 0.014, far below the 0.22–0.47 of AR methods, showing that the representation is extremely compact.
Figure 5 (定性对比). AR 方法推理时间显著更长，且频繁因 early stopping 导致几何不完整；MeshFlow 高效生成高质量网格。Figure 5 (Qualitative Comparison). AR methods take substantially longer at inference and frequently yield incomplete geometry because of early stopping; MeshFlow generates high-quality meshes efficiently.
消融研究（Ablation）Ablation
以下表格对比了 MeshVAE 不同下采样策略对重建质量的影响（所有数值乘以 100）：The table below compares how different MeshVAE downsampling strategies affect reconstruction quality (all values scaled by 100):
TokenMerge 是确保训练收敛和高保真重建的关键。即使 4× 下采样，F1 仍达 88.82%，说明方法具有良好的压缩鲁棒性。TokenMerge is the key to ensuring training convergence and high-fidelity reconstruction. Even at 4× downsampling, F1 still reaches 88.82%, showing that the method is robust to compression.
Normal Consistency (NC) 指标说明：由于 MeshFlow 直接预测法线，NC 指标对其异常偏高，并不具有真实比较意义，故论文未报告此指标。作者指出 CD/HD 也不能完整反映网格拓扑质量，呼吁未来工作开发更全面的网格质量评测指标。On the Normal Consistency (NC) metric: because MeshFlow predicts normals directly, its NC is abnormally high and not meaningfully comparable, so the paper does not report this metric. The authors note that CD/HD also fail to fully reflect mesh topology quality, and call for future work on more comprehensive mesh quality metrics.
04局限性 LimitationsLimitations
说明：以下局限性均为作者在论文 Conclusion and Discussion 部分明确陈述。Note: all limitations below are explicitly stated by the authors in the Conclusion and Discussion section of the paper.
仅支持三角网格（Triangular Faces Only）Triangular Faces Only
当前方法假设所有面均为三角形。然而，专业美术师在制作时往往偏好 quad（四边形）等多边形，三角面的限制可能使生成结果与实际工作流有所脱节。The current method assumes that all faces are triangles. However, professional artists often prefer polygons such as quads, so the triangle-only restriction may leave generated results somewhat disconnected from real production workflows.
生成结果存在局部空洞（Holes from Inaccurate Diffusion Predictions）Holes from Inaccurate Diffusion Predictions
由于 diffusion 预测的不精确，生成的网格中偶尔出现小洞（holes）。论文采用基于短环检测（short-cycle detection）的启发式后处理进行修补。精化 diffusion 过程或使用更强的 DiT 模型可能有助于缓解此问题。Because diffusion predictions are imprecise, small holes occasionally appear in the generated meshes. The paper patches them with a heuristic post-process based on short-cycle detection. Refining the diffusion process or using a stronger DiT model may help mitigate this issue.
Figure 6 (失败案例). 部分生成结果中存在局部空洞，需依赖后处理修补。Figure 6 (Failure Cases). Some generated results contain local holes that have to be patched by post-processing.标准评测指标难以充分评估拓扑质量Standard Metrics Struggle to Assess Topology Quality
"Common metrics such as CD and HD struggle to effectively evaluate artifacts in the generated meshes, including flipped normals and holes." 未来工作应聚焦于开发能评估网格拓扑质量的指标，但这在生成任务中仍具挑战性。Future work should focus on developing metrics that can assess mesh topology quality, which remains challenging for generative tasks.
模型仅生成几何（顶点 + 连通性），未考虑 UV mapping 或材质纹理。作者指出，扩展至 UV mapping 生成以支持高质量纹理，是一个值得探索的未来方向。The model generates geometry only (vertices + connectivity), without considering UV mapping or material textures. The authors note that extending to UV mapping generation to support high-quality textures is a promising future direction.
