MeshFIM: Local Low-Poly Mesh Editing via Fill-in-the-Middle Autoregressive Generation

Abstract: Autoregressive (AR) models can generate high-quality low-poly meshes from point clouds, but they still operate in an all-or-nothing manner: when a local region is unsatisfactory, the entire mesh must be regenerated, wasting computation and destroying satisfactory mesh structure elsewhere. We introduce MeshFIM, a Fill-in-the-Middle (FIM) framework that regenerates a target region of a low-poly mesh conditioned on the surrounding context. MeshFIM addresses three mesh-specific challenges: enforcing exact attachment along the exposed boundary, preserving topological order in the context, and suppressing overflow beyond the intended region. It does so with five complementary design choices: boundary vertex markers, context positional embeddings, expanded context width, context augmentation, and a low-poly geometry encoder whose gated subtraction mechanism focuses generation on the missing region by leveraging the difference between the reference surface and the existing mesh. Detailed ablation studies are presented to show the effectiveness of every introduced component. Based on MeshFIM, we demonstrate two applications: interactive brush-based editing and automatic defect repair on low-poly mesh (see Figure 1). Last but not least, experiments show that MeshFIM outperforms a range of baselines in mesh refinement, mesh repair and whole mesh generation plus stitch-back scheme.

• The paper introduces an autoregressive Fill-in-the-Middle paradigm enabling precise local mesh edits that maintain seamless boundary continuity.
• It employs boundary vertex markers, context positional embeddings, and context augmentation to robustly preserve mesh topology during localized generation.
• Results show nearly perfect boundary recovery (99.75% A-VMR) and 95.54% local geometric improvement, outperforming global regeneration baselines.

MeshFIM: Local Low-Poly Mesh Editing via Fill-in-the-Middle Autoregressive Generation

Introduction and Motivation

Autoregressive mesh generation has yielded impressive advances in producing high-quality low-poly meshes from point clouds, yet the inability of existing models to perform targeted local edits restricts their applicability in practical workflows. Most current AR models necessarily regenerate an entire mesh for even small local changes, leading to inefficiency and unnecessary disruption of well-formed structures elsewhere in the mesh. MeshFIM introduces an autoregressive Fill-in-the-Middle (FIM) paradigm for meshes, enabling local region regeneration conditioned on surrounding mesh context and a reference geometry, overcoming key challenges in enforcing boundary continuity, topological preservation, and region-localized generation.

Methodology

Problem Formulation

Given an initial low-poly mesh $M$ and a selected local target region $B$, MeshFIM aims to regenerate $B$ by predicting replacement faces $B'$ such that $B'$ seamlessly attaches to the context $A \setminus B$, while adhering to the reference geometry $P$. The task is formalized as conditional infilling:

$p(B' \mid A\setminus B, P)$

where the mesh is serialized into FIM token sequences containing context, target, and sentinel tokens, with a transformer decoding $B'$ autoregressively.

Key Technical Innovations

MeshFIM addresses three fundamental challenges in local mesh editing: precise boundary attachment to avoid cracks or T-junctions, preservation of context topology amidst global serialization, and suppressing overflow of generated faces beyond the intended region. To this end, five complementary architectural mechanisms are introduced:

• Boundary Vertex Markers: Learned token-level markers identify context vertices at open boundaries, serving as explicit cues for boundary reuse during patch generation. Ablations show this contributes the largest single gain in perfect boundary match rates.
• Context Positional Embeddings: Locally learned position encodings are added to serialized tokens, restoring the spatial structure necessary to maintain edge flow and face density transitions during generation.
• Expanded Context Width: Conditioning is provided not just on immediate neighbors but on several rings of contextual faces, furnishing sufficient geometric information to constrain the generation and minimize overflow.
• Context Augmentation: During training, random noise is injected into context vertex positions to close the train-inference gap, enhancing model robustness to boundary imperfections and artifacts at test time.
• Low-Poly Geometry Encoder with Gated Subtraction: A shared Perceiver-based encoder extracts positionally aligned latents from both the reference mesh and low-poly context; a learned gating mechanism modulates the input, suppressing reference conditioning in regions already well-covered by the context, thereby spatially focusing generation on missing areas.

Serialization and Decoding

To mitigate the issue that localized surface regions may be scattered when serialized, MeshFIM adopts independent serialization of context and target regions, rather than relying on a token order dictated by global mesh lexicographic sorting. This design decouples sequence length and token locations from unknown target geometry, a critical difference from previous AR mesh models.

Results and Evaluation

Datasets and Baselines

MeshFIM is evaluated on low-poly meshes from ShapeNetV2, Objaverse, and ObjaverseXL, with ground-truth reference geometries provided. Existing methods—including classical subdivision (Modified Butterfly, Neural Subdivision), mesh repair (Liepa Hole Filling, SeMIGCN), and recent AR generative models (BPT, MeshAnythingV2) with post-hoc stitch-back—serve as baselines.

Metrics

Given the importance of both seamless boundary matching and local geometric fidelity, a suite of region-specific metrics are employed:

• A-VMR: Average boundary vertex matching ratio.
• O-CDIR: One-way Chamfer Distance Improvement Rate.
• CD-PR: Fraction of samples improving local Chamfer distance versus original.
• #F-Inc: Face increase ratio for quantifying local mesh refinement without over-generation.
• OvR/A-Overflow: Metrics for overflow quantification, measuring the proportion and extent of generated faces exceeding intended boundaries.

Comparative Performance

MeshFIM achieves nearly perfect boundary vertex recovery (A-VMR 99.75%) and a high rate of geometric improvement (O-CDIR 8.25%; CD-PR 95.54%), outperforming all baselines by wide margins. Notably, all baselines either (a) cannot operate locally, (b) fail to produce topologically/visually plausible patches in context, or (c) are computationally inefficient due to global regeneration. Ablation studies demonstrate that all five architectural contributions yield additive benefits, with boundary vertex markers and context augmentation providing especially strong gains. MeshFIM's overflow rates approach the theoretical lower bound imposed by the proximity threshold, indicating precisely localized generation. These results generalize across mesh types and editing scenarios.

Applications

MeshFIM directly supports two principal use cases:

• Interactive Brush-Based Editing: A practical web interface allows target region selection via brush strokes, with faces merged into a single connected selection obeying the topological constraints necessary for the FIM paradigm.
• Automatic Defect Repair: Automatic defect detection based on rasterization and raycasting identifies broken regions, permitting iterative local repairs. On AR-generated meshes, MeshFIM-based iterative repair fixes over 88% of severe defects with high reliability, demonstrating practical utility for quality assurance in generative pipelines.

Implications and Future Directions

MeshFIM decisively advances the state of the art in local mesh generation, enabling tools and workflows where only relevant regions are edited or repaired while preserving the surrounding mesh. The FIM-based autoregressive paradigm stands to be extended to multi-component or disconnected regions, a known limitation of the current formulation. Additionally, hybrid schemes integrating MeshFIM with differentiable simulation and semantic conditioning offer promising directions for mesh-based design and scan refinement tasks in graphics, CAD, and AR/VR content creation.

Conclusion

MeshFIM establishes a highly effective architectural framework for local low-poly mesh editing via Fill-in-the-Middle autoregressive generation. By integrating explicit boundary, topology, and region-focusing mechanisms, it enables efficient, robust, and boundary-faithful local mesh generation that outperforms alternative approaches both quantitatively and qualitatively. This development not only addresses a practical gap in 3D mesh modeling workflows but also opens up new research avenues in localized structured geometry generation (2605.08744).