The Missing GAP: From Solving Square Jigsaw Puzzles to Handling Real World Archaeological Fragments

Abstract: Jigsaw puzzle solving has been an increasingly popular task in the computer vision research community. Recent works have utilized cutting-edge architectures and computational approaches to reassemble groups of pieces into a coherent image, while achieving increasingly good results on well established datasets. However, most of these approaches share a common, restricting setting: operating solely on strictly square puzzle pieces. In this work, we introduce GAP, a set of novel jigsaw puzzles datasets containing synthetic, heavily eroded pieces of unrestricted shapes, generated by a learned distribution of real-world archaeological fragments. We also introduce PuzzleFlow, a novel ViT and Flow-Matching based framework for jigsaw puzzle solving, capable of handling complex puzzle pieces and demonstrating superior performance on GAP when compared to both classic and recent prominent works in this domain.

• The paper introduces a benchmark (GAP) and a novel PuzzleFlow framework that employs Vision Transformers and flow matching to tackle irregular, eroded puzzle fragments.
• The paper demonstrates significant performance gains over legacy solvers, achieving 28.5% perfect accuracy on GAP-3 and robust spatial relationship accuracy improvements.
• The study highlights that integrating RGBA shape encoding and iterative flow matching is essential for reconstructing realistic archaeological puzzles in digital heritage.

Bridging Academic and Archaeological Jigsaw Puzzle Solving: The GAP Dataset and PuzzleFlow Framework

Motivation and Context

The computational task of jigsaw puzzle solving has evolved from its origins as a heuristic challenge into a domain deeply interlinked with digital security, unsupervised representation learning, and cultural heritage restoration. While recent advances have leveraged deep learning and transformer-based architectures, the vast majority of benchmarks and solvers operate within restrictive settings: puzzles with strictly square, non-eroded pieces and simplified gap models. This paradigm is misaligned with real-world scenarios such as archaeological artifact reconstruction, which require handling heavily eroded, irregularly-shaped fragments. "The Missing GAP: From Solving Square Jigsaw Puzzles to Handling Real World Archaeological Fragments" (2605.12077) addresses this fundamental disconnect by introducing the GAP benchmark dataset and the PuzzleFlow model, establishing a robust foundation for practical puzzle assembly in cultural heritage contexts.

GAP Dataset: Realistic Archaeological Puzzle Generation

GAP (Generated Archaeological-fragments Puzzles) is a new benchmark comprising two large-scale datasets (GAP-3 and GAP-5) featuring puzzles with synthetic fragments emulating the morphology and erosion of authentic archaeological artifacts. Fragment shapes are sampled from a variational autoencoder (VAE) trained on binary masks from the RePAIR dataset, thereby capturing the statistical geometry and edge complexity observed in real artifacts.

GAP puzzles are generated via a pipeline that overlays structured $n\times n$ grids onto diverse artwork images from the Metropolitan Museum of Art's Open Access collection, applies learned VAE masks, and extracts textured fragments. This process yields puzzles where fragment topology is grid-based, but individual pieces exhibit variable, non-linear erosion and shape, closely mirroring real-world scenarios.

Figure 1: Puzzle generation pipeline producing puzzles with archaeologically realistic fragment shapes and textures derived from the MET art collection.

Geometric validation of synthetic fragments demonstrates high fidelity in core shape properties (mean area, aspect ratio, solidity differ by $<1$-$3\%$ from real samples), with expected smoothing in edge complexity metrics (perimeter, concavities) due to VAE regularization. PCA embeddings confirm substantial distributional overlap between real and synthetic fragments, ensuring the absence of mode collapse and authentic geometric diversity.

Figure 2: PCA projection showing that VAE-generated fragments overlap real archaeological fragments, validating morphological realism.

Figure 3: Comparison of geometric properties (area, solidity, edge complexity) between real and synthetic fragments, indicating high similarity.

The GAP datasets, with 20,000 puzzles each, provide a scalable, controlled yet realistic testbed for algorithmic development and evaluation. Their public availability ensures reproducible research and establishes a new community benchmark for irregular, eroded puzzle assembly.

PuzzleFlow: Holistic Visual Reasoning via Flow Matching

PuzzleFlow presents a novel framework for jigsaw puzzle assembly leveraging Vision Transformers and discrete flow matching over the permutation space of fragment placements. Unlike prior approaches reliant on boundary compatibility or local features, PuzzleFlow is designed to enable global relational reasoning across arbitrary fragment geometries.

The model utilizes a pretrained ViT backbone, adapted to process RGBA fragments (where alpha encodes shape masks), and augments embeddings with positional and flow time information. Fragment features are combined and passed through four additional transformer layers, facilitating cross-fragment attention and contextual refinement. Flow-matching enables iterative denoising from random permutations toward the ground truth configuration through stochastic interpolation, with a training objective composed of time-dependent cross-entropy losses.

Figure 4: PuzzleFlow architecture allowing fragment-level visual encoding, positional conditioning, and relational transformer reasoning with iterative flow matching.

This design is critical for handling irregular, eroded fragments where edge information is unreliable, boundary continuity is broken, and spatial structure must be inferred holistically.

Experimental Results and Benchmarking

PuzzleFlow and seven baselines (classical, deep learning) were evaluated extensively on GAP-3 and GAP-5. Evaluation metrics include Perfect Accuracy (PA), Absolute Accuracy (AA), and Spatial Relationship Accuracy (SRA).

Key numerical findings:

• On GAP-3, classical solvers and most deep learning baselines achieve near-random accuracy (PA/AA $\approx$ 0-15\%), confirming that boundary-based approaches are inadequate.
• PuzzleFlow achieves the highest results (PA: 28.5\%, AA: 62.9\%, SRA: 55.7\%), significantly outperforming FCViT (PA: 25.2\%, AA: 60.7\%, SRA: 47.6\%) and DiffAssemble (PA: 16.4\%, AA: 50.5\%, SRA: 43.4\%).
• On GAP-5 (25 pieces), combinatorial complexity increases substantially ($25!$ configurations). PuzzleFlow leads in all metrics (PA: 0.3\%, AA: 29.1\%, SRA: 19.8\%) while state-of-the-art baselines degrade to near-random accuracy.
• PuzzleFlow's performance demonstrates robust spatial coherence, as evidenced by SRA improvements over all baselines.

Ablation studies further substantiate critical design choices:

• Fine-tuning the ViT backbone accounts for the largest accuracy improvement (+21.1 PA points).
• RGBA adaptation (utilizing alpha channel for shape encoding) is essential for irregular fragments (+19.3 PA points).
• Iterative flow matching yields consistent performance gains over direct prediction.
• Moderate architectural depth (4 transformer layers) balances expressivity and computational efficiency.

Qualitative analysis showcases the efficacy of PuzzleFlow in reconstructing GAP puzzles with irregular, eroded fragments, with failure cases often arising from extreme erosion or repetitive textures.

Figure 5: PuzzleFlow reconstructs puzzles with irregular fragments; failure cases occur under extreme erosion or visual ambiguity.

Practical and Theoretical Implications

The introduction of the GAP datasets and PuzzleFlow architecture closes a longstanding gap between academic puzzle benchmarks and practical archaeological reconstruction. GAP exposes the limitations of legacy solvers and establishes a new baseline for algorithmic evaluation on realistic, eroded fragments. PuzzleFlow's permutation-based flow matching and holistic relational reasoning represent meaningful advances toward practical digital heritage workflows.

Implications:

• Practical: GAP enables systematic testing and development of artifact reconstruction algorithms at scale, allowing researchers to benchmark under conditions that approximate real heritage restoration tasks.
• Theoretical: PuzzleFlow's generative modeling over permutation spaces via flow matching may have broader applications in combinatorial optimization and set ordering tasks, transcending jigsaw puzzle assembly.

Future research may extend PuzzleFlow to support missing fragments, non-grid spatial arrangements, physical constraints, and scalable inference for puzzles with hundreds or thousands of pieces. Integrating multimodal cues (material properties, object semantics) and applying these methods to reconstructed archaeological assets represent promising directions.

Conclusion

The GAP dataset and PuzzleFlow framework constitute a significant step toward making computational jigsaw puzzle assembly relevant to cultural heritage and archaeological restoration. By introducing large-scale, realistically eroded, irregular puzzles and a solver capable of global visual reasoning independent of boundary continuity, this work establishes a new benchmark for the field and sets the stage for future developments in digital heritage preservation and combinatorial reasoning.