Instance Jigsaw Puzzle Problem

• Instance Jigsaw Puzzle Problem is defined as reconstructing an original image from unordered fragments by assigning correct grid positions and orientations.
• Compatibility metrics, ranging from hand-crafted dissimilarity functions to deep neural network estimates, are key to evaluating adjacent-edge matches.
• Global optimization frameworks, including genetic algorithms, spectral methods, and Monte Carlo Tree Search, integrate local measures for effective reassembly.

Searching arXiv for recent and foundational papers on the instance jigsaw puzzle problem. The instance jigsaw puzzle problem concerns reconstructing an original whole from an unordered collection of fragments. In the classical square-tile formulation, the input is the unordered multiset of all $n\cdot m$ non-overlapping $K\times K$ image tiles extracted from an unknown original image, with no information about global placement, orientation, or puzzle dimensions, and the objective is to assign each tile to a grid position and orientation so that all abutting edges match (Sholomon et al., 2017). Closely related formulations distinguish Type-1 puzzles, in which piece orientations are known and fixed, from Type-2 puzzles, in which both locations and orientations are unknown (Rika et al., 2022); model the unknowns as a permutation $\sigma\in S_n$ together with rotations $R_i\in Z_4$ for each patch (Huroyan et al., 2018); extend the pieces from equal squares to convex polygons generated by straight cuts through a global polygonal shape (Harel et al., 2020); or reinterpret the “pieces” as shuffled instances in a bag of whole-slide image features and seek a permutation-restoring operator $\widehat{\mathbf P}$ (Chen et al., 10 Jul 2025).

1. Formalizations and problem variants

A standard square-piece instance consists of image tiles arranged originally on a rectangular grid. Each piece has four edges, and a valid reconstruction requires a globally consistent assignment of pieces to cells such that neighboring edges are compatible (Sholomon et al., 2017). In the terminology used by later reconstruction work, Type-1 refers to unknown locations with known orientations, whereas Type-2 additionally allows each piece to rotate by $0^\circ,90^\circ,180^\circ,270^\circ$ (Rika et al., 2022).

Several important specializations alter the observational model rather than the combinatorial core. In the graph-connection formulation, the observation is a shuffled and arbitrarily rotated set of patches $$Q=\{(R_i(P_{\sigma(i)}),(R_i\circ f)|_{P_{\sigma(i)}})\}_{i=1}^n$$, and reconstruction amounts to recovering the image $f$, or equivalently the permutation $\sigma$ and rotations $\{R_i\}$ (Huroyan et al., 2018). In Deepzzle, the task is image reassembly with wide space between fragments, missing fragments, outsider fragments, and a distinguished central fragment, which changes the scoring problem from direct border continuity to fragment-to-position prediction (Paumard et al., 2020).

The notion of an instance can also depart from square pieces entirely. Polygonal jigsaw puzzles generated by the lazy caterer model begin with a convex polygonal canvas $K\times K$0 and a set of straight cuts $K\times K$1, producing convex pieces $K\times K$2 whose vertices, edges, and internal angles become the primitive data for reconstruction (Harel et al., 2020). A different abstraction appears in the line-only puzzle setting, where each piece $K\times K$3 is a bounded planar region $K\times K$4 carrying a finite set of oriented line segments $K\times K$5, and compatibility ignores color and conventional shape cues in favor of Wertheimer’s law of good continuation (Khoroshiltseva et al., 2024).

A recent domain-specific reinterpretation arises in whole-slide image analysis. There, one starts from a bag of instance embeddings $K\times K$6, applies a random shuffling operator $K\times K$7 to obtain $K\times K$8, and trains an encoder $K\times K$9 so that $\sigma\in S_n$0 (Chen et al., 10 Jul 2025). The combinatorial object is again a hidden permutation, but the semantic content is spatial correlation among instances rather than physical edge matching.

2. Search space, structure, and hardness

The basic computational obstacle is combinatorial explosion. For square puzzles, exhaustive search over all permutations is infeasible, with complexity $\sigma\in S_n$1 (Sholomon et al., 2017). A key structural observation is that full reconstruction need not identify every correct adjacency immediately: the assembly can be viewed as selecting a spanning tree of correct adjacent-edge pairs, and only $\sigma\in S_n$2 true adjacencies are needed to connect all pieces (Sholomon et al., 2017). This is why high-precision compatibility estimates are often more valuable than uniformly high recall; the DNN-Buddies analysis notes that, in the extreme, perfect precision with recall $\sigma\in S_n$3 can still yield a correct spanning selection of edges (Sholomon et al., 2017).

Complexity-theoretic results show that this difficulty persists even in highly restricted layouts. For rotating and placing $\sigma\in S_n$4 square tiles into a $\sigma\in S_n$5 array, exact signed edge-matching (jigsaw) is NP-hard (Bosboom et al., 2016). The same work proves stronger inapproximability results: it is NP-hard to approximate the max-placement version within factor $\sigma\in S_n$6, and the max-matched variant inherits a corresponding hardness result on the number of matched edges (Bosboom et al., 2016). On the positive side, the same paper gives an easy $\sigma\in S_n$7-approximation and a $\sigma\in S_n$8-approximation for the max-tiles version (Bosboom et al., 2016).

Random models illuminate a different aspect of solvability: identifiability from local clues. In shotgun assembly of random jigsaw puzzles on an $\sigma\in S_n$9 grid whose edge colors are drawn uniformly from $R_i\in Z_4$0, earlier work had shown unique assembly for $R_i\in Z_4$1 and failure of unique edge assembly for $R_i\in Z_4$2; the improved result proves that for every fixed $R_i\in Z_4$3, if $R_i\in Z_4$4, then with high probability the puzzle has unique vertex assembly, and a deterministic algorithm with running time $R_i\in Z_4$5 reconstructs the planted assembly (Bordenave et al., 2016). This line of work is not about natural images, but it clarifies when local matching information is information-theoretically sufficient.

These hardness and identifiability results delimit what compatibility learning and global optimization can plausibly achieve. A plausible implication is that practical solvers are best understood as exploiting strong structure in natural-image statistics, restricted motion groups, or specialized acquisition models rather than circumventing the worst-case combinatorics.

3. Compatibility measures and adjacency estimation

Most modern solvers factor the problem into a local compatibility stage and a global assembly stage. Classical square-piece methods use hand-crafted dissimilarity functions on candidate abutting edges. One such boundary score compares the right edge of $R_i\in Z_4$6 to the left edge of $R_i\in Z_4$7 by

$R_i\in Z_4$8

which serves as a simple dissimilarity baseline in DNN-Buddies (Sholomon et al., 2017). The graph-connection pipeline instead initializes pairwise relations with the Mahalanobis-Gradient Compatibility metric across all four relative orientations and directions (Huroyan et al., 2018).

DNN-Buddies introduced the first deep neural network-based estimation metric for the jigsaw puzzle problem (Sholomon et al., 2017). Its feed-forward fully connected network takes 336 scalar inputs formed from the two abutting pixel columns and their immediate neighboring columns on each piece, using YUV channels, and outputs a two-way softmax indicating whether the two edges should be adjacent (Sholomon et al., 2017). Training on 970,224 labeled edge-pairs from 2,755 IAPR TC-12 images with informed undersampling yields 95.04% training accuracy and 94.62% held-out test accuracy; when used to select the top-1 most compatible edge on the 20-image, 432-piece benchmark of Cho et al., the reported precision is 94.83% (Sholomon et al., 2017).

Later neural compatibility models concentrated on eroded-boundary settings, where the outermost boundary pixels are missing or unreliable. TEN represents a piece by an embedding produced by twin CNN encoders $R_i\in Z_4$9 and $\widehat{\mathbf P}$0, compares the embeddings with Euclidean distance, and trains them with a triplet loss of margin $\widehat{\mathbf P}$1 (Rika et al., 2022). In the reported experiments, TEN-Large improves top-1 accuracy over the best classical method MGC on all three datasets, and for reconstruction with the GA-based solver raises Type-2 neighbor accuracy from 40.2% to 55.5% on DIV2K, from 47.5% to 59.4% on PIRM, and from 32.3% to 49.1% on MIT (Rika et al., 2022). The same study reports that computing all $\widehat{\mathbf P}$2 compatibility measures is 450–2770× faster than a conventional end-to-end CNN across puzzle sizes from $\widehat{\mathbf P}$3 to $\widehat{\mathbf P}$4 (Rika et al., 2022).

Edge2Vec refines the embedding approach by modifying the architecture and replacing standard triplet training with hard batch triplet loss (Rika et al., 2022). It defines compatibility as the negative Euclidean distance between learned edge embeddings, uses grouped fully connected projections to reduce parameter count, and adds an $\widehat{\mathbf P}$5 regularizer on batch embeddings (Rika et al., 2022). On the reported benchmarks, Edge2Vec improves Type-2 top-1 edge-matching accuracy from 64.1% to 71.4% over the DNN-E2E baseline, and with the GA solver improves Type-2 reconstruction neighbor accuracy from 73.3% to 82.3% (Rika et al., 2022).

Not all compatibility measures are color-based. The good-continuation framework computes a purely line-based score $\widehat{\mathbf P}$6 by matching oriented line segments across pieces with a linear assignment problem, penalizing unmatched lines and normalizing by a threshold $\widehat{\mathbf P}$7 (Khoroshiltseva et al., 2024). In polygonal puzzles, compatibility can also be defined geometrically through relaxed edge-length and angle predicates under bounded noise $\widehat{\mathbf P}$8, e.g., $\widehat{\mathbf P}$9 together with angle-sum constraints (Harel et al., 2020). This diversity of compatibility models suggests that the local signal depends strongly on the degradation model and on whether the puzzle is pictorial, apictorial, or geometric.

4. Global reconstruction and optimization frameworks

A compatibility measure alone does not solve the puzzle; it must be integrated into a global optimizer. In DNN-Buddies, the base solver is a genetic algorithm whose candidate assemblies are represented by relative edge-to-edge assignments (Sholomon et al., 2017). Its crossover operator builds one child from two parents in five ordered phases: assign all relative relations common to both parents; assign all “DNN-buddy” relations present in either parent; assign all best-buddy relations; assign the most-compatible relations by simple dissimilarity; and assign random relations until $0^\circ,90^\circ,180^\circ,270^\circ$0 relations have been selected (Sholomon et al., 2017). Elevating DNN-buddy pairs to second priority heavily biases the search toward high-precision learned adjacencies.

The graph connection Laplacian approach separates orientation recovery from shuffle recovery. It builds a weighted connection graph $0^\circ,90^\circ,180^\circ,270^\circ$1, forms the connection adjacency $0^\circ,90^\circ,180^\circ,270^\circ$2, the degree matrix $0^\circ,90^\circ,180^\circ,270^\circ$3, the graph connection weight matrix $0^\circ,90^\circ,180^\circ,270^\circ$4, and the normalized graph connection Laplacian $0^\circ,90^\circ,180^\circ,270^\circ$5 (Huroyan et al., 2018). Rotations are estimated by taking the top two eigenvectors of $0^\circ,90^\circ,180^\circ,270^\circ$6, extracting the $0^\circ,90^\circ,180^\circ,270^\circ$7 block for each piece, and projecting it onto the nearest element of $0^\circ,90^\circ,180^\circ,270^\circ$8 by Frobenius norm (Huroyan et al., 2018). Once the rotations are fixed, any standard Type-1 solver can be used to recover the permutation. The authors further describe an iterative update cycle based on a neighbor-averaged MGC score, with up to five iterations, retaining the solution with minimum global Err score (Huroyan et al., 2018).

Deepzzle changes the optimization target from local edge matching to global position assignment. A Siamese CNN predicts $0^\circ,90^\circ,180^\circ,270^\circ$9 for a fragment $$Q=\{(R_i(P_{\sigma(i)}),(R_i\circ f)|_{P_{\sigma(i)}})\}_{i=1}^n$$0 occupying position $$Q=\{(R_i(P_{\sigma(i)}),(R_i\circ f)|_{P_{\sigma(i)}})\}_{i=1}^n$$1, and these scores define a directed acyclic graph whose shortest path represents the maximum-probability assignment under a simplified product model (Paumard et al., 2020). The graph is pruned by a branch-cut threshold $$Q=\{(R_i(P_{\sigma(i)}),(R_i\circ f)|_{P_{\sigma(i)}})\}_{i=1}^n$$2: with $$Q=\{(R_i(P_{\sigma(i)}),(R_i\circ f)|_{P_{\sigma(i)}})\}_{i=1}^n$$3, inference takes approximately 20,000 s per puzzle; with $$Q=\{(R_i(P_{\sigma(i)}),(R_i\circ f)|_{P_{\sigma(i)}})\}_{i=1}^n$$4, approximately 2,000 s with no accuracy loss; with $$Q=\{(R_i(P_{\sigma(i)}),(R_i\circ f)|_{P_{\sigma(i)}})\}_{i=1}^n$$5, approximately 20 s with an approximately 1% accuracy drop (Paumard et al., 2020). This is specifically suited to settings with large gaps, outsider fragments, and missing pieces.

GANzzle reframes the problem as retrieval from a generated “mental image” rather than pairwise edge comparison (Talon et al., 2022). A permutation-invariant encoder pools piece features into a global latent code $$Q=\{(R_i(P_{\sigma(i)}),(R_i\circ f)|_{P_{\sigma(i)}})\}_{i=1}^n$$6, a multi-scale GAN reconstructs an image $$Q=\{(R_i(P_{\sigma(i)}),(R_i\circ f)|_{P_{\sigma(i)}})\}_{i=1}^n$$7, RoIAlign extracts per-slot features from a generator layer matching the puzzle grid, and a shared embedding space yields similarities $$Q=\{(R_i(P_{\sigma(i)}),(R_i\circ f)|_{P_{\sigma(i)}})\}_{i=1}^n$$8 between piece and slot embeddings (Talon et al., 2022). Global assignment is then enforced with a Sinkhorn relaxation followed by Hungarian attention, making the matching process end-to-end (Talon et al., 2022).

Alphazzle applies single-player Monte Carlo Tree Search to the full permutation space. A state consists of the next patch to place, the current partial reassembly, and the occupancy dictionary of the canvas; legal actions are the empty slots (Paumard et al., 2023). Selection uses a PUCT rule with policy prior $$Q=\{(R_i(P_{\sigma(i)}),(R_i\circ f)|_{P_{\sigma(i)}})\}_{i=1}^n$$9, leaf evaluation uses a learned value network $f$0 instead of rollouts, and self-play fine-tuning on MCTS-visited states provides further improvement (Paumard et al., 2023). In contrast, polygonal puzzles are solved through a multi-body spring-mass system combined with hierarchical loop constraints and layered reconstruction (Harel et al., 2020), while the good-continuation formulation encodes all placements as a polymatrix game and finds a Nash equilibrium with multi-population replicator dynamics (Khoroshiltseva et al., 2024). The field therefore spans evolutionary search, spectral synchronization, shortest-path optimization, end-to-end assignment, tree search, physical simulation, and game-theoretic equilibrium computation.

5. Evaluation protocols and reported performance

Reported metrics are highly problem-dependent. Classical square-piece benchmarks frequently use neighbor comparison, defined as the fraction of correctly reconstructed abutting-edge pairs, together with the number of perfectly solved puzzles (Sholomon et al., 2017). The graph-connection literature additionally reports direct accuracy, largest component, and perfect reconstruction counts (Huroyan et al., 2018). Deepzzle introduces an “almost-perfect” reassembly criterion: for each slot $f$1, if the pixel-wise norm between the ground-truth fragment and the predicted fragment is at most $f$2, the swap is deemed visually acceptable, with $f$3 reported as best matching human tolerance (Paumard et al., 2020). GANzzle evaluates piece-to-slot retrieval by $f$4, and the whole-slide image reinterpretation uses Acc, F1, AUC, Avg.3, and C-index depending on whether the task is classification or survival prediction (Talon et al., 2022, Chen et al., 10 Jul 2025).

Representative results illustrate how strongly performance depends on the task definition and solver.

Method	Setting	Reported result
DNN-Buddies + GA	540-piece benchmark	96.37% neighbor accuracy; 11 perfect puzzles (Sholomon et al., 2017)
Deepzzle	Full $f$5 permutation space	39.2% image-level reassembly accuracy (Paumard et al., 2020)
Alphazzle	Full $f$6 permutation problem	51.5% full-puzzle, 75.1% patch-wise, 77.5% neighbor-wise, in $f$7 s/puzzle (Paumard et al., 2023)
TEN-Large + GA	Type-2 eroded puzzles	55.5%, 59.4%, 49.1% neighbor accuracy on DIV2K, PIRM, MIT (Rika et al., 2022)
Edge2Vec + GA	Type-2 eroded puzzles	82.3% neighbor accuracy (Rika et al., 2022)

On the standard benchmarks used in DNN-Buddies, adding the learned metric to the GA baseline raises neighbor accuracy from 94.88% to 95.65% on 432-piece puzzles, from 94.08% to 96.37% on 540-piece puzzles, and from 94.12% to 95.86% on 805-piece puzzles; the number of perfectly reconstructed puzzles increases by $f$8, respectively (Sholomon et al., 2017). The 96.37% result on the 540-piece set exceeds the previous best of 95.4% reported by Paikin and Tal (2015) (Sholomon et al., 2017). The same paper also notes that no formal statistical-significance tests or ablations beyond GA versus GA+DNN are provided (Sholomon et al., 2017).

Performance claims in difficult settings are more mixed. Deepzzle reports 24.7% almost-perfect puzzles and 64.6% well-placed fragments with no outsiders and no missing fragments, and under known versus unknown central fragment reports 44.4% versus 39.2% puzzle accuracy and 89.9% versus 71.1% fragment accuracy (Paumard et al., 2020). GANzzle is explicitly puzzle-size agnostic and, on PuzzleCelebA, reports direct accuracies of 72.18% on $f$9, 53.26% on $\sigma$0, 32.84% on $\sigma$1, and 12.94% on $\sigma$2 (Talon et al., 2022). This suggests that evaluation should be read together with the observation model: wide gaps, erosion, outsiders, unknown central fragments, variable puzzle sizes, and arbitrary shapes are not commensurate regimes.

6. Generalizations, cross-domain uses, and technical limits

A major trend is the move from ideal square tiles toward degraded or structurally richer inputs. Eroded-boundary puzzles motivate learned embeddings that use the entire piece rather than only the outermost pixels, because the boundary is unreliable by construction (Rika et al., 2022, Rika et al., 2022). Archaeological-style reconstruction with large gaps and outsider fragments motivates position-prediction models and branch-cut graph search rather than strict edge continuity (Paumard et al., 2020). Polygonal puzzles introduce bounded geometric noise, relaxed mate predicates, hierarchical loop consensus, and physical settling via a spring-mass dynamical system (Harel et al., 2020). The line-only problem goes further by deliberately ignoring conventional color and shape features and relying solely on geometric line continuation (Khoroshiltseva et al., 2024).

Another trend is conceptual transfer beyond image reassembly. Jigsaw percolation defines two graphs on the same vertex set—a social graph $\sigma$3 and a puzzle graph $\sigma$4—and iteratively merges clusters when there is both a social edge and a puzzle edge between them (Brummitt et al., 2012). For an Erdős–Rényi social network and the ring puzzle, the critical probability satisfies $\sigma$5; for any connected puzzle graph of bounded maximum degree, $\sigma$6 and $\sigma$7 for every $\sigma$8; and power-law social networks with $\sigma$9 cannot solve bounded-degree puzzles with high probability (Brummitt et al., 2012). This is not a reconstruction algorithm, but it recasts “solving a puzzle” as a phase transition in joint graph connectivity.

In whole-slide image analysis, the instance jigsaw puzzle becomes a regularizer against permutation-invariant aggregation. The proposed shuffling-equivariance loss is

$\{R_i\}$0

which encourages the network to undo the random permutation in feature space (Chen et al., 10 Jul 2025). On CAMELYON16, the Transformer variant reports 96.64 Acc, 96.39 F1, 98.00 AUC, and 97.01 Avg.3, compared with 96.31 Avg.3 for the best MIL baseline TransMIL; on TCGA-NSCLC, it reports 96.13 Avg.3 versus 95.59 for the best MIL baseline (Chen et al., 10 Jul 2025). For survival prediction, the same model reaches C-index 0.732 on TCGA-LUAD and 0.715 on TCGA-BRCA, compared with 0.714 and 0.701 for the best MIL baselines (Chen et al., 10 Jul 2025). Here the “pieces” are not physical fragments, yet the hidden permutation and the recovery of spatial correlations preserve the central logic of the jigsaw formulation.

The literature also records clear limitations. DNN-Buddies addresses class imbalance by informed undersampling, but does not explore more advanced remedies and does not detail the computational cost of classifying all $\{R_i\}$1 edge pairs (Sholomon et al., 2017). The graph-connection method is dominated by the $\{R_i\}$2 MGC precomputation and by the Type-1 solver (Huroyan et al., 2018). Deepzzle explicitly notes that the independence approximation $\{R_i\}$3 ignores spatial context beyond center-fragment pairs and that exponential graph size limits massively large puzzles beyond approximately 20 fragments despite pruning (Paumard et al., 2020). These limitations help explain why the field remains methodologically plural: no single formulation currently covers natural-image square puzzles, eroded fragments, polygonal pieces, line-only evidence, and instance-order recovery in bag-level learning with equal effectiveness.