RePAIR 2D Dataset: Benchmark for Fresco Reassembly

• RePAIR 2D Dataset is a real-world archaeological benchmark featuring high-quality 2D renders of fresco fragments with authentic erosion, missing pieces, and irregular shapes.
• The dataset is generated via a rigorous 3D digitization and orthographic rendering pipeline, with ground truth assemblies validated by expert archaeologists.
• It supports tasks like pairwise fragment matching, pose estimation, and global mosaic reconstruction, highlighting the gap between current methods and complex real-world scenarios.

Searching arXiv for the RePAIR dataset paper and closely related reassembly references. The RePAIR 2D dataset is the two-dimensional component of RePAIR, a real-world, multimodal dataset and benchmark for reassembling fragmented wall paintings from Pompeii. Its fragments originate from an actual collapse of ceiling and wall frescoes—first damaged in the AD 79 eruption and subsequently fragmented by World War II bombings at the Pompeii Archaeological Park—rather than from simulated cuts or synthetic polygonal decompositions. In the benchmark formulation, the 2D modality is derived from high-quality 3D digitization and orthographic rendering of each fragment’s painted surface, with archaeologist-validated assemblies serving as ground truth. The resulting benchmark is explicitly intended to challenge modern 2D and 3D reassembly methods that have often been validated on more constrained scenarios, and its reported baselines show a substantial gap between current methods and the difficulty of real fresco reassembly (Tsesmelis et al., 2024).

1. Definition and archaeological provenance

RePAIR differs from conventional 2D puzzle datasets because its source material is archaeological rather than synthetic. The fragments and fractures are realistic, caused by a collapse of a fresco during a World War II bombing at the Pompeii archaeological park, and the broader provenance includes prior damage from the AD 79 eruption. This yields irregular shapes, erosion, abrasion, missing pieces, and variable fracture volumes. In the 2D setting, these factors directly affect contour compatibility, texture continuity, and the reliability of local adjacency cues (Tsesmelis et al., 2024).

The benchmark was constructed from long-term archaeological work rather than from a known complete reference image. Archaeologists manually cleaned, inventoried, analyzed, and physically reassembled subsets of fragments into coherent groups, providing a unique ground truth after years of fieldwork. The abstract further states that ground truth was generated through several years of unceasing fieldwork, including the excavation and cleaning of each fragment, followed by manual puzzle solving by archaeologists of a subset of approx. 1000 pieces among the 16000 available. This distinguishes RePAIR from benchmarks in which the “correct” solution is algorithmically inherited from a pristine original image.

A common misconception is to treat the 2D component as merely another irregular jigsaw dataset. That characterization is incomplete. The paper emphasizes that the complete solution is unknown for some frescoes, that some fragments remain ungrouped as “Isolated,” and that erosion and missing content are integral to the task rather than incidental noise. This suggests that the benchmark is not only about geometric fitting, but also about reasoning under partial observability and historically contingent damage.

2. Scope, composition, and subset boundaries

The full digitized benchmark contains 1070 reconstructed 3D fragments. Of these, 1070 pieces were organized into 117 coherent groups by archaeologists, with each group comprising 2 to 44 connected fragments. In addition, 150 fragments are provided as “Isolated,” meaning ungrouped pieces whose solution is unknown and therefore remain an open-discovery component of the dataset (Tsesmelis et al., 2024).

The 2D component is smaller than the 3D component. The paper states that some pieces lack a planar painted surface, including stucco fragments and fragments located at wall–ceiling intersections, and these are excluded from 2D ground truth and 2D puzzles. Table 1 reports that RePAIR in 2D has 101 groups. The exact number of 2D fragments is not specified in the main text. This asymmetry is operationally important: there is no guaranteed 1:1 correspondence between the 2D and 3D benchmark instances.

Beyond the 117 groups, a substantial portion belongs to what archaeologists term “Décor 1,” reconstructed into five plaques and analyzed for plausible global layouts. Not all fragments of Décor 1 are recovered; two schematic solutions exist, with one selected by archaeologists for final rendering. This indicates that even when local reconstructions are accepted, global layout may remain partly interpretive.

The official split is 80/20 for training and testing, with an optional validation set, and this split applies to both the 2D and 3D tasks. The 2D benchmark therefore supports supervised development under a fixed protocol while preserving the archaeological reality that some fragments and some large-scale configurations remain unresolved.

3. Digitization, segmentation, and orthographic rendering

The 2D dataset is grounded in a 3D acquisition pipeline. Geometry capture was performed with a Polyga H3 structured-light scanner with 0.08 mm accuracy. Each fragment was placed on a turntable and captured from 18 viewpoints, plus top and bottom, with acquisition inside a lighting box to minimize shadows and reflections. For texture, the pipeline uses a Sony α7c camera with 24.2 megapixels and images up to $6000 \times 3376$ pixels. These images are used in a photogrammetry pipeline based on SfM to derive camera poses and a high-resolution texture model registered to the 3D geometry (Tsesmelis et al., 2024).

Background handling is explicit. Controlled lighting is provided by the lighting box, and segmentation masks are used to avoid reconstructing the background in photogrammetry. The mask-generation workflow is semi-supervised and consists of three steps: guided interactive segmentation for the first image per fragment; temporal consistency to extend masks to subsequent frames; and fine-tuning a pre-trained Mask R-CNN to obtain accurate masks, especially along borders. The 2D modality therefore inherits not only geometric accuracy from 3D capture, but also a curated separation between fragment and background.

Painted-surface isolation is the critical step that turns a 3D archaeological fragment into a 2D puzzle piece. The 3D fragment is segmented into surfaces; principal curvatures $K_1$, $K_2$ and curvedness are computed per vertex; and region growing clusters vertices into surfaces. The smoothest large surface, defined as the one with the lowest mean curvedness, is selected as the painted side. Curvedness is defined as

$C = \sqrt{\frac{K_1^2 + K_2^2}{2}}.$

After surface selection, each fragment’s painted surface is aligned so that its normal coincides with the $Z$-axis, and the fragment is rendered in Blender using orthographic projection. The rendering parameters are fixed: square image aspect ratio $1{:}1$, image resolution $S_{\mathrm{img}} = 2000$ pixels, orthographic scale $s_{\mathrm{ort}} = 2.714$, and scene scale factor for 3D meshes $s_{3D} = 0.01$. For the final 2D dataset, one orthographic 2D rendering of the painted surface per fragment is produced. The purpose of this normalization is to ensure consistent 2D representations for global assembly.

A plausible implication is that the 2D task is not a raw-photograph reassembly problem. Instead, it is a controlled projection of archaeologically meaningful surfaces extracted from 3D data, which reduces nuisance variation while preserving the breakage, wear, and incompleteness that make the benchmark difficult.

4. Ground truth, metadata, and data representation

Ground truth for the 2D benchmark is derived from 3D assemblies prepared in Blender using 6DoF poses. For 2D, each fragment is aligned so that the textured surface normal points along $Z$, making $K_1$0 the 2D plane. The 2D ground truth therefore consists of in-plane translation and rotation around $K_1$1. Translations are converted from millimeters to pixels via a global scale:

$K_1$2

$K_1$3

Here $K_1$4 is the 2D image size in pixels, $K_1$5 is the rendering scale applied to meshes, and $K_1$6 is the orthographic scale. Rotation is the Euler angle around $K_1$7 (Tsesmelis et al., 2024).

The dataset stores 2D and 3D assets in different formats.

Modality	Files	Notes
2D	.PNG, .TXT, .JSON	One orthographic rendering per fragment; each group has a ground-truth transformation file; additional metadata are stored as JSON
3D	.OBJ, .MTL, texture .PNG	Meshes are provided in assembled positions and orientations

Per-fragment JSON metadata include Acquisition Date; Artistic Style; Filename(s) for .ply and .obj; Fresco Family; Geometric Data including center of mass, dimensions, bounding box limits, position, scale, vertices/faces, diagonal size, and ground-truth transformation within the group; ID; Link; RGB file(s) for photogrammetry; Raw 3D file(s) for top and bottom; Texture filenames for low- and high-resolution .png; Version; and Weight in grams. The fields “Artistic Style” and “Fresco Family” are provided by archaeologists. The main text also states that archaeologists annotate material and salient back-side features such as mortar layers, traces of adhesion, smoothing strokes, and preparatory lines, which serve as alignment cues, although these are not enumerated as structured metadata fields beyond “Artistic Style” and “Fresco Family.”

The paper also clarifies what is not distributed explicitly. Ground-truth adjacency graphs are not specified as metadata; instead, adjacency relationships are derived for evaluation. This matters because downstream work that assumes a prepackaged graph-structured supervision signal would be misreading the benchmark design.

5. Benchmark tasks, adjacency, and evaluation

The 2D benchmark covers four tasks: pairwise fragment matching based on contour or texture compatibility; estimation of piece orientations as rotation around $K_1$8 and in-plane translations; recovery of local adjacency, called the “mating graph”; and global mosaic reconstruction of each group at plaque level (Tsesmelis et al., 2024).

Neighbor relations in 2D are defined geometrically. All fragments are dilated by more than half the maximal number of pixels observed between neighbors in the group, and any two pieces whose dilated silhouettes intersect are considered neighbors. The paper notes that this may fail if a thin fragment separates two fragments, but such anomalies were not identified. This definition is specific to the benchmark protocol and should not be conflated with a manually curated adjacency annotation.

The principal pose-quality metric is $K_1$9, an area-based measure in 2D. After rigidly aligning a reconstruction to the ground truth by anchoring the largest fragment, $K_2$0 measures the shared area between each ground-truth fragment and its predicted placement, weighted by fragment size:

$K_2$1

Translation and rotation errors are reported as

$K_2$2

Here $K_2$3 and $K_2$4 are the predicted translation in millimeters and rotation in degrees for fragment $K_2$5, while $K_2$6 and $K_2$7 are ground truth.

Neighbor recovery is measured through weighted precision, recall, and $K_2$8, with weights derived from fragment areas:

$K_2$9

$C = \sqrt{\frac{K_1^2 + K_2^2}{2}}.$0

These metrics emphasize correct recovery of larger-fragment adjacencies. The paper also notes that $C = \sqrt{\frac{K_1^2 + K_2^2}{2}}.$1 is indirectly influenced by RMSE: small angle or translation errors can substantially reduce shared-area overlap.

6. Baselines and empirical behavior

The benchmark evaluates three 2D baselines. The first is the method of Derech et al., a greedy archaeological puzzle solver that iteratively adds the next best fragment based on texture-based dissimilarity of extrapolated bordering regions; in the RePAIR benchmark, the original extrapolation is replaced with the stable-diffusion extrapolation process from Harel et al. The second is Genetic Optimization, formulated as evolutionary optimization over an $C = \sqrt{\frac{K_1^2 + K_2^2}{2}}.$2 matrix containing $C = \sqrt{\frac{K_1^2 + K_2^2}{2}}.$3 per piece, with fitness defined as the weighted sum of the area of the bounding rectangle and the intersection area of overlapping pieces, both to be minimized. The third is Greedy Geometric Matching, which starts from a random seed fragment, approximates contours with the Douglas–Peucker algorithm, creates segments at high-curvature vertices, and evaluates compatibility between segments using a spring–mass physical optimization as in Harel et al.; after convergence, compatibility is the intersection-over-union of the two bodies, although the exact IoU formula is not specified in the paper (Tsesmelis et al., 2024).

On the 2D test set, excluding one group, the reported results are as follows. Derech et al. obtains $C = \sqrt{\frac{K_1^2 + K_2^2}{2}}.$4, $C = \sqrt{\frac{K_1^2 + K_2^2}{2}}.$5, $C = \sqrt{\frac{K_1^2 + K_2^2}{2}}.$6, Precision $C = \sqrt{\frac{K_1^2 + K_2^2}{2}}.$7, Recall $C = \sqrt{\frac{K_1^2 + K_2^2}{2}}.$8, and $C = \sqrt{\frac{K_1^2 + K_2^2}{2}}.$9. Genetic Optimization obtains $Z$0, $Z$1, $Z$2, Precision $Z$3, Recall $Z$4, and $Z$5. Greedy Geometric Matching obtains $Z$6, $Z$7, $Z$8, Precision $Z$9, Recall $1{:}1$0, and $1{:}1$1.

The paper’s interpretation is restrained but clear. All methods struggle with RePAIR’s realistic erosion and irregular shapes, and even the texture-aware solver does not consistently outperform geometry-based baselines. The precision/recall trade-offs indicate difficulty in identifying true neighbors without over-connecting fragments. This suggests that the main challenge is not merely search over rigid transformations, but robust compatibility estimation under abrasion, missing material, faint or incomplete texture, and the absence of a clean global reference image.

7. Access, constraints, and future directions

The dataset is distributed through a Zenodo repository at https://zenodo.org/records/13993089, with a project page at https://repairproject.github.io/RePAIR_dataset/. The license is custom: the data can be downloaded, reviewed, and reused under attribution and non-commercial terms. The authors state that they accept full responsibility for rights and confirm licenses associated with the dataset and code (Tsesmelis et al., 2024).

Associated code is released for the principal stages of the pipeline: 2D baselines, rendering from 3D to 2D, fragment digitization, 3D segmentation for painted-surface isolation, ground-truth preparation in Blender, and 3D baselines. This release structure reflects the benchmark’s multimodal design: the 2D subset is not independent of 3D acquisition, segmentation, and rendering.

Several usage constraints are specific to the 2D subset. The paper recommends using the provided 2D renders and group-level transform files, applying the published scale conversion $1{:}1$2 if transforms are computed from 3D, and anchoring the largest fragment to remove global rigid-motion ambiguity in evaluation. It also recommends exploiting masks, segmentations, and painted-surface isolation to reduce background clutter when deriving contours or texture descriptors. Cross-modality strategies are explicitly suggested: leveraging 3D scans to improve 2D adjacency, using archaeologist metadata such as artistic style, fresco family, and back-side mortar patterns as priors, and combining texture cues with contour geometry.

The paper identifies several limitations and future directions. Coverage is incomplete because non-planar fragments are omitted from the 2D subset, and the exact number of 2D fragments is not specified beyond the report of 101 groups. Few existing 2D methods handle unrestricted, eroded shapes without a clean reference image. Suggested future work includes methods that cope with degraded geometry and scarce pictorial information, incorporation of archaeology and art-history knowledge into compatibility scoring, exploration of shape repair or completion to aid reassembly, and multimodal reasoning, including MLLMs, for integrating textual or knowledge cues with visual signals. No class labels or categorical annotations are defined for 2D beyond the archaeologists’ “Artistic Style” and “Fresco Family” metadata.

In that sense, the RePAIR 2D dataset occupies a specific position within puzzle-solving research: it is a benchmark in which realism is not an aesthetic property but the central methodological constraint. Its 2D instances are standardized enough to support reproducible evaluation, yet they retain the erosion, incompleteness, and archaeological uncertainty that expose failure modes in current reassembly systems.