SOM-based Data Augmentation for Photo-z

• SOM-based data augmentation is a technique that uses self-organizing maps to identify underrepresented regions in galaxy colour–magnitude space.
• It supplements spectroscopic training sets with simulated galaxies from independent catalogues, enhancing photo-z calibration under LSST-like conditions.
• The method reduces systematic bias and catastrophic failures in photometric redshift estimates, leading to more robust cosmological inferences.

Searching arXiv for the target paper and closely related SOM/photo-z references. SOM-based data augmentation denotes the use of Self-Organising Maps (SOMs) to identify under-sampled regions of galaxy colour–magnitude space and to augment spectroscopic training sets with simulated galaxies occupying those regions, with the specific aim of improving photometric redshift estimation. In the formulation introduced by Y.-H. Zhang et al., galaxy Spectral Energy Distributions (SEDs), represented by broad-band photometric colours and magnitudes, are projected onto a two-dimensional SOM; cells with sparse spectroscopic occupancy are then supplemented using simulated galaxies drawn from an independent synthetic catalogue, yielding a more representative training sample for photo-$z$ inference (Zhang et al., 28 Aug 2025). The method is developed in the context of LSST-like survey conditions using OpenUniverse2024 mock catalogues and CosmoDC2 simulations, and is reported to reduce systematic bias, decrease catastrophic failures by up to approximately a factor of 2, and lower information loss in conditional density estimation, particularly at high redshift $z_\mathrm{true} \gtrsim 1.5$ (Zhang et al., 28 Aug 2025).

1. Conceptual basis and problem setting

The immediate problem addressed by SOM-based data augmentation is spectroscopic incompleteness in photometric-redshift training sets. In large photometric surveys, the spectroscopic subsample does not uniformly sample the full distribution of galaxy SEDs; faint and high-redshift objects are especially likely to be underrepresented. In the Zhang et al. framework, this incompleteness is operationalised through occupancy on a SOM trained in colour–magnitude space, so that sparsely populated cells correspond to regions where the available spectroscopic data do not adequately cover the photometric sample (Zhang et al., 28 Aug 2025).

A SOM serves here as a topology-preserving discretisation of high-dimensional photometric feature space. Each cell, or neuron, represents a local region populated by galaxies with similar SEDs. The use of a two-dimensional map is motivated by visualisation and by the ability to distinguish sampled from under-sampled regions while still clustering the high-dimensional colour/magnitude data effectively (Zhang et al., 28 Aug 2025). This places SOM-based augmentation at the intersection of representation learning, survey selection-function diagnostics, and supervised calibration of photo-$z$ estimators.

The broader methodological context is the established use of SOMs for photometric redshift validation and calibration. The paper explicitly situates its procedure alongside earlier SOM-based photo-$z$ work such as Wright et al. (2020) and Buchs et al. (2019), while extending the role of the SOM from diagnostics and calibration to active augmentation of the training set (Zhang et al., 28 Aug 2025). A plausible implication is that SOMs are being used not merely as descriptive tools for coverage analysis, but as decision layers for targeted synthetic data insertion.

2. Representation of galaxy SEDs on the SOM

The input to the SOM is each galaxy’s observed SED encoded as broad-band photometric colours and magnitudes, written as a feature vector

$\vec{x}_i = (c_{i,1}, c_{i,2}, ..., c_{i,N_{\text{bands})}.$

The SOM is trained on these feature vectors so that neighbouring cells on the map correspond to nearby regions in the underlying colour–magnitude manifold (Zhang et al., 28 Aug 2025).

In the reported implementation, the SOM algorithm iteratively moves each cell’s weight vector toward the galaxy SEDs it represents while preserving topological neighbourhoods. The implementation is based on Somoclu, integrated within the RAIL photometric redshift platform (Zhang et al., 28 Aug 2025). This matters operationally because it embeds the augmentation procedure inside an existing photo-$z$ infrastructure rather than treating it as a standalone pre-processing step.

Map dimensionality is treated as adjustable in principle, but the method uses a 2D SOM. The stated reasons are twofold: visualisation and effectiveness for clustering high-dimensional colour/magnitude data while permitting clear detection of sampled versus under-sampled regions (Zhang et al., 28 Aug 2025). The map size is selected so that each cell ideally corresponds to a well-defined sub-population of SEDs, while retaining enough resolution to distinguish physically meaningful SED variation without producing predominantly empty cells (Zhang et al., 28 Aug 2025). This design criterion highlights a key tension in SOM-based augmentation: overly coarse maps obscure local incompleteness, whereas overly fine maps convert the representation itself into a source of sparsity.

3. Detection of sparse cells and augmentation workflow

Once the SOM has been trained, each galaxy in the spectroscopic sample is assigned to a cell via its Best Matching Unit. For each cell $j$, the number of real spectroscopic galaxies $n_{\text{obs}}(j)$ is counted (Zhang et al., 28 Aug 2025). A cell is defined as sparse when

$n_\text{obs}(j) < n_\text{min},$

where $n_\text{min}$ is typically set to 1 or a small empirically determined value motivated by redshift-estimation needs and statistical stability (Zhang et al., 28 Aug 2025).

The logic of the augmentation stage is straightforward: sparse or empty SOM cells mark regions of colour space that are underrepresented in the training set, often because of spectroscopic selection effects affecting faint or high-$z_\mathrm{true} \gtrsim 1.5$0 galaxies (Zhang et al., 28 Aug 2025). The method then supplements these cells using simulated galaxies whose photometric properties place them in the same cells.

The workflow described in the paper can be summarised as follows.

Step	Operation	Source
1	Identify empty or sparse SOM cells in the degraded spectroscopic sample	(Zhang et al., 28 Aug 2025)
2	Select simulated galaxies from CosmoDC2 that map to the same cells	(Zhang et al., 28 Aug 2025)
3	Append their colours, magnitudes, and true simulated redshifts to the training set	(Zhang et al., 28 Aug 2025)
4	Train or calibrate photometric redshift estimators on the augmented set	(Zhang et al., 28 Aug 2025)

The paper also gives an abstracted pseudocode form in which each cell $z_\mathrm{true} \gtrsim 1.5$1 is checked, and if $z_\mathrm{true} \gtrsim 1.5$2, one or more simulated galaxies mapping to that cell are added until the minimum occupancy criterion is satisfied (Zhang et al., 28 Aug 2025). The photo-$z_\mathrm{true} \gtrsim 1.5$3 estimators trained on the resulting augmented dataset may be template-based, machine-learning-based, or hybrid methods deployed on RAIL (Zhang et al., 28 Aug 2025).

This workflow is notable because it does not generate arbitrary synthetic points in feature space. Instead, it selects simulated galaxies from an independent high-fidelity synthetic sky catalogue that already occupy the relevant SOM cells (Zhang et al., 28 Aug 2025). This suggests a constrained augmentation regime in which the SOM acts as a matching and coverage-control mechanism rather than a generative model in its own right.

4. Data sources, survey realism, and experimental design

The framework is evaluated for the forthcoming Legacy Survey of Space and Time using mock galaxy catalogues from the OpenUniverse2024 project and synthetic datasets that mimic the expected photometric selections of LSST after one year (Y1) and ten years (Y10) of observation (Zhang et al., 28 Aug 2025). The use of both Y1 and Y10 conditions is methodologically important because the depth and selection characteristics differ substantially across these survey stages, altering the observed distribution of galaxy colours and magnitudes.

To emulate realistic spectroscopic survey assembly, the study constructs 501 degraded realisations by sampling galaxy colours, magnitudes, redshifts, and spectroscopic success rates (Zhang et al., 28 Aug 2025). These degraded realisations are intended to represent a wide array of realistic spectroscopic surveys, thereby introducing variability associated with selection, incompleteness, and observational success. Simulated augmentation targets are then sourced from the independent CosmoDC2 catalogues (Zhang et al., 28 Aug 2025).

The reliance on an independent simulation for augmentation is a central design choice. The OpenUniverse2024 mock photometric catalogue is used to identify cells lacking sufficient training data under LSST-like selections, while CosmoDC2 provides simulated galaxies that can fill those cells (Zhang et al., 28 Aug 2025). This separation between the catalogue used to diagnose incompleteness and the catalogue used to supply augmentation examples reduces circularity in the experimental construction. A plausible implication is that the method’s performance depends not only on SOM topology and occupancy thresholds, but also on the fidelity and domain compatibility of the auxiliary simulation used for augmentation.

5. Performance measures and reported effects

The reported effects of SOM-based augmentation are quantified using several photo-$z_\mathrm{true} \gtrsim 1.5$4 performance measures. For redshift bias, if $z_\mathrm{true} \gtrsim 1.5$5, the mean bias is

$z_\mathrm{true} \gtrsim 1.5$6

and the augmented datasets are reported to show a substantial reduction in systematic bias, especially for galaxies at high redshift $z_\mathrm{true} \gtrsim 1.5$7 where spectroscopic incompleteness is most acute (Zhang et al., 28 Aug 2025).

For catastrophic failures, the paper uses the standard criterion

$z_\mathrm{true} \gtrsim 1.5$8

and states that the fraction of catastrophic photo-$z_\mathrm{true} \gtrsim 1.5$9 failures is reduced by up to a factor of 2 in the most challenging regimes (Zhang et al., 28 Aug 2025). The abstract similarly reports a decrease in catastrophic failures by up to approximately a factor of 2 (Zhang et al., 28 Aug 2025).

The method is also evaluated in terms of information loss for conditional density estimation. The Kullback–Leibler divergence is written as

$z$0

and, at cell level,

$z$1

The paper states that average KL divergence after augmentation is lower, indicating improved modelling of uncertainties and tails in $z$2 (Zhang et al., 28 Aug 2025).

Across the 501 mock survey realisations, the method is reported to improve performance metrics consistently, which the paper interprets as robustness to sample variance and survey specifics (Zhang et al., 28 Aug 2025). Taken together, these results position SOM-based augmentation as a mechanism for improving both point-estimate behaviour and posterior-density fidelity, rather than only one of these aspects.

6. Relation to photometric-redshift pipelines and survey science

Within the reported implementation, the methodology is integrated into the RAIL photometric redshift platform, and the resulting augmented training set is used for photometric redshift estimator training and calibration (Zhang et al., 28 Aug 2025). This integration is significant because photo-$z$3 workflows for LSST-like analyses are inherently pipeline-centric, involving ingestion, representation, calibration, posterior estimation, and validation. The use of Somoclu and RAIL indicates that SOM-based augmentation is presented not simply as a conceptual procedure but as an operational component of survey-scale inference infrastructure (Zhang et al., 28 Aug 2025).

The scientific motivation lies in cosmological analyses that depend sensitively on photo-$z$4 quality. The paper states that improved photo-$z$5 estimates—specifically lower bias, fewer catastrophic outliers, and more accurate redshift posterior PDFs—yield more robust cosmological parameter inference for applications such as cosmic shear, galaxy clustering, and BAO (Zhang et al., 28 Aug 2025). It further states that this is critical for stringent systematic error budgets in Stage-IV surveys including LSST, Euclid, and Roman (Zhang et al., 28 Aug 2025).

The method is described as fully general and scalable, suitable for automated pipelines and adaptable as survey strategies or simulations improve (Zhang et al., 28 Aug 2025). The scripts and data are reported as open source, which supports reproducibility and external validation (Zhang et al., 28 Aug 2025). A plausible implication is that the approach could be re-instantiated whenever a survey’s effective selection function changes, with the SOM serving as a reusable interface between observed coverage and available simulation support.

7. Scope, limitations, and interpretive considerations

SOM-based data augmentation does not eliminate the need for representative spectroscopy; rather, it addresses documented gaps by identifying and supplementing under-sampled regions in a principled way (Zhang et al., 28 Aug 2025). The method’s validity therefore depends on at least three coupled assumptions stated or implied by the study: that the SOM adequately partitions the relevant colour–magnitude structure, that sparse occupancy on the map corresponds to consequential training-set deficiency, and that the independent simulated catalogue supplies galaxies with sufficiently realistic SEDs and true redshifts for those deficient cells.

A common misconception would be to treat the procedure as unconstrained synthetic oversampling. The paper instead describes a targeted cell-based augmentation strategy in which simulated galaxies are selected because they land in the same SOM cells as underrepresented observed populations (Zhang et al., 28 Aug 2025). The objective is representativeness of the spectroscopic training set, not arbitrary expansion of sample size.

Another potential misunderstanding is that the gains arise solely from better point prediction. The inclusion of KL-divergence metrics and the emphasis on conditional density estimation show that the framework is equally concerned with improving the quality of estimated redshift posteriors $z$6 (Zhang et al., 28 Aug 2025). This is especially relevant for modern cosmology pipelines, where full posterior characterisation often matters more than a single redshift summary statistic.

In the specific study, the most pronounced gains are reported for high-redshift galaxies, $z$7, where spectroscopic incompleteness is strongest (Zhang et al., 28 Aug 2025). This suggests that the efficacy of SOM-based augmentation is likely to be regime-dependent: its marginal benefit should be greatest where the observed training distribution most strongly diverges from the photometric target distribution. The paper therefore presents SOM-based augmentation not as a universal replacement for better data collection, but as a structured method for mitigating incompleteness in photo-$z$8 training under realistic survey constraints (Zhang et al., 28 Aug 2025).