Iterative Systematics Decontamination (ISD)

• Iterative Systematics Decontamination (ISD) is a template-based, iterative regression method designed to mitigate spatially varying observational systematics in galaxy survey data.
• It employs one-dimensional fits calibrated with uncontaminated mocks to convert significant survey-property responses into inverse-response weight maps that correct galaxy density measurements.
• Enhanced in DES Year 6 with masking and polynomial (up to third order) fits, ISD effectively reduces contamination bias in key cosmological parameters to below 0.5σ.

Searching arXiv for papers on Iterative Systematics Decontamination and closely related iterative systematics-cleaning methods. Iterative Systematics Decontamination (ISD) is a template-based, iterative, regression-to-weights methodology for mitigating spatially varying observational systematics in large-scale-structure analyses, particularly galaxy clustering. In the DES usage that has made the term most concrete, ISD diagnoses contamination through one-dimensional relations between observed galaxy number density and survey-property or foreground templates, calibrates significance with uncontaminated mocks, converts significant template responses into inverse-response weight maps, and repeats the procedure until no template remains significant (Rodríguez-Monroy et al., 2021). A later DES Year 6 development couples ISD tightly to footprint definition, showing that masking extreme regions in template space improves both contaminant detection and the stability of the corrective weights, while extending the response model from linear fits to polynomial fits “up to third order” to capture non-linear contamination (Rodríguez-Monroy et al., 9 Sep 2025). In a broader methodological sense, related CMB work implements an iterative, map-level reconstruction-and-subtraction of latent systematics fields, illustrating a parallel ISD logic outside galaxy clustering (Williams et al., 2021).

1. Definition and problem setting

In DES Year 3, ISD is the fiducial method for removing artificial clustering induced by observing conditions and foregrounds from lens-galaxy samples used in clustering and $3\times 2$pt cosmology (Rodríguez-Monroy et al., 2021). The underlying problem is that spatial variation in observing conditions and foregrounds changes the selection function across the observed footprint, so the observed galaxy number density no longer traces only cosmological large-scale structure. These contaminants bias the measured angular correlation function $w(\theta)$, and the impact is cosmologically material: the uncorrected analysis can shift inferred parameters by up to about $\sim 7\sigma$ in $\Omega_m$ and more than $4\sigma$ in galaxy bias, with specific Y3 clustering-only differences reported as $4.10\sigma$ in $b^3$ and $6.96\sigma$ in $\Omega_m$ for MagLim, and $7.69\sigma$ in $w(\theta)$0 and $w(\theta)$1 in $w(\theta)$2 for redMaGiC (Rodríguez-Monroy et al., 2021).

Operationally, ISD is iterative, template-based, regression-based, weighting-based, and map-level. It does not directly subtract contaminants in harmonic or correlation-function space. Instead, it fits the dependence of relative galaxy density on survey-property maps, interprets the fitted response as a multiplicative modulation, inverts that response into pixel weights, reapplies the measurement on the reweighted sample, and reevaluates significance after each cleaning step (Rodríguez-Monroy et al., 2021). The paper explicitly characterizes such procedures as regression methods in which the true overdensity field corresponds to the residuals after regressing the observed density field against a set of survey-property maps.

This suggests a useful generic description of ISD as a greedy sequential template cleaner acting on map-level weights rather than as a simultaneous multivariate estimator. That interpretation is especially apt for DES Y3, where the algorithm explicitly selects the most significant contaminant one map at a time, recomputes all significances after each update, and stops only when every candidate map falls below a preset threshold (Rodríguez-Monroy et al., 2021).

2. DES Year 3 algorithmic formulation

The DES Y3 implementation provides the clearest explicit operational definition of ISD. The algorithm begins with a library of survey-property maps, orthogonalized through PCA after standardization to zero mean and unit variance at $w(\theta)$3. The Y3 library contains 107 survey-property maps, and the fiducial basis retains the first 50 principal components, capturing about 98% of the total variance (Rodríguez-Monroy et al., 2021). This orthogonalization is not merely cosmetic: the paper argues that using only a small hand-picked subset imposes a hard prior of zero contamination from omitted maps, while highly correlated raw templates can destabilize sequential 1D fitting.

At each iteration, every candidate principal-component map is degraded to $w(\theta)$4, and the relation between template value $w(\theta)$5 and the observed relative number density $w(\theta)$6 is computed. This one-dimensional relation is binned into 10 equal sky-area bins. Using 1000 uncontaminated log-normal mocks, DES estimates a $w(\theta)$7 covariance matrix for each binned relation and fits both a null model $w(\theta)$8 and a linear model

$w(\theta)$9

The contamination statistic is then

$\sim 7\sigma$0

and its significance is normalized by the 68th percentile of the mock distribution,

$\sim 7\sigma$1

A template is considered significant when $\sim 7\sigma$2, with $\sim 7\sigma$3 tested and fiducial $\sim 7\sigma$4 (Rodríguez-Monroy et al., 2021).

The most significant map is selected, converted into a weight map through the inverse fitted response,

$\sim 7\sigma$5

normalized to mean unity, and applied to the pixelized galaxy counts,

$\sim 7\sigma$6

The procedure is repeated on the reweighted sample. The total ISD weight in a tomographic bin is multiplicative,

$\sim 7\sigma$7

again normalized to mean 1 (Rodríguez-Monroy et al., 2021).

The essential features are therefore explicit and unusually concrete: sequential selection, mock-calibrated significance, inverse-response weighting, recomputation after each update, and a stopping criterion defined by the absence of any remaining template with $\sim 7\sigma$8. These elements distinguish ISD from simultaneous penalized regression such as ENet and from mode-projection approaches.

3. Templates, masking, and footprint dependence

A central lesson from both DES Y3 and Y6 is that ISD is inseparable from the survey footprint over which it is applied. In Y3, the final common area is $\sim 7\sigma$9, defined at $\Omega_m$0 with cuts including fracdet $\Omega_m$1, $\Omega_m$2-band depth $\Omega_m$3, and sample-specific quality selections (Rodríguez-Monroy et al., 2021). In Y6, the coupling between footprint definition and decontamination is elevated to the main methodological theme: the authors argue that decontamination methods are only valid over some range of template space, and extreme template values can both degrade the quality of one-dimensional fits and destabilize the correction procedure (Rodríguez-Monroy et al., 9 Sep 2025).

The Y6 footprint construction begins from a seed footprint formed by the DES Y6 Gold footprint, the DES Y6 shear footprint, and an additional DECaLS-based LRG mask supporting the MagLim++ star–galaxy separation. On top of that seed, the analysis applies a baseline mask and a systematics mask. The baseline mask removes shallow regions unsuitable for the MagLim++ magnitude cut $\Omega_m$4, while the systematics mask targets precisely those regions where ISD-like correction is least trustworthy (Rodríguez-Monroy et al., 9 Sep 2025).

The Y6 systematics mask combines three elements. First, a cirrus/artifact mask removes $\Omega_m$5 and the top $\Omega_m$6 of $\Omega_m$7. Second, a 1D template-outlier mask uses Tukey-style fences,

$\Omega_m$8

with outliers outside

$\Omega_m$9

where $4\sigma$0. Third, a multivariate leverage mask removes the top $4\sigma$1 of pixels in the 19-dimensional template space used for weight estimation, using

$4\sigma$2

In linear regression language, the same paper notes that

$4\sigma$3

so high-leverage pixels are those where the inferred inverse weight is most sensitive to the observed overdensity (Rodríguez-Monroy et al., 9 Sep 2025).

The area evolution quantifies the trade. The seed footprint is $4\sigma$4, the baseline mask reduces this to $4\sigma$5, the addition of the systematics mask yields $4\sigma$6, and the final joint footprint is $4\sigma$7. The cumulative loss relative to the seed is $4\sigma$8, with the systematics mask alone removing $4\sigma$9, or about $4.10\sigma$0 of the seed area (Rodríguez-Monroy et al., 9 Sep 2025). The stated rationale is that a smaller but better-conditioned footprint enables simpler and more trustworthy decontamination.

A plausible implication is that ISD should be understood not as an invariant algorithm acting on arbitrary maps, but as a footprint-conditional procedure whose validity depends on the domain over which the density–template response is modeled.

4. Non-linear contamination and the DES Year 6 extension

The principal methodological extension in DES Y6 is the move beyond purely linear one-dimensional fits. In Y3, ISD uses linear $4.10\sigma$1 for principal-component maps and reports no strong evidence for nonlinearity in that basis (Rodríguez-Monroy et al., 2021). Y6, by contrast, emphasizes that with higher statistical power, linear response models can miss contaminants whose effects are visibly at the $4.10\sigma$2 level in the one-dimensional response curves. The new implementation therefore allows polynomial fits “up to third order,” explicitly to detect and correct non-linear contamination (Rodríguez-Monroy et al., 9 Sep 2025).

Within the Y6 footprint, the operational description remains recognizably ISD-like. For each template $4.10\sigma$3, the analysis computes a one-dimensional relation between observed galaxy number density and template value, fits both a null model and a contamination model, evaluates

$4.10\sigma$4

calibrates significance using uncontaminated mocks, and, if significant, defines a weight map

$4.10\sigma$5

This is repeated iteratively because correcting one contaminant changes the residual correlations with the others. The paper does not provide an explicit symbolic recursion for the updated density field, and it does not write a formal stopping equation, but it states a significance threshold $4.10\sigma$6 for the tests shown (Rodríguez-Monroy et al., 9 Sep 2025).

The construction of the 1D relations is more explicitly specified than in Y3. Maps are degraded to $4.10\sigma$7, template values are binned into 10 equal-width bins, and the bin-averaged template coordinate and observed number density are

$4.10\sigma$8

typically normalized by the footprint-averaged $4.10\sigma$9 so that the null relation is a horizontal line at 1 (Rodríguez-Monroy et al., 9 Sep 2025).

The paper’s most illustrative examples are $b^3$0 in the first redshift bin and $b^3$1 in the fourth redshift bin. Under the seed mask alone, these relations span wider value ranges and contain extreme bins; linear fits have very high $b^3$2; and cubic fits improve markedly after the joint mask is applied. The authors stress that these contaminants induce $b^3$3 effects on observed number density and may not be flagged as significant with linear ISD, but do trigger the contamination metric with cubic ISD (Rodríguez-Monroy et al., 9 Sep 2025).

An important practical issue is resolution mismatch. Y6 measures the 1D response at $b^3$4 but applies the corrective weights at $b^3$5. The paper argues that outliers can be partially hidden by degradation or binning and then reappear when $b^3$6 is evaluated on the high-resolution map, especially for cubic and quadratic terms. The joint mask reduces these long-tailed weight distributions. This motivates the claim that masking is not merely a separate quality-control step but a precondition for stable non-linear ISD (Rodríguez-Monroy et al., 9 Sep 2025).

5. Validation, uncertainty propagation, and scientific impact

DES Y3 gives an extensive validation program for ISD using uncontaminated and contaminated log-normal mocks. The uncontaminated suite contains 1000 mocks per galaxy sample and is used to estimate the covariance of the 1D relations and calibrate $b^3$7 significance. For residual-systematics validation, 400 contaminated mocks are created by multiplying counts by the inverse of a data-derived weight map before Poisson sampling,

$b^3$8

with ENet-STD107 weights used for contamination injection and ISD-PC<50 used for cleaning, specifically as a conservative mismatch between contamination model and decontamination model (Rodríguez-Monroy et al., 2021).

Three biases are then studied. Estimator bias compares decontaminated and uncontaminated mean $b^3$9 when the same map list is used for contamination and correction; false-correction bias quantifies overcorrection from chance survey-property–LSS correlations in uncontaminated mocks; and residual systematic bias measures imperfect recovery on contaminated mocks. For $6.96\sigma$0, false-correction bias is generally negative but small, typically at most about 20% of the statistical error, while residual systematic bias is nearly unbiased on large scales but shows some undercorrection at small scales in low-$6.96\sigma$1 bins and some overcorrection in high-$6.96\sigma$2 bins (Rodríguez-Monroy et al., 2021).

The resulting parameter shifts in $6.96\sigma$3, tested on mean mock vectors, are all below $6.96\sigma$4. The specific examples quoted include a false-correction shift in $6.96\sigma$5 of $6.96\sigma$6 for MagLim and $6.96\sigma$7 for redMaGiC, with the largest reported bias-parameter shift about $6.96\sigma$8 (Rodríguez-Monroy et al., 2021). These results underwrite the abstract statement that the fiducial decontamination induces biases no greater than $6.96\sigma$9 in the $\Omega_m$0 plane.

The DES analysis also propagates decontamination uncertainty into the covariance rather than treating the corrected data vector as fixed. Following analytic marginalization, if a data vector $\Omega_m$1 is biased by an additive systematic template $\Omega_m$2,

$\Omega_m$3

with Gaussian prior width $\Omega_m$4, then

$\Omega_m$5

Applied to clustering,

$\Omega_m$6

so the added systematic covariance is the sum of outer products of the method-difference vector and the residual-bias vector (Rodríguez-Monroy et al., 2021). This is a notable feature of the DES ISD framework: decontamination is treated as part of the uncertainty model, not only as a preprocessing correction.

In Y6, the effect of masking and weighting on $\Omega_m$7 is studied separately through

$\Omega_m$8

The conclusion is that masking alone changes $\Omega_m$9, but less than the decontamination weights do; however, the masking impact is not negligible and can be comparable to statistical uncertainties in some redshift bins and angular scales (Rodríguez-Monroy et al., 9 Sep 2025).

6. Comparisons, limits, and broader methodological context

ISD is not the only decontamination method used in DES, and much of its methodological meaning emerges from comparison. In Y3, the principal alternative is ENet, a simultaneous regression method with mixed $7.69\sigma$0 regularization. ENet estimates all contamination amplitudes at once by maximizing

$7.69\sigma$1

and constructs weights

$7.69\sigma$2

A neural-network weighting method is also used as a nonlinear cross-check, with

$7.69\sigma$3

ISD differs from both by being a greedy sequential 1D regression that repeatedly projects onto the single most significant template and updates the residual field after each correction (Rodríguez-Monroy et al., 2021).

That design has strengths and weaknesses. Its one-dimensional response curves are visually interpretable, which is precisely why the Y6 paper emphasizes ISD when discussing masking and non-linear contamination (Rodríguez-Monroy et al., 9 Sep 2025). But Y3 also cautions that ISD can miss contamination distributed weakly across many maps, because it relies on marginal one-dimensional projections rather than a simultaneous multivariate fit (Rodríguez-Monroy et al., 2021). The Y6 distinction between a broader set of maps used for masking and a smaller set of 19 baseline templates used for weight estimation similarly reflects concern about correlated templates, redundancy, and overcorrection (Rodríguez-Monroy et al., 9 Sep 2025).

A second limitation is basis dependence. Y3 finds that moving from a restricted standard-map basis to a principal-component basis improves agreement between ISD and ENet, but using all 107 principal components overcorrects, suppressing $7.69\sigma$4 too strongly and producing weight maps correlated with DES $7.69\sigma$5 maps. The adopted PC<50 basis captures $7.69\sigma$6 of the survey-property variance while avoiding the strongest evidence for cosmological-signal leakage (Rodríguez-Monroy et al., 2021). This makes clear that orthogonalization is useful but not sufficient: high-order orthogonal modes can still absorb true LSS.

A third limitation is perturbativity. Y6 explicitly states that ISD assumes contamination is small enough to be treated perturbatively, and motivates masking extremes precisely to keep the analysis inside that regime. The paper also notes that some maps may themselves contain leaked LSS, citing $7.69\sigma$7 and dust-map differences dominated by cosmic infrared background contamination as examples too dangerous for weight estimation even if useful for masking (Rodríguez-Monroy et al., 9 Sep 2025).

Beyond galaxy clustering, related CMB work offers an instructive analogue. “Blind Map Level Systematics Cleaning: A Quadratic Estimator Approach” performs an iterative quadratic-estimator analysis to reconstruct a latent leakage field $7.69\sigma$8, build a contamination template, subtract it in map space, and repeat until the reconstruction noise and cleaned $7.69\sigma$9-mode spectrum converge to their floors (Williams et al., 2021). The map-level cleaning step is

$w(\theta)$00

and the paper introduces a Gaussian filtering scheme because naive Wiener filtering can diverge in low-noise cases (Williams et al., 2021). Although this work is not DES-style template weighting, it embodies the same broader ISD logic: infer a nuisance field from the contaminated data, subtract its map-level imprint, and iterate until self-consistency. A plausible implication is that “Iterative Systematics Decontamination” names not only one DES algorithm but a wider class of iterative nuisance-reconstruction-and-cleaning schemes.

The acronym itself is also ambiguous across fields. Several unrelated papers use “ISD” for “Information Set Decoding,” “Iterative Support Detection,” or “Iterative Salience Decoder,” none of which concern observational-systematics mitigation. For the decontamination sense discussed here, the DES Y3 and Y6 papers provide the clearest direct definitions and implementations (Rodríguez-Monroy et al., 2021, Rodríguez-Monroy et al., 9 Sep 2025).