Coherency Control Strategy in Euclid SGS

• Coherency Control Strategy is a design discipline that standardizes quality metrics and data models across all pipeline levels in the Euclid SGS.
• It employs distributed pixel-level quality checks combined with centralized governance to ensure consistent, mission-quality data outputs.
• The strategy leverages common APIs, standardized thresholds, and incremental reports to harmonize quality evaluations across redundant Science Data Centers.

In the Euclid Science Ground Segment, a coherency control strategy denotes the design discipline by which the Data Quality Common Tools ensure that “data quality” has the same operational meaning at every pipeline stage, at every processing level, and across the redundant Science Data Centers despite high parallelism and distributed execution (Brescia et al., 2016). DQCT is a cross-cutting subsystem embedded from Level 1 through Level 3 and External Data, providing common tools and APIs for quality metrics, quality flags, statistical summaries, graphical diagnostics, and incremental quality reports stored in the Euclid Archive. The strategy is neither a separate pixel-level quality pipeline nor a purely centralized checker. Instead, pixel-level measures are implemented inside individual pipeline stages, while definitions, policies, aggregation, and reporting are managed at a higher level. This suggests that coherency control in Euclid is fundamentally a problem of semantic alignment, interface control, and software governance.

1. Architectural position within the Euclid SGS

The DQCT subsystem is described as a cross-cutting component of the Euclid SGS. Its software is embedded “everywhere” in the pipelines, spanning Level 1, Level 2, Level 3, and External Data, and is exposed as common tools and APIs that other pipeline modules call to compute quality metrics, generate quality flags, produce statistical summaries and plots, and build incremental quality reports stored in the Euclid Archive (Brescia et al., 2016).

A central architectural feature is the split between distribution and centralization. Pixel-level quality checks are implemented inside each individual pipeline stage for performance and locality, but responsibility for overall science data quality is managed at a higher level. The paper characterizes this as a system that is functionally distributed yet logically centralized. In practical terms, DQCT is not an auxiliary checker attached after the fact; it is a shared quality framework that pipeline stages are expected to use while still operating within their own execution context.

This architectural role matters because Euclid is a large, long-lived survey mission whose most valuable asset is its data. The paper therefore positions DQCT not as a convenience layer, but as the software substrate through which raw and derived products acquire scientifically interpretable quality semantics.

2. Standardization as the central coherency mechanism

The core problem identified by the paper is the risk of “potential diversity of solutions implemented for same functions” in a system with wide processing redundancy and high parallelism. The response is a standardization strategy built around common tools, common APIs, common development directives, and centrally designed quality data models (Brescia et al., 2016).

The APIs are intended to be available for all pipeline modules at all processing levels and to be shared between different modules, thereby ensuring full homogeneity. The quality information explicitly managed by DQCT includes quality flags, error estimates, and statistics such as mean, standard deviation, RMS, and S/N, as well as visual and graphical inspection products including thumbnails, trend analysis diagrams, histograms, and scatter plots. By routing such functions through common tools, the SGS avoids separate implementations of nominally identical quantities in different Science Data Centers or branches of the pipeline.

The conclusions further emphasize that, for each pipeline of the Euclid SGS, the DQCT team is supposed to design a specific data model dedicated to listing and describing all parameters and quality flags required for quality controls and analyses. This is a decisive coherency device. A centrally designed per-pipeline data model fixes parameter names, types, and the status of quantities as measurements, derived metrics, or flags. It therefore acts as a contract that each implementation must respect. The practical consequence is that “background noise,” “PSF size,” or “photometric zero point” are not merely labels attached to outputs; they are standardized objects with common semantics.

A common misconception is that coherency control is simply a matter of sharing code libraries. The paper presents a broader view. Shared software is only one part of the strategy; equally important are the shared data models and the explicit effort “to avoid redundancy and inhomogeneous calculation of the same quantities.”

3. Distributed implementation and centralized quality responsibility

Euclid’s data volume makes a separate pixel-level quality pass prohibitively expensive. The paper states that it is “prohibitively expensive to process large amount of data at the pixel level just for the purpose of quality evaluation,” and therefore all measures of quality at the pixel level are implemented in the individual pipeline stages and passed along as metadata in the production (Brescia et al., 2016).

This leads to a hybrid responsibility model. Implementation is distributed: calibration, reduction, and source-processing stages compute their own local quality measures. Governance is centralized: the DQCT subsystem defines required checks, quality parameters and flags, thresholding logic, and methods for long-term trend analysis. The point of the strategy is not to centralize computation, but to centralize meaning.

This division preserves computational efficiency while maintaining semantic coherence. It also ensures that downstream consumers of quality metadata are not forced to infer what each upstream module meant by a given flag or metric. In that sense, the coherency control strategy converts local measurements into globally interpretable quality information.

The same logic governs quality aggregation. Since pixel-level checks are performed locally, DQCT must specify how those local outputs become higher-level summaries and reports. This suggests that coherency control in Euclid is as much about disciplined metadata propagation as about metric computation itself.

4. Traceability across data levels and redundant Science Data Centers

The strategy extends across both the Euclid data-level hierarchy and the duplicated operational structure of the SGS. DQ common tools are foreseen to be available for all pipeline modules at all processing levels, and the resulting metadata are collected in incremental quality reports specific to each data level and stored in the Euclid Archive during pipeline processing (Brescia et al., 2016).

The paper explicitly states that DQCT APIs will also provide tools to collect quality reports at each transition between two data levels, “taking also traceability of the previous quality checks along the pipeline into account.” This traceability requirement is a central coherency mechanism. A Level-3 science product is not assessed only on its own final-form statistics; its quality record is linked to earlier checks on calibrated data, telemetry, and associated intermediate products. The strategy therefore enforces lineage of quality metadata, not just lineage of science products.

The same requirement appears at the organizational level. Euclid uses a mechanism of double Science Data Center for each processing function. High redundancy and high parallelism create a straightforward coherency risk: different centers might implement similar quality functions differently. The paper’s answer is a single DQCT team, common development directives defined by the System Team in architecture design, software development rules and processes, Euclid Archive System, and Monitoring and Control, plus testing and validation in collaboration with the Finnish Science Data Center.

Although the paper does not specify explicit reconciliation algorithms for outputs from redundant centers, the design intent is clear. Coherence is produced upstream by forcing common APIs, shared models, and common reporting structures before divergent implementations can create incompatible quality semantics.

5. Quality domains, thresholds, and decision logic

The paper identifies several major domains of quality evaluation relevant to imaging surveys: image artifact flagging, image background characterization, delivered image quality, astrometric fidelity and stability, and photometric fidelity and stability (Brescia et al., 2016). Examples include static bad pixels, cosmic rays, saturation, satellite trails, electronic cross-talk, ghost images, scattered light, detector health, moonlight, fringing, sky glow, PSF size, PSF shape or ellipticity, PSF spatial variation, uniformity of depth, and stellar locus dispersion in color-magnitude and color-color space.

Within the coherency control strategy, the critical point is not merely that these quantities are measured, but that their definitions and thresholds are standardized. The paper states that DQCT will contribute during commissioning and early operations to calibrate and assess stability and repeatability, and in particular to correctly set thresholds for quality parameters and flags. Thresholds are therefore mission-level control parameters rather than local coding choices.

This is linked directly to pipeline decision logic. The paper states that analysis of pipeline stage quality problems with DQCT will inform downstream decisions on whether to abort or otherwise alter processing. Coherency here means that a given off-nominal condition should trigger the same interpretation regardless of where it is encountered. If one branch treats a PSF-related flag as advisory while another treats it as fatal, the SGS would lose the possibility of coherent trend analysis and coherent downstream processing behavior.

One of the three main tasks of DQCT is “the automated computation and flagging of off-nominal conditions,” another is “long-term trend analyses,” and the third is “to avoid redundancy and inhomogeneous calculation of the same quantities.” The strategy binds these three together: off-nominal detection must be standardized if long-term trending is to remain scientifically meaningful.

6. Governance, implementation practice, and conceptual significance

Several governance elements sustain the strategy. The DQCT team is distributed across institutes in Finland, Italy, and Norway under Italian responsibility. Development follows System Team directives for architecture, software development rules and processes, archive integration, and monitoring and control. DQCT packages are tested in collaboration with the Finnish Science Data Center, and the Euclid Archive is expected to host a visualization and inspection tool supposed to be a tailored version of QualityWise (Brescia et al., 2016).

These practices address a practical challenge emphasized by the paper: DQCT’s path to realization is likely to differ from that of other SGS subsystems because data quality tooling must be widely spread over all pipeline segments and data levels while minimizing diversity of solutions for similar functions. In a more conventional subsystem, localized ownership can tolerate some variation in internal implementation. In DQCT, by contrast, local divergence directly threatens the comparability of quality assessments across time, centers, and data products.

A second misconception is that coherency control is synonymous with centralization. The paper argues for the opposite arrangement: local execution, centralized semantics. Pixel-level computations remain close to the data, but the criteria by which those computations become mission-quality evidence are standardized. A plausible implication is that Euclid treats quality information as infrastructure, not merely as monitoring output.

In broader methodological terms, the coherency control strategy can be summarized as central governance of definitions, data models, and policies; distributed implementation through common tools and APIs; standardized metadata and incremental reports stored in the archive; and trend analysis grounded in homogeneous inputs. Within Euclid, this is the mechanism that allows a massively parallel and redundant processing system to behave as a single scientific instrument with respect to data quality.