Online Observation Quality System (OOQS)

• OOQS is a real-time quality-assurance layer in the ASTRI SCADA system that monitors data streams from Cherenkov cameras and SI3 instruments.
• The system’s two-tier design uses ACS-based management components and an RTADP analysis pipeline on Kubernetes for modular scalability and fault isolation.
• OOQS implements configurable quality checks on trigger rates, ADC values, and gain ratios to enable automated corrective actions and support post-run analyses.

The Online Observation Quality System (OOQS) is the real-time observation-quality subsystem of the ASTRI Mini-Array Supervisory Control and Data Acquisition (SCADA) environment. Within the ASTRI Mini-Array—an international collaboration led by the Italian National Institute for Astrophysics and designed to operate nine Imaging Atmospheric Cherenkov Telescopes for very-high-energy gamma-ray astrophysics and stellar intensity interferometry—OOQS performs online data-quality checks on streams acquired from the Cherenkov cameras and, when deployed, the SI3 instrument. Its role is to detect abnormal conditions that would degrade science data, feed that information back to SCADA and the Operator, and persist quality assessments in a Quality Archive for later investigation (Parmiggiani et al., 21 Jul 2025).

1. Scientific and operational setting

The ASTRI Mini-Array is intended to construct and operate an array of nine telescopes at the Teide Observatory for observations in the TeV regime and for stellar intensity interferometry. OOQS operates at the on-site data center as one of the SCADA subsystems. Its stated purpose is twofold: immediate detection of abnormal conditions that would degrade science data, and archival persistence of the quality assessment to support post-run analysis and investigation (Parmiggiani et al., 21 Jul 2025).

The system is explicitly tied to the operational requirements of remote, unattended observations. In that setting, online quality assessment is not limited to passive monitoring. Abnormal-condition reports produced by OOQS are consumed by SCADA so that automated corrective actions can be taken to protect the telescopes’ duty cycle and prevent acquisition when quality requirements are not met. The Operator receives the same quality stream through the Human Machine Interface (HMI), which provides quick-look visualization and alerts in real time (Parmiggiani et al., 21 Jul 2025).

A recurrent point in the ASTRI software literature is that OOQS is not the data-acquisition system itself and not the supervisory controller itself. Rather, it occupies the quality-assessment layer between raw data acquisition and supervisory action. Earlier descriptions place it between the Array Data Acquisition System (ADAS), the control layer, the alarm and monitoring systems, and the Quality Archive, with the Operator using OOQS outputs as soon as they are produced (Parmiggiani et al., 2021).

2. Architectural composition in the updated design

The 2025 architecture divides OOQS into two main parts that are deployed differently but interoperate over explicit messaging interfaces: management components implemented with Alma Common Software (ACS) and an analysis pipeline implemented with the rta-dataprocessor (RTADP) framework and containerized on Kubernetes (Parmiggiani et al., 21 Jul 2025).

The management tier comprises three ACS components. OOQSManager controls the OOQS lifecycle and interfaces with the Telescope Control System (TCS) to receive start/stop commands for quality-analysis tasks. OOQSMonitor supervises the status and health of the data-quality pipelines, exposes telemetry to SCADA, and raises alarms when pipeline performance deviates from nominal ranges. OOQSCommand provides control operations to manage pipelines, including restart operations and threshold reconfiguration, and coordinates corrective actions when abnormal conditions are detected. These ACS components run on the site virtualization infrastructure and exchange intra-OOQS control and status messages through ZeroMQ, chosen for lightweight, low-latency messaging (Parmiggiani et al., 21 Jul 2025).

The analysis tier is implemented as a real-time, high-throughput, low-latency pipeline using RTADP. It is deployed in Docker containers orchestrated by Kubernetes in the on-site cluster. The architectural separation between ACS lifecycle management and RTADP analysis is one of the defining features of the updated system. The design supports elasticity, high availability, and reduced single points of failure. The deployment model is explicitly one pipeline instance per telescope; failure of a pipeline instance does not impact the other eight telescopes (Parmiggiani et al., 21 Jul 2025).

This separation also clarifies subsystem boundaries. ACS components handle lifecycle, monitoring, alarms, and supervisory coordination, while the RTADP layer handles the data path. A plausible implication is that this reduces coupling between control traffic and analysis traffic, which the design further reinforces by using ZeroMQ for internal control messaging and Kafka for high-throughput inter-subsystem data streams.

3. Data flow, interfaces, and messaging

The 2025 description presents OOQS as a streaming system centered on Kafka. During observations, Cherenkov cameras stream packets to ADAS, and ADAS publishes these packet streams onto Kafka topics for OOQS consumption. The camera packet types listed are scientific event packets, calibration packets, variance packets—including high-gain/low-gain information—and housekeeping. Each camera contains 37 Photon Detection Modules (PDMs), each with 64 pixels (Parmiggiani et al., 21 Jul 2025).

For each telescope, the RTADP pipeline consumes the Kafka streams and performs the defined checks. Results are then published back to Kafka for consumption by the Operator HMI and persisted to the Quality Archive. The HMI subscribes to the “quick-look” topics to display status and alerts in real time, while SCADA ingests abnormal-condition reports to decide automated responses such as pausing data acquisition or reconfiguring a subsystem (Parmiggiani et al., 21 Jul 2025).

The end-to-end flow is described in five stages. First, acquisition occurs at the Cherenkov cameras and ADAS. Second, ADAS streams the packets to OOQS through Kafka. Third, OOQS ingests and processes the streams telescope by telescope. Fourth, quality results are published to Kafka and stored in the Quality Archive. Fifth, SCADA and TCS use the resulting information for feedback and control, with TCS aligning quality checks to the observation plan through OOQSManager (Parmiggiani et al., 21 Jul 2025).

The interconnections are deliberately split by function. ADAS → OOQS uses Kafka for high-rate streaming; ACS ↔ Pipeline uses ZeroMQ for configuration, run control, and status collection; OOQS → HMI and Archive uses Kafka for result dissemination. The 2025 paper does not specify Kafka topic names or schemas, and it does not describe the Quality Archive database technology, schema, indices, or retention policies. It states only that results are stored for later investigation (Parmiggiani et al., 21 Jul 2025).

4. Implemented quality checks and online analyses

The Cherenkov-camera pipeline performs a defined set of real-time checks whose thresholds are configurable and will be optimized during commissioning. The checks enumerated in the updated architecture are: trigger number and trigger rate checks, per-pixel ADC value range checks, high-gain/low-gain consistency from variance packets, pointing deviation and PSF size checks, and quick-look sampling at one camera plot per second for each packet type (Parmiggiani et al., 21 Jul 2025).

For trigger stability, the pipeline tallies trigger numbers for the full camera and for each PDM, then verifies that the observed trigger rate $r$ lies within a configured interval $[r_{\min}, r_{\max}]$, expressed as

$r_{\min} \le r \le r_{\max}.$

To characterize stability, the description notes that standard rate statistics can be used over sliding windows, including a signal-to-noise ratio defined as $SNR = \mu/\sigma$, where $\mu$ and $\sigma$ are the mean and standard deviation of $r(t)$ over the window (Parmiggiani et al., 21 Jul 2025).

For per-pixel signal validation, OOQS checks that each pixel’s ADC sample lies within a configured range:

$A_{\min} \le ADC_i \le A_{\max}.$

Pixels with persistent out-of-range values are flagged as abnormal. For variance packets, the system computes a per-PDM high-gain/low-gain ratio

$R_{HL} = HG/LG,$

and checks it against a configured interval $[R_{\min}, R_{\max}]$ to detect channel mismatches or calibration drifts. Pointing and image quality are monitored through checks on pointing deviation $[r_{\min}, r_{\max}]$0 and PSF size, conceptually constrained by

$[r_{\min}, r_{\max}]$1

Deviations are interpreted as possible signatures of mispointing, focus issues, or environmental impacts (Parmiggiani et al., 21 Jul 2025).

The system also produces one camera plot per second for each packet type as a quick-look summary. The Operator HMI is described as displaying trigger rates and trigger counts per camera and per PDM, per-pixel ADC range summaries, high-gain/low-gain ratios by PDM, pointing deviation and PSF size indicators, and the one-camera-plot-per-second quick-look stream (Parmiggiani et al., 21 Jul 2025).

For stellar intensity interferometry, the 2025 description states that OOQS executes data-quality checks on SI3 observations, but SI3-specific metrics, thresholds, and decision logic are not described in that paper. The accompanying technical notes identify the second-order correlation

$[r_{\min}, r_{\max}]$2

and visibility $[r_{\min}, r_{\max}]$3 as typical SI3 quality quantities, but explicitly state that the paper does not specify the SI3 thresholds or decision logic and that these will be established during commissioning and instrument integration (Parmiggiani et al., 21 Jul 2025).

A frequent misconception is that OOQS is an all-purpose scientific reconstruction chain. The ASTRI descriptions instead define it as an online quality-analysis layer centered on trigger behavior, ADC validity, gain consistency, pointing, PSF, and quick-look visualization, with richer instrument-specific logic to be finalized during commissioning.

5. Historical development and architectural evolution

The OOQS architecture developed through several documented stages. The 2021 description presents a distributed design in which Redis Pub/Sub moved data from ADAS to per-telescope OOQS instances, while Slurm scheduled multiple data-quality analyses in parallel and Singularity packaged the software for reproducible deployment. That design included an OOQS Master, per-telescope OOQS Manager instances, a Cherenkov Camera Data Quality Checker (CCDQC), and an SI3 Data Quality Checker (SI3DQC), with the main challenge framed as online checking across nine telescopes and, for SI3, data rates up to 4.5 GB/s (Parmiggiani et al., 2021).

The 2023 architecture shifted the transport layer to Kafka and Avro, retained Slurm for scalable execution, and described OOQS as a SCADA subsystem designed to manage data rates up to 4.5 GB/s. In that formulation, ADAS streamed DL0 data to OOQS through Kafka, OOQS consumers deserialized Avro messages, and Slurm executed the analyses exploiting parallel analyses and scalability. It also described the system as running nine OOQS instances as Docker containers on the computing cluster (Parmiggiani et al., 2023).

The 2024 implementation paper introduced a prototype distributed application with three main components—Kafka Consumer, DQ-Analysis, and DQ-Aggregator—for the maximum array data rate of 1.15 Gb/s in the Cherenkov pipeline. The Consumer deserialized Avro messages, batched them into HDF5 files, DQ-Analysis scheduled analyses via Slurm, and DQ-Aggregator inserted aggregated results into the Quality Archive. Reported prototype figures include a C++ Kafka consumer throughput of 5800 packets/s, DQ-Analysis latency of about 260 ms per 1-second batch, DQ-Aggregator insertion time of about 30 ms per update, and an OOQS server write-speed requirement of 86 MB/s for nine telescopes (Castaldini et al., 2024).

The updated 2025 design departs from that batching-centered prototype by separating lifecycle management from data-path analysis, moving the analysis layer to RTADP on Kubernetes, and adopting a one-pipeline-per-telescope model under ACS control. The paper emphasizes modularity, resilience, low latency, and clean scaling to nine telescopes, but it does not quantify latency or throughput improvements relative to the previous design (Parmiggiani et al., 21 Jul 2025).

Stage	Main technologies	Characteristic features
2021 (Parmiggiani et al., 2021)	Redis Pub/Sub, Slurm, Singularity, ACS	OOQS Master/Manager, CCDQC, SI3DQC, parallel scheduling
2024 (Castaldini et al., 2024)	Kafka, Avro, HDF5, Slurm	Kafka Consumer, DQ-Analysis, DQ-Aggregator, measured prototype timings
2025 (Parmiggiani et al., 21 Jul 2025)	ACS, RTADP, Docker, Kubernetes, Kafka, ZeroMQ	Separation of management and processing, one pipeline per telescope

6. Resilience, commissioning, and open technical questions

Resilience is a central design criterion in the updated OOQS. The one-pipeline-per-telescope model improves fault isolation, since problems in one pipeline do not propagate to the others. Kubernetes provides health checks, automatic restarts, and resilient deployment on the on-site cluster, while ACS components on the virtualization layer provide controlled lifecycle management, externalized monitoring, and separation of concerns from the data path. ZeroMQ further decouples management and control from the analysis pipeline (Parmiggiani et al., 21 Jul 2025).

At the same time, the papers are explicit about what remains unspecified. The 2025 description does not report numeric message rates or end-to-end latency targets, beyond the statement that OOQS produces one camera plot per second for quick-look visualization. It also does not define a quality-flag taxonomy, aggregation rules across metrics, logging schemas, dashboards, Quality Archive schema, or Kafka topic names and schemas. Thresholds and decision criteria are configurable and will be tuned during commissioning (Parmiggiani et al., 21 Jul 2025).

The treatment of SI3 is similarly provisional. OOQS is stated to cover SI3 observations, but the implemented SI3-specific metrics and decision logic are deferred to future commissioning work. The same applies to additional engineering telemetry checks and more sophisticated camera image-parameter checks. The 2025 notes state that engineering telemetry checks such as temperature, voltage, and timing, and more advanced camera-image checks such as Hillas parameters and muon-ring based calibration, are common in IACT operations and could be incorporated as the system matures (Parmiggiani et al., 21 Jul 2025).

This leaves OOQS in a characteristic position within the ASTRI software stack: architecturally mature in its separation of control and processing, operationally oriented toward unattended observing, but still commissioning-driven in the detailed calibration of thresholds, interferometric metrics, and archive semantics. The published record consistently presents OOQS not as a finished static product, but as the online quality-assurance layer through which ASTRI intends to connect high-rate instrument telemetry, operator situational awareness, and automated supervisory action (Parmiggiani et al., 21 Jul 2025).