---
title: Online Observation Quality System (OOQS)
url: https://www.emergentmind.com/topics/online-observation-quality-system-ooqs
type: topic
---

# Online Observation Quality System (OOQS)

The **Online Observation Quality System (OOQS)** is the real-time observation-quality subsystem of the ASTRI Mini-Array **Supervisory Control and Data Acquisition (SCADA)** environment. Within the ASTRI Mini-Array—an international collaboration led by the Italian National Institute for Astrophysics and designed to operate nine **Imaging Atmospheric Cherenkov Telescopes** for very-high-energy gamma-ray astrophysics and **stellar intensity interferometry**—OOQS performs online data-quality checks on streams acquired from the Cherenkov cameras and, when deployed, the SI3 instrument. Its role is to detect abnormal conditions that would degrade science data, feed that information back to SCADA and the Operator, and persist quality assessments in a **Quality Archive** for later investigation [2507.15656].

## 1. Scientific and operational setting

The ASTRI Mini-Array is intended to construct and operate an array of nine telescopes at the Teide Observatory for observations in the TeV regime and for stellar intensity interferometry. OOQS operates at the on-site data center as one of the SCADA subsystems. Its stated purpose is twofold: immediate detection of abnormal conditions that would degrade science data, and archival persistence of the quality assessment to support post-run analysis and investigation [2507.15656].

The system is explicitly tied to the operational requirements of remote, unattended observations. In that setting, online quality assessment is not limited to passive monitoring. Abnormal-condition reports produced by OOQS are consumed by SCADA so that automated corrective actions can be taken to protect the telescopes’ duty cycle and prevent acquisition when quality requirements are not met. The Operator receives the same quality stream through the **Human Machine Interface (HMI)**, which provides quick-look visualization and alerts in real time [2507.15656].

A recurrent point in the ASTRI software literature is that OOQS is not the data-acquisition system itself and not the supervisory controller itself. Rather, it occupies the quality-assessment layer between raw data acquisition and supervisory action. Earlier descriptions place it between the **Array Data Acquisition System (ADAS)**, the control layer, the alarm and monitoring systems, and the Quality Archive, with the Operator using OOQS outputs as soon as they are produced [2108.04515].

## 2. Architectural composition in the updated design

The 2025 architecture divides OOQS into two main parts that are deployed differently but interoperate over explicit messaging interfaces: **management components** implemented with **Alma Common Software (ACS)** and an **analysis pipeline** implemented with the **rta-dataprocessor (RTADP)** framework and containerized on **Kubernetes** [2507.15656].

The management tier comprises three ACS components. **OOQSManager** controls the OOQS lifecycle and interfaces with the **Telescope Control System (TCS)** to receive start/stop commands for quality-analysis tasks. **OOQSMonitor** supervises the status and health of the data-quality pipelines, exposes telemetry to SCADA, and raises alarms when pipeline performance deviates from nominal ranges. **OOQSCommand** provides control operations to manage pipelines, including restart operations and threshold reconfiguration, and coordinates corrective actions when abnormal conditions are detected. These ACS components run on the site virtualization infrastructure and exchange intra-OOQS control and status messages through **ZeroMQ**, chosen for lightweight, low-latency messaging [2507.15656].

The analysis tier is implemented as a real-time, high-throughput, low-latency pipeline using RTADP. It is deployed in **Docker** containers orchestrated by **Kubernetes** in the on-site cluster. The architectural separation between ACS lifecycle management and RTADP analysis is one of the defining features of the updated system. The design supports elasticity, high availability, and reduced single points of failure. The deployment model is explicitly **one pipeline instance per telescope**; failure of a pipeline instance does not impact the other eight telescopes [2507.15656].

This separation also clarifies subsystem boundaries. ACS components handle lifecycle, monitoring, alarms, and supervisory coordination, while the RTADP layer handles the data path. A plausible implication is that this reduces coupling between control traffic and analysis traffic, which the design further reinforces by using ZeroMQ for internal control messaging and Kafka for high-throughput inter-subsystem data streams.

## 3. Data flow, interfaces, and messaging

The 2025 description presents OOQS as a streaming system centered on **Kafka**. During observations, Cherenkov cameras stream packets to ADAS, and ADAS publishes these packet streams onto Kafka topics for OOQS consumption. The camera packet types listed are **scientific event packets**, **calibration packets**, **variance packets**—including high-gain/low-gain information—and **housekeeping**. Each camera contains **37 Photon Detection Modules (PDMs)**, each with **64 pixels** [2507.15656].

For each telescope, the RTADP pipeline consumes the Kafka streams and performs the defined checks. Results are then published back to Kafka for consumption by the Operator HMI and persisted to the Quality Archive. The HMI subscribes to the “quick-look” topics to display status and alerts in real time, while SCADA ingests abnormal-condition reports to decide automated responses such as pausing data acquisition or reconfiguring a subsystem [2507.15656].

The end-to-end flow is described in five stages. First, acquisition occurs at the Cherenkov cameras and ADAS. Second, ADAS streams the packets to OOQS through Kafka. Third, OOQS ingests and processes the streams telescope by telescope. Fourth, quality results are published to Kafka and stored in the Quality Archive. Fifth, SCADA and TCS use the resulting information for feedback and control, with TCS aligning quality checks to the observation plan through OOQSManager [2507.15656].

The interconnections are deliberately split by function. **ADAS → OOQS** uses Kafka for high-rate streaming; **ACS ↔ Pipeline** uses ZeroMQ for configuration, run control, and status collection; **OOQS → HMI and Archive** uses Kafka for result dissemination. The 2025 paper does not specify Kafka topic names or schemas, and it does not describe the Quality Archive database technology, schema, indices, or retention policies. It states only that results are stored for later investigation [2507.15656].

## 4. Implemented quality checks and online analyses

The Cherenkov-camera pipeline performs a defined set of real-time checks whose thresholds are configurable and will be optimized during commissioning. The checks enumerated in the updated architecture are: trigger number and trigger rate checks, per-pixel ADC value range checks, high-gain/low-gain consistency from variance packets, pointing deviation and PSF size checks, and quick-look sampling at one camera plot per second for each packet type [2507.15656].

For trigger stability, the pipeline tallies trigger numbers for the full camera and for each PDM, then verifies that the observed trigger rate $r$ lies within a configured interval $[r_{\min}, r_{\max}]$, expressed as
$$
r_{\min} \le r \le r_{\max}.
$$
To characterize stability, the description notes that standard rate statistics can be used over sliding windows, including a signal-to-noise ratio defined as $SNR = \mu/\sigma$, where $\mu$ and $\sigma$ are the mean and standard deviation of $r(t)$ over the window [2507.15656].

For per-pixel signal validation, OOQS checks that each pixel’s ADC sample lies within a configured range:
$$
A_{\min} \le ADC_i \le A_{\max}.
$$
Pixels with persistent out-of-range values are flagged as abnormal. For variance packets, the system computes a per-PDM high-gain/low-gain ratio
$$
R_{HL} = HG/LG,
$$
and checks it against a configured interval $[R_{\min}, R_{\max}]$ to detect channel mismatches or calibration drifts. Pointing and image quality are monitored through checks on pointing deviation $\Delta \theta$ and PSF size, conceptually constrained by
$$
|\Delta \theta| \le \Delta \theta_{\max}, \qquad PSFsize \le PSF_{\max}.
$$
Deviations are interpreted as possible signatures of mispointing, focus issues, or environmental impacts [2507.15656].

The system also produces one camera plot per second for each packet type as a quick-look summary. The Operator HMI is described as displaying trigger rates and trigger counts per camera and per PDM, per-pixel ADC range summaries, high-gain/low-gain ratios by PDM, pointing deviation and PSF size indicators, and the one-camera-plot-per-second quick-look stream [2507.15656].

For **stellar intensity interferometry**, the 2025 description states that OOQS executes data-quality checks on SI3 observations, but SI3-specific metrics, thresholds, and decision logic are not described in that paper. The accompanying technical notes identify the second-order correlation
$$
g^{(2)}(\tau)=\frac{\langle I(t)I(t+\tau)\rangle}{\langle I(t)\rangle^2}
$$
and visibility $V=|\gamma|$ as typical SI3 quality quantities, but explicitly state that the paper does not specify the SI3 thresholds or decision logic and that these will be established during commissioning and instrument integration [2507.15656].

A frequent misconception is that OOQS is an all-purpose scientific reconstruction chain. The ASTRI descriptions instead define it as an online quality-analysis layer centered on trigger behavior, ADC validity, gain consistency, pointing, PSF, and quick-look visualization, with richer instrument-specific logic to be finalized during commissioning.

## 5. Historical development and architectural evolution

The OOQS architecture developed through several documented stages. The 2021 description presents a distributed design in which **Redis Pub/Sub** moved data from ADAS to per-telescope OOQS instances, while **Slurm** scheduled multiple data-quality analyses in parallel and **Singularity** packaged the software for reproducible deployment. That design included an **OOQS Master**, per-telescope **OOQS Manager** instances, a **Cherenkov Camera Data Quality Checker (CCDQC)**, and an **SI3 Data Quality Checker (SI3DQC)**, with the main challenge framed as online checking across nine telescopes and, for SI3, data rates up to **4.5 GB/s** [2108.04515].

The 2023 architecture shifted the transport layer to **Kafka** and **Avro**, retained **Slurm** for scalable execution, and described OOQS as a SCADA subsystem designed to manage data rates up to **4.5 GB/s**. In that formulation, ADAS streamed DL0 data to OOQS through Kafka, OOQS consumers deserialized Avro messages, and Slurm executed the analyses exploiting parallel analyses and scalability. It also described the system as running nine OOQS instances as **Docker** containers on the computing cluster [2302.13603].

The 2024 implementation paper introduced a prototype distributed application with three main components—**Kafka Consumer**, **DQ-Analysis**, and **DQ-Aggregator**—for the maximum array data rate of **1.15 Gb/s** in the Cherenkov pipeline. The Consumer deserialized Avro messages, batched them into **HDF5** files, DQ-Analysis scheduled analyses via Slurm, and DQ-Aggregator inserted aggregated results into the Quality Archive. Reported prototype figures include a **C++ Kafka consumer** throughput of **5800 packets/s**, **DQ-Analysis** latency of about **260 ms per 1-second batch**, **DQ-Aggregator** insertion time of about **30 ms per update**, and an OOQS server write-speed requirement of **86 MB/s** for nine telescopes [2404.02075].

The updated 2025 design departs from that batching-centered prototype by separating lifecycle management from data-path analysis, moving the analysis layer to **RTADP on Kubernetes**, and adopting a **one-pipeline-per-telescope** model under ACS control. The paper emphasizes modularity, resilience, low latency, and clean scaling to nine telescopes, but it does not quantify latency or throughput improvements relative to the previous design [2507.15656].

| Stage | Main technologies | Characteristic features |
|---|---|---|
| 2021 [2108.04515] | Redis Pub/Sub, Slurm, Singularity, ACS | OOQS Master/Manager, CCDQC, SI3DQC, parallel scheduling |
| 2024 [2404.02075] | Kafka, Avro, HDF5, Slurm | Kafka Consumer, DQ-Analysis, DQ-Aggregator, measured prototype timings |
| 2025 [2507.15656] | ACS, RTADP, Docker, Kubernetes, Kafka, ZeroMQ | Separation of management and processing, one pipeline per telescope |

## 6. Resilience, commissioning, and open technical questions

Resilience is a central design criterion in the updated OOQS. The one-pipeline-per-telescope model improves fault isolation, since problems in one pipeline do not propagate to the others. Kubernetes provides health checks, automatic restarts, and resilient deployment on the on-site cluster, while ACS components on the virtualization layer provide controlled lifecycle management, externalized monitoring, and separation of concerns from the data path. ZeroMQ further decouples management and control from the analysis pipeline [2507.15656].

At the same time, the papers are explicit about what remains unspecified. The 2025 description does not report numeric message rates or end-to-end latency targets, beyond the statement that OOQS produces one camera plot per second for quick-look visualization. It also does not define a quality-flag taxonomy, aggregation rules across metrics, logging schemas, dashboards, Quality Archive schema, or Kafka topic names and schemas. Thresholds and decision criteria are configurable and will be tuned during commissioning [2507.15656].

The treatment of SI3 is similarly provisional. OOQS is stated to cover SI3 observations, but the implemented SI3-specific metrics and decision logic are deferred to future commissioning work. The same applies to additional engineering telemetry checks and more sophisticated camera image-parameter checks. The 2025 notes state that engineering telemetry checks such as temperature, voltage, and timing, and more advanced camera-image checks such as **Hillas parameters** and **muon-ring based calibration**, are common in IACT operations and could be incorporated as the system matures [2507.15656].

This leaves OOQS in a characteristic position within the ASTRI software stack: architecturally mature in its separation of control and processing, operationally oriented toward unattended observing, but still commissioning-driven in the detailed calibration of thresholds, interferometric metrics, and archive semantics. The published record consistently presents OOQS not as a finished static product, but as the online quality-assurance layer through which ASTRI intends to connect high-rate instrument telemetry, operator situational awareness, and automated supervisory action [2507.15656].

Source: https://www.emergentmind.com/topics/online-observation-quality-system-ooqs