The Array Control and Data Acquisition software of the Cherenkov Telescope Array Observatory

Abstract: The Cherenkov Telescope Array Observatory (CTAO) aims to advance knowledge of the gamma-ray sky as the largest gamma-ray observatory ever built. The CTAO will be deployed at two sites, one in the Northern Hemisphere and the other in the Southern Hemisphere, containing telescopes of three sizes to cover different energy domains. Commissioning of the prototype CTAO Large-Sized Telescope (LST-1) is being finalized at the northern site, while three additional LSTs are under construction. Additional calibration and environmental monitoring instruments, such as laser imaging detection and ranging (LIDAR) systems and weather stations, will support telescope operations. The Array Control and Data Acquisition (ACADA) system serves as the central element for on-site CTAO operations. ACADA controls, supervises, and handles the data generated by the telescopes and the auxiliary instruments. It drives the efficient planning and execution of observations while managing the multi-gigabit-per-second data streams produced by each CTAO telescope. The ACADA system contains the CTAO Science Alert Generation Pipeline - a real-time data processing and analysis pipeline, dedicated to automatically generating science alert candidates as data are acquired. These science alerts, along with external alerts received from other scientific instruments, are managed by the Transients Handler (TH) component. The TH informs ACADA's Short-Term Scheduler (STS) about relevant science alerts, enabling modification of ongoing observations on sub-minute timescales. This capability for rapid response, combined with the fast slewing of CTAO telescopes, makes the Observatory an excellent instrument for studying high-impact astronomical transients.

• The paper demonstrates a robust system design that integrates modular control, resource management, and real-time data reduction for CTAO operations.
• It employs MBSE methodologies, hierarchical supervision, and multi-language software to meet stringent throughput and fault tolerance demands.
• The system is validated in single-telescope operation with plans for multi-telescope scalability, paving the way for future CTAO expansion.

ACADA: Array Control and Data Acquisition Software for CTAO

System Requirements and Design Principles

The ACADA system is architected to fulfill stringent operational, scalability, and fault tolerance requirements fundamental to the CTAO’s mission as a dual-site, multi-instrument gamma-ray observatory. Specification was achieved through MBSE methodologies, leveraging past experience from IACTs and iterative validation via formal reviews. Key design imperatives include simultaneous multi-subarray operation; support for up to 100 telescopes and extensive environmental/auxiliary instrumentation; real-time management of aggregate data rates exceeding 1 Tb/night; robust event filtering and compression to achieve sustained throughput below 5 Gb/s; automatic error and alarm management with actionable mitigation; seamless interface with transient alert networks for sub-minute response latency; and minimal-operator, high-availability control interface deployable under standard RHEL-compatible environments.

User access control delineates roles across Operator, Support Astronomer, Configuration Manager, privileged Operator+, and daytime Engineer/Technician functions, each mapped to over 100 UC-driven operational scenarios framing ACADA’s development lifecycle.

Architecture and Component Integration

The ACADA software stack is a highly modular, distributed system implemented in C++, Java, Python, and JavaScript, and underpinned by the Alma Common Software (ACS) middleware, with extensive reliance on proven industry technologies. Component orchestration and lifecycle management are realized through a hierarchical supervision tree paradigm, wherein systemic faults are confined via parent-child supervision and rapid root cause isolation.

Figure 1: ACADA’s system context, detailing external and CTAO-specific systems interfaced by ACADA.

High-level architecture partitions ACADA into critical subsystems: Resource Manager (RM), Central Control (CC), Human-Machine Interface (HMI), Array Data Handler (ADH), Science Alert Generation (SAG), Transients Handler (TH), Short-Term Scheduler (STS), Monitoring/Logging/Alarm (MON/AAS), Configuration Database, and Reporting System. Each subsystem exposes clearly defined interfaces and ports, facilitating decoupled development, integration, and scale-out.

Figure 2: Logical view of ACADA’s main components and data flows, emphasizing core interfaces and high-volume Cherenkov camera streams.

RM operates as a fault-tolerant meta-controller, maintaining cluster-wide state, orchestrating resource allocation, providing persistence and lookup services, and ensuring seamless recovery via cold/hot supervisor redundancy.

Component supervised instantiation follows the supervision tree model, minimizing system-wide restarts (Figure 3).

Figure 3: Diagram of the ACADA supervision tree for hierarchical component supervision and recovery.

CC governs SB and OB execution, coordinating observation block queuing, script-driven operational modes, resource allocation, and environmental contingency management via plugin-based extension. Observation and status synchronization is maintained with SAG, ADH, and STS for consistent system state propagation.

Data Acquisition, Reduction, and Real-Time Analysis

ADH orchestrates high-throughput data streams from Cherenkov cameras and auxiliary instrumentation, enforces data volume reduction through pixel and waveform relevance pruning (leveraging camera/pixel health metrics), and implements SWAT for cross-telescope event correlation and rejection. C++ and Python implementations assure performance for streaming and auxiliary tasks.

SAG executes online low-latency analysis through modular pipelines: image parameter extraction and fast reconstruction (SAG-RECO), online data quality assessment (SAG-DQ), high-level real-time science analysis (SAG-SCI), and supervisor modules. ML-based event classification (Random Forests) trained on MC simulations, quick-look DL3/DL5 generation, and subarray GTI metrics are standard. Real-time feedback (<20s latency) enables rapid internal alert generation and direct operator action.

TH processes both internal and incoming external alerts via a broker/receiver—validates, prioritizes, and programs urgent SB injections for scheduled reactivity in coordination with STS. Policy-driven observation ranking and automatic operator notification are integral, ensuring response within <4s of alert receipt.

Human-Machine Interfaces and Control Room Visualization

ACADA HMI delivers comprehensive, scalable visual supervision, status tracking, and manual override capabilities, designed for two operators across multi-monitor workstations plus wall panels. The FM/DM architecture aggregates data via Redis, exposes interactive web panels, and manages operator-driven schedule manipulations, health assessment, and quality assurance.

Figure 4: Distribution strategy of ACADA HMI panels across control room workstations and wall display.

Schedule and observation status are visualized in specialized panels, integrating SB/OB progress, AE allocation, execution phases, and dynamic color-coded status representation.

Figure 5: Schedule Overview Panel for user visualization of observation block progress, phases, and AE allocation.

Monitoring, Logging, Alarms, and Configuration

MON and AAS provide scalable time-series and alarm management—Apache Kafka and Cassandra back core monitoring, persistent SB/OB execution histories, and sophisticated alarm collection, filtering, and rule-based reduction. Alarms can trigger CC-driven mitigation scripts, with operator shelving/acknowledgment, Out-of-Service/Return-to-Service states, and advanced diagnosis via HMI.

Configuration data models are managed via OpenAPI/SQL/JSON/Schema with FastAPI, with full versioning, validation, and agentic assistants for cross-language interface and modeling development.

Software Engineering and Development Lifecycle

ACADA employs a rigorous incremental/iterative SDLC, combining agile sprints, CI/CD pipelines, SonarQube-based static analysis, Git submodule versioning anchored by tagged commits, and multi-layered automated testing stratified by runtime (quick/nightly/long). System integration proceeds via off-site cluster testing, interface validation, and full end-to-end operational verification before on-site deployment.

Figure 6: Overview of the Git workflow and branching policies utilized for ACADA subsystem and product integration.

Current Status and Future Roadmap

As of ACADA REL1, the system is verified for single-telescope operation (LST-1), with ~50% core codebase delivered and ~25% requirements formally validated. Subsequent releases (REL2/REL3) target multi-telescope support (up to five telescopes, dual subarrays, LST/MST/SST integration), auxiliary instrument expansion, autonomous nighttime operation, extended HMI panels, improved restart and reaction times, and full implementation of supervision tree-based robust recovery. Later releases will further scale AE support, concurrent subarrays, advanced alarm mechanisms, and KPI-driven performance reporting.

Conclusion

ACADA represents a mature, robust, modular solution for large-scale astronomical array management, incorporating automated high-throughput data acquisition, dynamic scheduling and rapid transient response, scalable monitoring and alarms, and operator-centric control interfaces. It is positioned to support the evolving operational complexity of the CTAO, with incremental enhancements aligning with telescope deployment, system integration, and advancing scientific requirements. Successful integration with LST-1 and formal QA reviews underscore readiness for multi-telescope configurations, paving the way for streamlined gamma-ray science operations as CTAO construction proceeds.