YAML-Driven Workflow for Instrument Control

• YAML-driven workflow is a declarative configuration method that decouples hardcoded logic from operational commands, enabling modular instrument control.
• It employs asynchronous RPCs, schema-driven messaging, and a layered architecture to streamline real-time telemetry and controller responsiveness.
• The workflow integrates hardware drivers, TIMS clients, and PyQt-based GUIs to achieve precise control, scalability, and robust observatory performance.

A YAML-driven workflow in modern scientific instrumentation control refers to the use of structured, externally defined configuration and command schemas—often specified in YAML or JSON—to orchestrate the interaction of graphical interfaces, device drivers, and distributed control processes. Such an approach enables modular, reproducible, and flexible operation of complex hardware environments by abstracting operational sequences and parameters away from hardcoded logic into declarative specifications. The NEID spectrograph, operating on the WIYN 3.5-m telescope at Kitt Peak and documented in “Real-time exposure control and instrument operation with the NEID spectrograph GUI” (Gupta et al., 2022), exemplifies a design wherein distributed software components—hardware proxies, asynchronous control servers, and PyQt-based operator GUIs—communicate and coordinate via precisely described, interoperable APIs and message schemas. These systems leverage the Twisted Python framework and the TIMS client-server architecture to ensure efficient, reliable, and low-latency execution of device orchestration, exposure control, environmental monitoring, and metadata management.

1. Layered System Architecture of Instrument Control

The NEID implementation is structured as a multi-tiered architecture, supporting operational modularity and efficient development cycles:

• Hardware Drivers: Device-specific drivers, written in Python or C-wrapped Python, utilize Twisted abstractions such as serial.SerialPort, TCPClient, ZeroMQReceiver, and XMLRPCClient. These poll physical devices at fixed intervals, maintain a cached in-memory state machine, and expose RPC methods (e.g., get_pressure(), set_heater(state)).
• TIMS Clients: Lightweight Twisted applications interface with the drivers, maintain synchronized state caches, log telemetry locally at defined intervals, and register as RPC servers for higher-level system components. Examples include EnviroClient (managing 11 devices), CalibClient (14 devices), ArchonClient (CCD controller), and ExposureMeterClient.
• TIMS Relay / Server: The central Twisted server aggregates TCP/IP connections from clients and GUIs, routing RPCs and alert messages in a loosely coupled asynchronous manner, supporting both publish/subscribe and request/response interaction patterns.
• Graphical User Interfaces: The suite of operator GUIs—including the Spectrograph GUI, Exposure Control GUI, and Header-Editor GUI—are implemented as PyQt4, PyQt5, or PySide applications, embedding the Twisted reactor for non-blocking, event-driven interactions with backend TIMS clients.

These layers interact according to strict asynchronous programming paradigms to minimize UI latency and maximize operational resilience. Logical separation enables the robust addition or replacement of hardware subsystems by simply implementing corresponding driver and client classes.

2. Command Flow, Data Propagation, and Polling Strategies

Instrument operations are instantiated via GUI interactions that are propagated through the control hierarchy as asynchronous remote procedure calls (RPCs). A typical sequence for toggling a device, such as the ThAr calibration lamp, follows:

1. GUI Event: User toggles “ThAr ON” in the GUI.
2. RPC Dispatch: GUI issues set_lamp_state('ThAr', True) via the TIMS Relay.
3. Client Execution: The CalibClient invokes low-level driver commands, e.g., drive_serial("LAMP_ON\n").
4. Result Propagation: Acknowledgement is deferred back up the chain, and UI elements (button color, status fields) are updated accordingly.

Polling intervals are optimized for the telemetry type: the exposure meter and other fast telemetry are polled at 1 Hz, while environmental and hardware status updates occur at lower frequencies. The data-flow pattern is:

[Hardware] ← drivers poll @1 Hz → [Driver State Machine] → [Client RPC Server] ↑ ↑ │ │ [GUI Poller @1 Hz] ← RPC get_*() → [TIMS Relay] → [Client RPC Server] ↓ [Other Clients / Alerts]

Local caching of state machine values avoids redundant hardware queries, further reducing latency and network load.

3. Software Module Structure and Class Interfaces

A typical workflow to implement such a control system encompasses several coordinated modules, with clearly delineated class responsibilities:

• Base Client Wrapper (e.g., TimsClientBase): Encapsulates Twisted connection logic, deferred initialization, and generic RPC invocation:

  class TimsClientBase(object): def __init__(self, reactor, host, port): self.reactor = reactor self.host, self.port = host, port self._ready = Deferred() self._factory = PBClientFactory() reactor.connectTCP(host, port, self._factory) self._factory.onReady(self._ready.callback) @inlineCallbacks def remote_call(self, method_name, *args, **kwargs): root = yield self._ready remote = yield root.callRemote('getClient', self.client_name) result = yield remote.callRemote(method_name, *args, **kwargs) returnValue(result)

• Specialized Device Clients (e.g., CalibClient): Subclass the base, implement device-specific RPCs such as get_source_state, set_source_state, etc.
• PyQt GUI Integration: PyQt GUI class instances (e.g., SpectrographGUI, ExposureControlGUI) instantiate relevant TIMS clients, build interface elements, and start polling or control loops by scheduling periodic function calls via the Twisted reactor.
• Asynchronous Execution: Decorators like @inlineCallbacks provide non-blocking execution, with UI state always updated through deferred results.

A modular class structure with clear separation of GUI, client, and driver code facilitates maintainability and extension.

4. Message Schemas, Protocols, and Telemetry Handling

The communication between clients, server, and GUIs leverages a combination of serialization strategies:

• Twisted Perspective Broker: Handles pickled Python object RPCs for native applications within the Python ecosystem.
• JSON Over TCP: Provides schema-driven, language-agnostic message passing, e.g.,

  { "cmd": "set_lamp_state", "params": { "source": "ThAr", "state": true } }

• ZeroMQ Multipart Messages: Enables further decoupling for scalable architectures.
• Telemetry Logging: Device status and telemetry data are persisted to disk as CSV or JSON lines at ~1 Hz per channel, supporting later analysis and provenance.
• Alerting: Critical system alerts, such as power-fail or LN₂ fill completion, are distributed via SMTP or XML-RPC hooks to external paging/email systems.

Message schemas are strictly versioned and documented. Example schema for an exposure-meter update:

{ "device": "ExposureMeter", "timestamp": 1636567890.123, "instant_counts": [1023, 945, ... ], "cumulative_counts": [1023, 1968, ... ] }

This approach facilitates robust end-to-end testing and straightforward integration of new instrumentation or telemetry channels.

5. Operator Workflow and Interface Design Patterns

Key interaction patterns and practices for operator GUIs in this system include:

• Asynchronous RPC: All GUI interactions are deferred, ensuring the main thread never blocks and user experience remains responsive.
• UI Polling Rates: Recommended at 1 Hz for rapid-response telemetry (exposure meter) and at lower frequencies for slower environmental data to balance UI responsiveness and backend load.
• Color-Coded Indicators: Interface elements—buttons, status fields, plots—are color-coded for operational clarity: green (ready/nominal), orange (active), red (error/stale).
• Session Robustness: System state is preserved in TIMS server and client caches, allowing GUIs to be restarted mid-session without disrupting ongoing operations.
• Field Lockout and Metadata Management: Queue-operated fields are locked, with manual override controls for exceptional circumstances; header keyword editing is centralized in a dedicated Header Editor GUI for batch operations and metadata reinjection.
• Exposure Modes: Both fixed-time and SNR-thresholded exposure sequences are supported, with a hard upper bound to safeguard against runaway exposures.

These design patterns are critical in the context of distributed, multi-operator, and long-duration observational campaigns, ensuring consistent operational outcomes and traceable provenance.

6. Performance Metrics and Scaling Properties

Empirical performance evaluations from NEID operations include:

• Telemetry Update Latency: Exposure-meter telemetry updates at 1 Hz yield SNR trigger latencies ≲1 s.
• Imaging Readout: CCD readout time is approximately 30 s, underpinning automated exposure sequences.
• System Telemetry Throughput: The control stack successfully polls approximately 200 telemetry channels across 11 TIMS clients, with measured UI latency under 100 ms.
• Measurement Precision: The system enables sub-m/s radial-velocity precision, attributed to real-time SNR thresholding ($\sigma_\mathrm{SNR} \lesssim 2 \mathrm{\,cm/s}$ for SNR ≥120) and precise barycentric midtime computation from real-time exposure-meter counts.

This architecture demonstrates scalability in both the number of devices and operational throughput. Since each hardware subsystem is encapsulated in a Twisted-based client, new devices can be integrated by implementing appropriate driver and client classes, with immediate GUI availability and no additional process-management requirements.

7. Generalization and Modularity

The described pattern is both general and modular. By abstracting device logic into Twisted/TIMS clients and defining clear, schema-driven RPC protocols for GUIs, the workflow supports rapid extension: any new physical device requires only a driver and small Twisted client, after which all existing GUIs can poll or control the device immediately without the need for further threading or networking scaffolding. This architecture exemplifies a robust and extensible approach for observational instrument control, abstracting complexity behind declarative workflows and structured messaging (Gupta et al., 2022).