DDF Pipeline: Radio Imaging & Calibration
- DDF Pipeline is a sophisticated data processing suite that employs direction-dependent calibration and imaging techniques in radio interferometry.
- It integrates specialized tools like killMS and DDFacet to convert MeasurementSets into high-quality FITS sky images for surveys such as LoTSS.
- Profiling reveals calibration as the primary performance bottleneck, highlighting optimization opportunities for high-volume deployments like the SKA.
DDF Pipeline is a radio astronomy data processing tool initially designed for the LOw-Frequency ARray (LOFAR) radio-telescope and presented as a candidate for processing data from the Square Kilometre Array (SKA). It is a composite application that transforms MeasurementSets, the standard data format in radio interferometry, into high-quality sky images in FITS format through direction-dependent calibration and imaging. The system is described as integrating specialized calibration and imaging software within a scriptable workflow, and its published characterization emphasizes both its software architecture and a coarse-grain profiling execution (Certenais et al., 3 Sep 2025).
1. Definition and software composition
The DDF Pipeline is a sophisticated software suite designed to process large volumes of radio astronomy data, specifically tailored for telescopes like LOFAR and under consideration for adoption by the SKA. Its core architecture is built around two main specialized tools: DDFacet, described as advanced imaging software for direction-dependent calibration and high-fidelity sky imaging, and killMS, described as calibration software for solving direction-dependent effects in radio interferometry (Li et al., 2018, Tasse, 2014).
This composition is central to the pipeline’s identity. Rather than functioning as a single monolithic executable, the DDF Pipeline orchestrates calibration and imaging steps across multiple subprocesses and data subsets. The emphasis on direction-dependent processing distinguishes it from workflows limited to direction-independent calibration, because the relevant effects include ionospheric or atmospheric distortions and antenna-specific responses, which are crucial for high-fidelity radio images (Certenais et al., 3 Sep 2025).
The input-output contract is explicit. The pipeline ingests MeasurementSets representing raw and intermediate visibilities from telescopes and produces science-oriented FITS sky images. In the broader workflow, it fits directly after central correlator output and handles calibration, correction, and imaging to deliver science-ready images (Certenais et al., 3 Sep 2025).
2. Processing workflow and data products
The processing workflow is organized in two main stages. Stage 1 processes a subset of the input MeasurementSets and follows the sequence calibration, imaging, and partial data products. Stage 2 operates on all available MeasurementSets to produce final, survey-quality images, using a similar sequence but on the full data volume (Certenais et al., 3 Sep 2025).
| Stage | Process | Calls |
|---|---|---|
| 1 | killMS | 24 |
| 1 | DDFacet | 12 |
| 2 | killMS | 96 |
| 2 | DDFacet | 7 |
In the default configuration, killMS is invoked 120 times, with 96 calls in stage 2. DDFacet is called 19 times, of which 12 process a subset, specifically 6 out of 24 data subsets, and the remaining 7 process the entire MeasurementSet in stage 2 (Certenais et al., 3 Sep 2025).
The final product images are 20,000×20,000 pixels at 1.5 arcsec resolution. This output scale is consistent with the pipeline’s role in survey imaging, where calibration and imaging must operate on large, partitioned visibility data before being reassembled into final products (Certenais et al., 3 Sep 2025).
3. Direction-dependent calibration, imaging, and survey role
A defining feature of the DDF Pipeline is the integration of direction-dependent processing. Both killMS and DDFacet target direction-dependent effects, including ionospheric or atmospheric distortions and antenna-specific responses. The pipeline therefore addresses one of the central technical challenges in modern low-frequency radio interferometry: the need to solve and image in the presence of spatially varying calibration terms (Certenais et al., 3 Sep 2025).
The system has been fundamental in producing images for large surveys like the LOFAR Two-metre Sky Survey and deep field projects. Reported usage includes processing 290 TB of data from 505 hours of observations over 7.5 years for the LoTSS and Deep Fields, producing science-ready images. This operational record is part of the rationale for its consideration in SKA contexts, where anticipated data volume is stated as 700 petabytes/year and therefore demands extreme performance and automation (Certenais et al., 3 Sep 2025).
The pipeline is also described as modular and parallel. Data is split into many subsets and processed largely in parallel, improving scalability and efficiency on high-performance compute resources. Its workflow is configurable and scriptable via configuration files, supporting both reproducibility and flexibility in subprocess selection, resolution, and output structure (Certenais et al., 3 Sep 2025).
4. Profiling environment and execution envelope
The published coarse-grain profiling used an HPE ProLiant DL385 Gen10 Plus system with 2x AMD EPYC 7543 32-Core CPUs, 120 CPUs total, and 512 GB RAM. The execution environment was a Singularity container with DDF Pipeline 3.1, DDFacet 0.7.2, and killMS 3.1 (Certenais et al., 3 Sep 2025).
| Metric | Value | Description |
|---|---|---|
| Input tarred MeasurementSets | 24 | Dataset count |
| Compressed input size | 38.72 GB | Tarred input volume |
| Decompressed input size | 134.4 GB | Expanded input volume |
| Run duration | 68.87 hours | End-to-end execution |
| Output size | 594 GB | Total output volume |
| Output images | 15 | Final image count |
| Image volume | 17.92 GB | Size of produced images |
This execution profile provides a concrete operating envelope for the tool on a single node. The output volume substantially exceeds the decompressed input size because the 594 GB total includes intermediate files, logs, and image products, not only the final FITS images (Certenais et al., 3 Sep 2025).
5. Runtime distribution, resource usage, and bottlenecks
The time distribution reported for the profiled run assigns 53.4% of total time, approximately 132,400 s, to killMS and 33.9%, approximately 84,000 s, to DDFacet. Miscellaneous or idle intervals with no profile account for 12.1%, while other steps such as mask creation and clustering are minor fractions (Certenais et al., 3 Sep 2025).
| Process | Share of total time | Characterization |
|---|---|---|
| killMS | 53.4% | Dominant calibration cost |
| DDFacet | 33.9% | Major imaging cost |
| Misc./idle | 12.1% | No-profile intervals |
Average resource usage further differentiates the two dominant components. For DDFacet, the reported averages are CPU User 25.1, CPU Sys 1.5, Mem Used 45.7 GB, and Mem Cached 428.5 GB. For killMS, they are CPU User 9.2, CPU Sys 2.5, Mem Used 33.6 GB, and Mem Cached 414.4 GB. The profiling interpretation identifies high memory usage, with hundreds of GB cached, and moderate per-process CPU usage, supporting significant parallelism (Certenais et al., 3 Sep 2025).
Disk activity is highly non-uniform. DDFacet records approximately 5,488 total reads and approximately 13,034 total writes, while killMS records approximately 11,641 total reads and approximately 18,398 total writes. Certain specialized steps are markedly more I/O intensive: DDFacet maskdico is reported at approximately 294,101 reads and approximately 15,249 writes, and DDFacet mkmask at approximately 647,272 reads and approximately 55,727 writes. The profiling interpretation therefore identifies image masking and cluster assignment as sources of very high I/O bursts (Certenais et al., 3 Sep 2025).
A common simplification is to treat imaging as the dominant expense in such workflows. The profiling results contradict that simplification: the performance bottleneck is calibration, with killMS identified as the most time-consuming and resource-intensive process (Certenais et al., 3 Sep 2025).
6. Operational significance and projected evolution
Within radio astronomy data systems, the DDF Pipeline is positioned as an automated, end-to-end imaging workflow that can be containerized via Singularity and orchestrated via cluster managers like SLURM. Its use in large surveys is presented as evidence of provenance, reproducibility, and standardization through scriptable, configurable workflows (Certenais et al., 3 Sep 2025).
The same profiling results also delimit the likely optimization surface. High memory usage is described as consistent with the requirements for large-scale FFT and gridding steps in image synthesis, while the spiky I/O pattern is identified as a potential avenue for optimization, especially if the pipeline is ported to HPC environments with different storage architectures. Upcoming work is described as involving porting to larger compute facilities, including the Jean Zay HPC center, in preparation for the data volume anticipated from next-generation radio telescopes (Certenais et al., 3 Sep 2025).
Taken together, these characteristics define the DDF Pipeline as a survey-scale, direction-dependent calibration and imaging system rather than merely a wrapper around individual astronomy codes. Its importance lies in combining DDFacet and killMS into a reproducible operational workflow, and in demonstrating, through coarse-grain profiling, where current large-scale radio astronomy processing spends time, memory, and I/O bandwidth as it moves from LOFAR-era survey production toward possible SKA deployment (Certenais et al., 3 Sep 2025).