Synthetic Data Pipeline

• Synthetic Data Pipeline is a framework that algorithmically generates artificial datasets by simulating real-world observational conditions and controlled corruption effects.
• It integrates end-to-end modules including physical modeling, noise injection, and calibration to produce realistic, analysis-ready data for benchmarking and algorithm development.
• Its application in fields like VLBI demonstrates its value in testing reconstruction fidelity, guiding instrument design, and validating computational methods under controlled scenarios.

Synthetic data pipelines are engineered frameworks designed to algorithmically generate artificial datasets for use in computational science, machine learning, and domain-specific simulations. These pipelines aim to replicate, extend, or augment real-world data while providing explicit control over the underlying generative and corruption processes. Their application is essential for robust algorithm development, instrument characterization, and systematic benchmarking in scenarios where real data are costly, rare, or logistically impractical to obtain. The field draws on advances in physical modeling, statistical noise simulation, and calibration emulation, exemplified in the context of very long baseline interferometry (VLBI) by the SYMBA pipeline (Roelofs et al., 2020), which models the full sequence from astrophysical signal propagation through data corruption and calibration.

1. End-to-End Pipeline Architecture and Workflow

A prototypical synthetic data pipeline, as instantiated by SYMBA, encompasses multiple tightly integrated stages that together mimic the complete observational, noise, and calibration pathway. The entire workflow is orchestrated via a central configuration, such as a master ASCII schedule in SYMBA, that stipulates all observational parameters—including schedule, antenna characteristics (dish diameter, aperture efficiency, receiver SEFD), and environmental conditions (precipitable water vapour, pressure, ground temperature).

The general workflow encompasses:

• Definition of input source models (ranging from analytic point sources to fully 3D GRMHD time-dependent simulations).
• Calculation of “ideal” instrument-ready visibilities via the 2D Fourier transform of the input sky model $I(l,m)$.
• Application of physical, atmospheric, and instrumental corruptions to visibilities, leveraging modules such as MEQSILHOUETTE.
• Calibration steps using algorithms such as Schwab–Cotton fringe fitting, amplitude calibration based on measured opacity, and optional network-based redundancy corrections (e.g., via rPICARD, HOPS, eht-imaging).
• Output of data in standardized formats (e.g., UVFITS) for downstream imaging and analysis.
• Workflow containerization (e.g., with Docker) is adopted to facilitate reproducibility and cross-platform deployment.

2. Physical Modeling of Signal Corruption

The fidelity of synthetic data depends critically on the accurate simulation of all relevant corruption mechanisms. In SYMBA, the signal corruption model is factored into atmospheric and instrumental domains:

• Atmospheric Effects: The mean and turbulent troposphere are treated distinctly. Amplitude attenuation and frequency-dependent phase slopes are calculated by integrating the radiative transfer equation

$$\frac{dI_\nu(s)}{ds} = \epsilon_\nu(s) - \kappa_\nu(s)I_\nu(s)$$

supplemented by Kirchhoff’s law: $\epsilon_\nu/\kappa_\nu = B_\nu(T)$, with $B_\nu(T)$ denoting the Planck function. Turbulent phase fluctuations, crucial at mm wavelengths, follow a power-law phase structure function:

$$D_\phi(\vec{x}, \vec{x}') \approx \mu\left(\frac{r}{r_0}\right)^\beta$$

where $\beta=5/3$ for Kolmogorov turbulence, $r_0$ is the phase coherence length, and $\mu = \csc(\mathrm{elevation})$ encapsulates airmass scaling.

• Instrumental Effects: Receiver/system noise is added per station using station effective flux density (SEFD), quantization efficiency $\eta_Q$, channel bandwidth $\Delta\nu$, and integration time $t_{\mathrm{int}}$:

$$\sigma_{mn} = \frac{1}{\eta_Q\sqrt{\frac{\mathrm{SEFD}_m\mathrm{SEFD}_n}{2\Delta\nu t_\mathrm{int}}}}$$

Antenna pointing errors induce amplitude losses according to the primary beam’s Gaussian profile (FWHM $\mathcal{P}_\mathrm{FWHM}$), modulated by a normally-distributed pointing offset $\rho$:

$$\frac{\Delta Z_{mn}}{Z_{mn}} = \exp\left[-8\ln^2(2)\left(\frac{\rho_m^2}{\mathcal{P}_\mathrm{FWHM},m^2} + \frac{\rho_n^2}{\mathcal{P}_\mathrm{FWHM},n^2}\right)\right]$$

Additionally, D-term polarization leakage and complex gain errors are incorporated.

3. Calibration and Data Recovery Steps

After imposing all corruption effects, realistic synthetic data are not directly usable until calibration is performed, reproducing the data processing applied to real observations. The sequence involves:

• Fringe Fitting: Application of the Schwab–Cotton algorithm to remove rapid tropospheric phase variations.
• Amplitude Calibration: Use of measured atmospheric opacity (attenuation by factor $\exp(\tau)$) and a priori gain models to calibrate amplitudes.
• Network Calibration: For arrays with intra-site baselines, gain errors are corrected by enforcing redundant station calibration.
• Data Averaging and Output: After calibration, data are averaged and exported for imaging, ensuring that the synthetic datasets are as realistic and analysis-ready as real-world VLBI data.

4. Quantitative Assessment via Case Studies

The effectiveness of synthetic data pipelines is established by systematic case studies:

Input Source	Corruption Profile	Key Observational Result	Quantitative Metric
Point source (4 Jy)	Full corruption + calib.	Recovery of amplitude/phase trends	Visibilities vs. flux
Geometric Crescent	Thermal noise, Full corr.	Impact on image fidelity	Comparison of reconstructions
GRMHD M87 Models	Full corruption + calib.	Distinction: ring thickness, knot	$\rho_{\mathrm{NX}}$ (norm. cross-corr.)

• Thermal jet vs. $\kappa$-jet (GRMHD): Key features such as emission ring geometry and jet sheath brightness are captured/reproduced with high fidelity, as measured by normalized cross-correlation

$$\rho_{\mathrm{NX}}(X,Y) = \frac{1}{N}\sum_i \frac{(X_i-\langle X\rangle)(Y_i-\langle Y\rangle)}{\sigma_X \sigma_Y}$$

Additions of new stations (GLT, KP, PDB, AMT) improve angular resolution, dynamic range, and the ability to quantitatively distinguish between theoretical models.

5. Impact of Pipeline Components and Design Trade-offs

The detailed physical and instrumental fidelity in the corruption and calibration stages is non-negotiable for simulating millimeter-wave VLBI. Atmospheric and instrumental corruptions dominate the observed visibilities—a pipeline omitting these (i.e., “thermal noise only”) produces overoptimistic reconstructions, uninformative regarding calibration robustness and imaging limitations.

Trade-offs include:

• Physical realism vs. computational complexity: High-fidelity turbulence and full calibration models increase computational demands but are required for scientifically meaningful tests.
• Calibration completeness: Omitting steps such as network calibration produces more realistic “imperfect” datasets for testing algorithmic resilience.

6. Implementation, Resources, and Reproducibility

SYMBA operationalizes the pipeline as a containerized, modular software stack, facilitating deployment across analysis environments and supporting reproducibility and scalability. Configuration via a single master file allows high-throughput experimentation across varying array configurations and weather scenarios. All corruption and calibration algorithms are open for parameterization, supporting robust simulation campaigns for future instrument design and algorithmic benchmarking.

7. Scientific and Practical Implications

Synthetic data pipelines, exemplified by SYMBA, serve as indispensable test benches for both instrument design and algorithm development. They:

• Quantify recoverability limits of physical source features in the presence of atmospheric and instrumental uncertainty.
• Enable targeted calibration algorithm development (testing fringe fitting, network calibration, and self-calibration robustness).
• Underpin array design decisions (e.g., station placement, SEFD optimization) via simulated upgrades and direct comparison of reconstruction metrics.
• Offer a reproducible mechanism for benchmarking theoretical model discrimination in light of real-world observability limits.

In sum, end-to-end synthetic data pipelines model every relevant aspect of the data acquisition, corruption, and recovery chain. Inclusion of realistic atmospheric and instrumental effects, as well as calibration emulation, is essential for accurate analysis of instrument performance, model-testing, and future instrument design. The availability of such pipelines accelerates methodological development and standardizes experimental evaluation frameworks within the astronomical community.