Search for quasar pairs with Gaia astrometric data. II. Photometric redshift prediction with machine learning for the MGQPC catalogue

Abstract: The identification of physically associated kiloparsec-scale quasar pairs is important for understanding galaxy evolution, the growth of supermassive black holes, and their co-evolution with host galaxies. However, their rarity and the high contamination from stellar superpositions and projected alignments require efficient pre-selection methods. We develop a machine-learning framework to produce photometric-redshift point estimates and redshift probability density functions for quasars, with the main goal of identifying high-probability quasar pair candidates in the MGQPC catalogue. We construct two large spectroscopically confirmed quasar samples with multi-wavelength photometry, based on SDSS and DESI Legacy Imaging Surveys data. CatBoost is used for point-estimate photometric-redshift regression, and FlexZBoost is used for full redshift-PDF estimation. The workflow achieves robust performance, with a normalised median absolute deviation of 0.036 and an outlier fraction of 5.6% on the test sample. Applying the trained model to the MGQPC catalogue, we identify 185 high-probability quasar pair candidates based on photometric-redshift consistency. Among them, 20 systems have been subsequently confirmed as genuine physical pairs by independent spectroscopic observations. The resulting MGQPC photometric-redshift catalogue provides a useful resource for future spectroscopic follow-up of quasar pairs and dual supermassive black holes.

• The paper establishes a validated ML pipeline using CatBoost and FlexZBoost for precise photometric redshift estimation in Gaia-selected quasar pairs.
• It demonstrates strong photo-z performance with σNMAD values as low as 0.036 and highlights the critical role of u-band photometry in reducing errors.
• The approach successfully identifies 185 high-probability quasar pair candidates from the MGQPC catalogue, paving the way for targeted spectroscopic follow-up.

Machine Learning-Based Photometric Redshift Prediction for Gaia-Selected Quasar Pairs

Scientific Motivation and Context

The identification of kiloparsec-scale quasar pairs is of considerable interest for studies of galaxy evolution and the growth and co-evolution of supermassive black holes (SMBHs). However, the inherent rarity of genuine physical quasar pairs and substantial contamination from random alignments—especially foreground stars—poses major selection and validation challenges. Upcoming wide-field imaging surveys such as LSST and Euclid will vastly increase the sample of quasar candidates, the majority of which will lack immediate spectroscopic redshifts. Therefore, robust and well-calibrated photometric redshift (photo-$z$) predictions, both point-estimate and full probabilistic, are needed for candidate pre-selection and statistical analyses. This study presents a dedicated ML framework for photo-$z$ estimation and probability density function (PDF) calibration, optimized for the MGQPC (Million Quasars–Gaia Pair Candidate) catalogue—an astrometric selection of quasar pairs derived from MQC and Gaia DR3.

Data and Training Sample Construction

Two large, spectroscopically-confirmed quasar samples were constructed for ML model training:

• KSTS: An SDSS-based quasar sample ($u,g,r,i,z$), providing key blue-end leverage via the $u$ band.
• KSTD: A DESI-LS-based sample ($g,r,i,z,W1,W2$), exploiting deep photometry over a broad sky but with incomplete $i$ coverage and no $u$-band.

Each was matched to Gaia DR3 for uniform astrometric filtration and multi-wavelength aperture/model photometry, and subjected to stringent selection in photometric quality and completeness. The spectroscopic redshift ($z_\mathrm{spec}$) distributions for these samples are shown in (Figure 1):

Figure 1: Spectroscopic redshift distributions for the KSTS (black) and KSTD (grey) quasar samples. The vertical axis is logarithmic.

This ensures a broad, representative training set with coverage into the high-$z$ quasar regime.

ML Methods and Feature Selection

Photo-$z$ regression was performed via CatBoost, an advanced implementation of gradient-boosted decision trees with robust handling of heterogeneous tabular data, missing values, and minimal hyperparameter sensitivity. For conditional density estimation (CDE) and production of full $z$0 PDFs, the FlexZBoost method was employed, which regresses basis function coefficients to achieve flexible, instance-wise photo-$z$1 PDFs.

Feature selection was guided by SHAP (Shapley Additive explanations), quantifying the marginal importance of each input variable (colours, magnitudes, morphological type) toward reducing the validation root mean squared error (RMSE). Only the features contributing above a defined threshold were retained, yielding a compact and physically motivated predictor set. The SHAP-based feature importance and RMSE progression for KSTS is shown in (Figure 2):

Figure 2: Normalised SHAP importance for the top features (bars, left axis), and validation RMSE as features are cumulatively added (dashed line, right axis).

Photometric Redshift Performance

Performance was evaluated for both point-estimate and PDF-based metrics:

• Normalised median absolute deviation (NMAD): $z$2 (KSTS, FlexZBoost), $z$3 (KSTS, CatBoost) on hold-out test sets.
• Outlier fraction: $z$4 (KSTS), higher for KSTD.
• RMSE and systematic errors are strongly reduced in the presence of $z$5-band photometry.
• Full PDF calibration was assessed via probability integral transform (PIT) diagnostics.

Scatter plots and residual distributions for $z$6 versus $z$7 are shown in (Figure 3):

Figure 3: Photometric versus spectroscopic redshift performance for FlexZBoost (left) and CatBoost (right); KSTD (top), KSTS (bottom). Dashed lines denote the catastrophic outlier threshold.

Photo-$z$8 PDFs exhibit close-to-uniform PIT histograms, confirming high-fidelity probabilistic calibration (Figure 4):

Figure 4: PIT-QQ calibration curves show agreement with the identity line, evidencing well-calibrated redshift PDFs.

Performance as a function of spectroscopic redshift reveals minimal bias and scatter for $z$9, with degradation at $u,g,r,i,z$0 where filter coverage is less diagnostic and emission line aliasing dominates. Catastrophic outlier rates and scatter increase for point-like morphologies and low-S/N sources. These redshift trends are explored in (Figure 5):

Figure 5: Redshift-binned bias ($u,g,r,i,z$1), scatter ($u,g,r,i,z$2), and outlier fraction by method and sample.

Extensive control experiments confirm the preeminent role of $u,g,r,i,z$3-band coverage in reducing both scatter and outlier fraction; survey-dependent photometric systematics (e.g., LS10 bright-source artifacts) provide a secondary contribution.

Application to MGQPC and Physical Pair Selection

Applying the trained models to the full MGQPC catalogue, photo-$u,g,r,i,z$4s and PDFs were computed for both members of all candidate pairs, all strictly disjoint from training data. Redshift consistency was quantified via line-of-sight velocity difference $u,g,r,i,z$5, using either the spectroscopic or predicted $u,g,r,i,z$6 for the primary and model-based $u,g,r,i,z$7 for the secondary. This yielded 185 high-probability candidate quasar pairs, with 20 of these subsequently confirmed by independent spectroscopy in the literature.

Implications and Theoretical Perspective

The study demonstrates the efficacy of a modern ML workflow in the challenging regime of quasar photo-$u,g,r,i,z$8 prediction—characterized by complex SED features, multi-modal colour degeneracies, and extreme rarity of physical pairs within dense contamination. The robust integration of interpretable feature ranking, tree-based regression for point estimates, and instance-wise CDE for calibrated PDFs achieves performance metrics competitive with the best in the literature, with explicit validation on both hold-out and external samples. Systematic limitations remain for extreme-redshift, low S/N, blended, or morphologically ambiguous cases and cannot be fully mitigated without improved filter coverage or next-generation spectroscopic samples.

Practically, this approach directly addresses the need for scalable, high-purity selection of rare, physically associated quasar systems from the deluge of candidates anticipated in the LSST, Euclid, and related survey era. The resulting MGQPC photo-$u,g,r,i,z$9 catalogue and filtered pair sample provide a prioritized target list for spectroscopic validation of dual SMBH candidates and critical input for statistical studies of AGN clustering, merger rates, and gravitational radiation backgrounds.

Theoretically, the framework provides a template for CDE-driven, uncertainty-calibrated cosmological inference in other high-dimensional, rare-object regimes, and foreshadows broader applications of interpretable ML in astronomical survey science.

Conclusion

This work establishes a robust, interpretable, and validated ML pipeline for photometric redshift estimation in the context of Gaia-selected quasar pairs, achieving strong photo-$u$0 accuracy and high-fidelity PDF calibration. The synergy of gradient-boosted decision trees and flexible CDE produces reliable candidate selection even in rare-object search scenarios, laying crucial groundwork for spectroscopic follow-up and large-scale dual SMBH studies in the survey-driven astronomy of the next decade (2605.09450).