Diversity-aware Conformal Selection (DACS)

• DACS is a framework that integrates conformal prediction with diversity metrics to ensure selections are statistically valid and representative.
• It employs layered optimization and optimal stopping techniques to balance false discovery rate control with fairness and diversity in candidate pools.
• The method has practical applications in fields like drug discovery, hiring, and data pruning, demonstrating a strong trade-off between utility and equitable representation.

Diversity-aware conformal selection (DACS) is a methodological framework for subset selection that combines rigorous statistical validity—most notably, False Discovery Rate (FDR) control—with formal diversity constraints. Its aim is to produce selection sets (for tasks such as candidate ranking, resource allocation, recommendation, or experimental design) that are both high quality in terms of predictive or scientific utility and maximally representative across specified diversity dimensions, including demographic attributes, group representation, feature dissimilarity, or composite fairness objectives. The approach unifies conformal selection principles, optimal stopping theory, combinatorial optimization, and diversity metrics to achieve practical selections with guarantees on both selection power and diversity.

1. Theoretical Foundations: Conformal Selection and Diversity Metrics

Conformal selection methods use exchangeability, reweighting, or calibration-based scores to provide model-free prediction intervals or selection sets with finite-sample error control. When outputting a selection set $\mathcal{R}$, a typical guarantee is control of the FDR, defined for set selection as $$FDR = \mathbb{E}\left[\frac{|\mathcal{R} \cap H_0|}{|\mathcal{R}| \vee 1}\right]$$, where $H_0$ is the set of null (non-interesting) candidates. In diversity-aware selection, one augments this with an explicit diversity metric $\varphi(\mathcal{R})$ computed on candidate features, group labels, or similarity structures.

Key diversity measures include:

• Underrepresentation index: $$\varphi^{\mathrm{Underrep}}(\mathcal{R}) = \min_{c \in [C]} \frac{N_c(\mathcal{R})}{|\mathcal{R}|}$$, where $N_c(\mathcal{R})$ is the number of candidates of category $c$.
• Metric-based diversity (e.g., Sharpe ratio, Markowitz objective): $\varphi(\mathcal{R}) = f(|\mathcal{R}|, \Sigma)$, with $\Sigma$ as the similarity matrix between diversification features.
• Minimum pairwise distance, set entropy, or coverage-based objectives for fair representation.

These metrics quantify spread, balanced representation, and avoidance of redundancy in the selection set (Nair et al., 19 Jun 2025, Yang et al., 2019, Moumoulidou et al., 2020).

2. Optimization and Algorithmic Strategies

The DACS methodology formalizes the dual-objective optimization—maximizing diversity subject to FDR control—via constrained combinatorial or integer programming, optimal stopping, or iterative refinement.

Layered Optimization (Optimal Stopping)

Let $(W_1, ..., W_{n+m})$ be sorted imputed calibration/test scores. Define e-values $e_i^{(\tau)}$ at stopping time $\tau$, and seek selection sets $\mathcal{R}$ that are self-consistent with FDR constraints—i.e., $e_i^{(\tau)} \geq \frac{m}{\alpha |\mathcal{R}|}$ for $i \in \mathcal{R}$ (Nair et al., 19 Jun 2025). The optimization is layered:

1. Inner Problem: $$\max_{\mathcal{R}\ \mathrm{self-consistent}} \varphi(Z^{\mathrm{test}}_{\mathcal{R}})$$.
2. Outer Problem: Choose optimal stopping time $\tau^*$ to maximize expected diversity via the Snell envelope:

$$\tau^* = \max\{ t: R_t \geq E_t \},\quad R_t = \mathbb{E}_{P_{\mathrm{exch}}}[ O_t | \mathcal{F}_t ]$$

This guarantees that FDR is controlled at the desired level while the selected set is maximally diverse.

Diversity Constraints and Fairness Trade-offs

Balancing group representation (for example, via minimum quotas $\ell_{v,p}$) and in-group fairness (preserving ranking order within subgroups), as shown by the lexicographic minimization (leximin) approach, ensures no group is disproportionately disadvantaged by diversity constraints (Yang et al., 2019). This is typically solved as an integer linear program: $$\sum_{i\in I_v}\sum_{q=1}^p x_{i,q} \geq \ell_{v,p}\ \forall v,p;\quad a_v \geq q_v b_v$$ with auxiliary fairness constraints (ratio and aggregate fairness measures).

Diversity-aware Feature Selection and Data Pruning

For large-scale tuning tasks (e.g., LLMs), parameteric models in feature space are optimized to enforce both distributional consistency and feature-level diversity: $$L = -\mathbb{E}_{f_i \in F_D}\left[\frac{\mathrm{sim}(f_i, \theta_S^{(c_i)})}{\tau}\right] + \mathbb{E}_{j \in [m]}\left[\log\sum_{k \neq j}\exp\left(\frac{\mathrm{sim}(\theta_S^j, \theta_S^k)}{\tau}\right)\right]$$ (Lyu et al., 3 Jul 2025), enabling high-quality, diverse data selection with efficient batch-wise optimization and scalable to large datasets.

3. Diversity Objectives: Definitions and Implementation

Diversity is operationalized along several lines:

• Group-level Diversity: Ensuring minimum representation for protected groups (e.g., race, gender, socioeconomic status) via quota constraints or matroid bases (Moumoulidou et al., 2020, Yang et al., 2019).
• Feature/Cluster Diversity: Maximizing spread in embedding or physical feature space; for instance, using min pairwise distances or clustering approaches.
• Composite Diversity: Integrating different facets (perspectives, representativeness, contextualization) as quantified in the "Diversity Triangle," where an adjusted score is computed per candidate:

$$g(x) = f(x) + \lambda_1 r(x) + \lambda_2 p(x) + \lambda_3 c(x)$$

and selected sets are those surpassing a conformally calibrated threshold (Natarajan et al., 8 Oct 2024).

Diversity constraints are often balanced against selection utility; trade-off analysis quantifies the marginal decrease in utility for gains in diversity/fairness (Yang et al., 2019).

4. FDR Control and Statistical Guarantees

DACS consistently incorporates finite-sample FDR control across methodologies:

• Conformal p-values: For multivariate or multi-condition settings, regionally monotonic nonconformity scores $V(x, y)$ are constructed such that for $y \in R$, $V(x, y) \leq V(x, y')$ for $y' \in R^c$ (Bai et al., 1 May 2025, Hao et al., 9 Oct 2025). This ensures conservativeness of p-values and validity of selection.
• Global Correction: Multi-condition settings use aggregation of p-values across all conditions (e.g., intervals), followed by a global Benjamini–Hochberg procedure (Hao et al., 9 Oct 2025).

Compliance with exchangeability and martingale arguments under adaptive data exploration (e.g., ACS framework) ensures preservation of statistical guarantees even as models or selection orders adapt with incoming data or analyst preferences (Gui et al., 21 Jul 2025).

5. Applications and Empirical Demonstrations

DACS has demonstrated empirical effectiveness in diverse domains:

• Drug Discovery: Selection of chemically diverse candidate compounds with controlled FDR on binding affinity or multi-condition targets (Nair et al., 19 Jun 2025, Bai et al., 1 May 2025).
• Job Hiring: Balanced representation of demographic clusters while controlling for errors in hiring decisions (Nair et al., 19 Jun 2025).
• LLM Deployment: Filtering of diverse, trustworthy outputs (e.g., via p-value calibrated self-evaluation or feedback augmentation) (Gui et al., 21 Jul 2025).
• Code Recommendation and Data Pruning: Efficient selection of diverse and representative code samples for LLM fine-tuning, improving performance and training efficiency (Lyu et al., 3 Jul 2025).
• Active Learning: Density-aware sample selection elevates informativeness by favoring sparse, difficult regions (Kim et al., 2022).
• Fairness in Machine Learning: DCAST framework selects pseudo-labeled data that balances class-aware representation and diversity, mitigating selection bias even in complex high-dimensional settings (Tepeli et al., 30 Sep 2024).

6. Practical Implementation and Computational Considerations

Implementing DACS requires attention to tractable optimization and scalability:

• Relaxation and Approximation: Integer programs enforcing self-consistency/FDR are relaxed with continuous variables and Bernoulli rounding, attaining approximate error control (e.g., FDR bounded by $1.3\alpha$) (Nair et al., 19 Jun 2025).
• Monte Carlo Sampling: Expectations for stopping times and reward calculations are efficiently approximated via coupled sampling.
• Projected Gradient Solvers: Custom PGD solvers and adaptive restarts expedite optimization over diversity metrics.
• Batch-wise Feature Selection: Exploiting neural and code embeddings, parametric loss functions, and batch requirements enable scaling to millions of candidates (Lyu et al., 3 Jul 2025).

These methods maintain statistical validity while yielding highly diverse selection sets in practical computation time.

7. Impact, Extensions, and Future Directions

DACS establishes a framework where selection is not only statistically valid but adaptively diversified along quantifiable and application-specific axes. Real-world experimentation confirms trade-offs are often modest—a small loss in utility for considerable gains in fairness, diversity, and trustworthiness (Yang et al., 2019, Nair et al., 19 Jun 2025). This suggests wide applicability in resource-constrained and high-stakes domains, especially as diversity objectives interact with societal and regulatory imperatives.

Plausible implications are the extension of DACS to interactive, human-in-the-loop environments (ACS), multi-modal data, and continual adaptation as more information becomes available. Diversity-aware selection continues to evolve in its integration with fairness theory, optimal design, and scalable data analysis—highlighting the significance of its precise, combinatorial, and statistically rigorous underpinnings for future research and application.