IDOD: Independence and Orthogonal Discrimination

• IDOD is a framework that defines diversity as the maximization of independent, non-redundant features using rank-based and geometric formulations.
• It employs orthogonality constraints to replace hard-to-enforce statistical independence, improving fairness, robustness, and interpretability in models.
• The approach integrates independent enrichment, joint novelty discovery, and continuous incrementation, effectively balancing discrimination with diverse signal accumulation.

Independence-based Diversity and Orthogonality-based Discrimination (IDOD) denotes a family of ideas in which diversity is promoted by maximizing independence, decorrelation, or non-redundancy, while discrimination is strengthened by enforcing orthogonality among features, classifiers, prototypes, kernels, latent axes, or Jacobian subspaces. In a general rank-based formulation, dependence is identified with inability to create more diversity, and independence with the presence of maximum diversity (Galliani et al., 2019). In recent machine learning literature, the term also names a specific continuous category discovery method whose three modules are independent enrichment of diversity, joint discovery of novelty, and continuous increment by orthogonality (Jiang et al., 15 Aug 2025). Across these uses, IDOD is not a single algorithm but a recurring design principle: separate information sources so that each contributes distinct variation, and suppress redundant or confounding directions by geometric constraints.

1. Conceptual and mathematical foundation

The broadest formalization underlying IDOD is given by a diversity rank framework over finite subsets $x \subseteq_f M$, with a rank-like quantity $|x|$ interpreted as diversity (Galliani et al., 2019). In that framework, total dependence and total independence are defined by how much diversity is created when sets are combined:

$=(x,y) \iff |x \cup y| = |x|$

$x \perp y \iff |x \cup y| = |x| + |y|$

Under this view, total dependence means that adding $y$ creates no new diversity, while total independence means that diversity is additive and therefore maximal (Galliani et al., 2019). The same paper states that the framework covers linear dependence in algebra, dependence of variables in logic, entropy-based dependence in probability, and functional dependence in databases. In algebraic diversity, $|x|$ can be the dimension of the vector space spanned by $x$; in probability, $|x|$ can be Shannon entropy; in relational settings, $|x|$ can be the logarithm of the number of distinct tuples (Galliani et al., 2019).

This foundation is important for later IDOD-style methods because it turns “diversity” from an informal desideratum into a measurable property of representations, rules, kernels, or latent factors. A recurrent theme in the later literature is that maximal utility is not obtained by simply accumulating many signals; it is obtained by accumulating signals that contribute non-overlapping diversity.

2. Core geometric mechanisms

A central operationalization of IDOD is to replace hard-to-enforce statistical independence with orthogonality constraints. In fair representation learning, “orthogonality constraints as a proxy for independence” are used to disentangle meaningful and sensitive representations (Sarhan et al., 2020). In that setting, the encoder maps an input $\mathbf{x}$ to a target representation $|x|$0 and a sensitive representation $|x|$1, and the disentanglement loss is expressed as

$|x|$2

with orthogonal priors, while agnosticism to the sensitive attribute is reinforced by an entropy loss that pushes $|x|$3 toward a uniform distribution (Sarhan et al., 2020).

A more explicit linear construction appears in linear orthogonalization for fair and interpretable representations. For mean-centered feature vectors, Pearson correlation is

$|x|$4

so zero correlation is equivalent to geometric orthogonality. Given protected variables spanning a subspace with projector $|x|$5, each feature $|x|$6 is transformed to

$|x|$7

which is orthogonal to every protected feature and therefore linearly uncorrelated with them (He et al., 2019). The same method introduces an interpolating parameter $|x|$8,

$|x|$9

so that $=(x,y) \iff |x \cup y| = |x|$0 yields strict debiasing and $=(x,y) \iff |x \cup y| = |x|$1 yields no debiasing (He et al., 2019).

In generative identifiability, orthogonality is lifted from vectors to tangent subspaces. Content-style differential independence (CSDI) requires that infinitesimal variations in content and style induce orthogonal directions on the data manifold, equivalently

$=(x,y) \iff |x \cup y| = |x|$2

blockwise for all $=(x,y) \iff |x \cup y| = |x|$3 (Timilsina et al., 18 May 2026). This moves IDOD from feature decorrelation to a differential-geometric statement about local generative structure.

3. Fairness, invariance, and protected-attribute suppression

In fairness-oriented work, IDOD-style constructions are used to remove protected-attribute information while retaining task-relevant structure. The linear orthogonalization method of “Learning Fair and Interpretable Representations via Linear Orthogonalization” removes correlations between data and any number of protected variables, provides an adjustable fairness-accuracy trade-off through $=(x,y) \iff |x \cup y| = |x|$4, and is model-agnostic across linear regression, random forest, and multilayer perceptrons (He et al., 2019). The resulting features are reported as interpretable, with mean correlations to the original features of approximately $=(x,y) \iff |x \cup y| = |x|$5–$=(x,y) \iff |x \cup y| = |x|$6 on the tested datasets, and with regression coefficients mathematically identical or close to those for the original features when controlling for protected variables (He et al., 2019).

The paper further reports benchmark results on German Credit, COMPAS, and Adult Income, showing higher accuracy than state-of-the-art fair AI algorithms such as Zafar et al. and Unified Adversarial Invariance for a given level of fairness, and superior fairness-accuracy Pareto curves in the reported figures (He et al., 2019). Its computational cost is also described as extremely fast, with under 200 ms on the Adult dataset with 45k rows and over 100 features (He et al., 2019).

“Fairness by Learning Orthogonal Disentangled Representations” makes the same principle explicit in learned latent spaces: $=(x,y) \iff |x \cup y| = |x|$7 should be agnostic to the sensitive attribute $=(x,y) \iff |x \cup y| = |x|$8, $=(x,y) \iff |x \cup y| = |x|$9 should capture the sensitive information, and orthogonality plus entropy maximization are both required for the best invariance-fairness trade-off (Sarhan et al., 2020). The method is evaluated on five publicly available datasets and is reported to achieve state-of-the-art performance on three and comparable performance on the rest; ablative studies show that removing either orthogonality or entropy increases sensitive leakage (Sarhan et al., 2020).

In clinical decision-making, the orthogonal approach of “Thinking Outside the Box: Orthogonal Approach to Equalizing Protected Attributes” suppresses confounder effects by discriminant dimensionality reduction with explicit orthogonality between primary and protected attribute directions (Liu et al., 2023). For binary settings, $x \perp y \iff |x \cup y| = |x| + |y|$0 is the Fisher direction for the primary task, while $x \perp y \iff |x \cup y| = |x| + |y|$1 maximizes the protected-attribute Fisher criterion subject to $x \perp y \iff |x \cup y| = |x| + |y|$2 (Liu et al., 2023). The paper reports improved AUC scores, including an $x \perp y \iff |x \cup y| = |x| + |y|$3 improvement in CheXpert pleural effusion disease prediction, and frames orthogonality as a way to separate confounding variation from clinical inference (Liu et al., 2023).

4. Robustness and continuous category discovery

In adversarial robustness, IDOD appears as model-level diversity. “Towards Robust Neural Networks via Orthogonal Diversity” proposes DIversity via Orthogonality (DIO), in which multiple classifier heads share a backbone and are regularized to be mutually orthogonal through

$x \perp y \iff |x \cup y| = |x| + |y|$4

and are further separated by a margin-maximization term $x \perp y \iff |x \cup y| = |x| + |y|$5 (Fang et al., 2020). The full objective is

$x \perp y \iff |x \cup y| = |x| + |y|$6

The paper states that orthogonality alone is weak in high dimensions because random vectors tend to be nearly orthogonal, motivating the additional margin constraint (Fang et al., 2020). DIO is reported to improve robustness against white-box and black-box attacks, remain robust under adaptive attacks, and combine flexibly with TRADES and DDPM (Fang et al., 2020).

In continuous category discovery, IDOD is the explicit name of the method. The framework addresses the contradiction between diverse features for discovery and discriminative features for classification, the accumulation of pseudo-labeling errors in multi-stage novelty discovery, and the storage cost of replay-based forgetting mitigation (Jiang et al., 15 Aug 2025).

Module	Mechanism	Stated effect
Independent enrichment of diversity	contrastive training of backbone with learnable prototypes, then freezing	diverse features, stable representations
Joint discovery of novelty	single-stage discovery with dynamic pool, Affinity Propagation, GMM, dynamic merging	reduced error accumulation
Continuous increment by orthogonality	mutually orthogonal prototypes and representative representation replay	classification and forgetting mitigation

The orthogonality module generates mutually orthogonal prototypes via $x \perp y \iff |x \cup y| = |x| + |y|$7, trains a projector with cross-entropy against those prototypes, and uses representative representation replay rather than full data replay (Jiang et al., 15 Aug 2025). On CUB200, the reported values are $x \perp y \iff |x \cup y| = |x| + |y|$8, $x \perp y \iff |x \cup y| = |x| + |y|$9, and $y$0, and the storage overhead is reported as $y$1M versus $y$2M for GM, $y$3M for PA-CGCD, and $y$4M for PromptCCD (Jiang et al., 15 Aug 2025).

5. Data selection, testing, and structured latent spaces

IDOD-style reasoning also appears in data curation and statistical testing. “Selection of LLM Fine-Tuning Data based on Orthogonal Rules” represents each rule as a score vector over a representative sample batch, defines a rule correlation score

$y$5

and uses a determinantal point process with kernel $y$6 to select maximally orthogonal rules (Li et al., 2024). The selected rules then score the full dataset, and data are sampled stochastically with probability $y$7 (Li et al., 2024). The reported result is that DPP-based rule selection improves both alignment with human ratings and downstream LLM fine-tuning performance across IMDB, Medical, Math, and Code domains (Li et al., 2024).

A related data-selection perspective appears in ODiS, which argues that score-based selection collapses correlated dimensions and overlooks diversity (He et al., 21 Oct 2025). ODiS computes $y$8-dimensional scores, decorrelates them by PCA, trains RoBERTa-based regressors for the principal-component scores, and selects top-scored data within each orthogonal dimension (He et al., 21 Oct 2025). The paper reports less than $y$9 inter-dimension overlap, and states that models trained on ODiS-selected data significantly outperform baselines on downstream benchmarks (He et al., 21 Oct 2025).

In nonparametric hypothesis testing, DUAL formalizes the same tension between diversity and effectiveness. It measures kernel diversity by the covariance of kernel $|x|$0-statistics, aggregates them with

$|x|$1

and explicitly identifies a trade-off between diversity among kernels and the test power of individual kernels (Zhou et al., 13 Oct 2025). The paper provides Type-I error control, consistency, and asymptotic analysis, and reports superior performance on both two-sample and independence testing benchmarks (Zhou et al., 13 Oct 2025).

In representation learning, ODIN extends the orthogonality idea to significance-ordered nonlinear latent spaces. Its dendritic decoder reconstructs from every latent prefix, with

$|x|$2

while orthogonality is enforced by

$|x|$3

The paper states that, in the linear regime, this ordered loss recovers PCA principal components in order, and in the nonlinear regime it encourages the same structure while retaining the expressive power of deep networks (Schreiber et al., 6 Jul 2026).

6. Interpretive issues, scope, and limitations

Several papers clarify what IDOD does and does not guarantee. First, orthogonality is repeatedly used as a proxy for independence rather than full statistical independence (Sarhan et al., 2020). Linear orthogonalization guarantees zero linear Pearson correlation, not unrestricted nonlinear independence (He et al., 2019). A plausible implication is that “independence-based” in IDOD often means decorrelation or orthogonality relative to a chosen representation, test statistic, or Jacobian block, rather than a universal probabilistic factorization.

Second, diversity and discrimination are not always aligned automatically. DIO explicitly notes that orthogonality alone can be weak in high dimensions, because random vectors tend to be nearly orthogonal, and therefore adds a margin-maximization term (Fang et al., 2020). DUAL makes the same point in a different form by identifying a trade-off between the diversity among kernels and the test power of individual kernels (Zhou et al., 13 Oct 2025). The continuous category discovery version of IDOD likewise begins from “contradictory feature demands” between novel class discovery and classification (Jiang et al., 15 Aug 2025).

Third, orthogonality can be imposed at multiple structural levels. In fair AI it acts on features or latent codes (He et al., 2019, Sarhan et al., 2020); in clinical models, on discriminant directions (Liu et al., 2023); in robustness, on classifier heads (Fang et al., 2020); in data selection, on rule score vectors or PCA dimensions (Li et al., 2024, He et al., 21 Oct 2025); in testing, on covariance-corrected kernel statistics (Zhou et al., 13 Oct 2025); and in generative identifiability, on Jacobian subspaces (Timilsina et al., 18 May 2026). This suggests that IDOD is best understood as a transferable geometric principle rather than a domain-specific recipe.

Finally, CSDI shows that the orthogonality component of IDOD can even replace stronger assumptions. It allows content and style to be statistically dependent and the Jacobian to be dense, provided that their infinitesimal effects are blockwise orthogonal on the data manifold (Timilsina et al., 18 May 2026). That extension is significant because it preserves the IDOD emphasis on distinct directions of variation while relaxing classical independence and sparsity assumptions.