Information Imbalance in Data Representations

• Information imbalance is a set of quantitative measures that characterize asymmetric information content between representations, modalities, or groups.
• It leverages nearest-neighbour rank preservation to guide feature selection, causal inference, and representation analysis without estimating probability densities.
• The concept is applied across diverse domains including multimodal learning, graph analysis, federated systems, and public infrastructure to improve predictive fairness and performance.

Searching arXiv for papers on "information imbalance" and closely related formulations across feature selection, causality, multimodal learning, and representation analysis. Information imbalance is a family of quantitative concepts used to characterize asymmetric information content or uneven information allocation between representations, modalities, graph regions, social groups, clients, or ranked result sets. In one major line of work, it is a nearest-neighbour-rank statistic $\Delta(A\to B)$ that asks whether points that are close in one space remain close in another; in other lines, it denotes imbalance in modality contributions, topology-dependent supervision flow, subgroup-specific knowledge of an opaque rule, or coverage of an information spectrum. Across these settings, the measure is used both diagnostically and operationally: to select features, infer causality, analyze learned representations, redesign training objectives, and guide deployment or interface policies (Wild et al., 2024, Xu et al., 15 Feb 2025, Sun et al., 2022).

1. Rank-based and differentiable formulations

A central formalization defines information imbalance between two feature spaces $A$ and $B$ on the same $N$ points through neighbour-rank preservation. If $r_{ij}^A$ is the rank of point $j$ in the ordered neighbour list of $i$ in space $A$, and $r_{ij}^B$ is the corresponding rank in space $B$, then the asymmetric imbalance is

$A$0

In this convention, $A$1 means that knowledge of $A$2 almost perfectly predicts neighbourhoods in $A$3, whereas $A$4 means that $A$5 and $A$6 are effectively independent at the nearest-neighbour level. Because $A$7 in general, the formalism separates redundancy, independence, and directional informativeness. The same construction is also written as $A$8 in the clinical feature-selection literature (Sharma et al., 2024, Wild et al., 2024).

The causal-discovery literature extends the definition from the single nearest neighbour to the $A$9 nearest neighbours,

$B$0

thereby trading variance against locality. That work emphasizes that the method is model-free and density-free: no probability densities are estimated, and no explicit dynamical model is fitted (Tatto et al., 2023).

A differentiable variant, Differentiable Information Imbalance (DII), replaces the hard nearest-neighbour indicator with soft neighbourhood weights. With a weighted predictor distance $B$1 and kernel scale $B$2,

$B$3

and the objective becomes

$B$4

In the $B$5 limit, the soft assignment recovers the original rank-based imbalance. This differentiable form supports gradient-based feature weighting, adaptive scaling across heterogeneous units, and sparse selection through $B$6 regularization or elimination heuristics (Wild et al., 2024, Salvagnin et al., 21 Aug 2025).

The VAE analysis literature adds a geometric interpretation: $B$7 is related to the conditional copula entropy $B$8, but avoids direct entropy estimation by relying on rank preservation. This suggests that information imbalance functions as a non-parametric proxy for directional information flow between internal representations (Camboulin et al., 2024).

2. Feature selection and scientific inference

In glassy dynamics, information imbalance is used to rank structural descriptors by how informative they are about future dynamical propensity. The target variable is the propensity $B$9 at time $N$0, and each structural descriptor $N$1 is evaluated through $N$2. The procedure is purely rank-based, non-parametric, and uses only nearest-neighbour structure; for $N$3 particles at $N$4, with $N$5 Monte-Carlo trajectories to estimate propensities, the authors report that a forward greedy search saturates by $N$6 features in the supervised setting and by $N$7 in the unsupervised setting. In the downstream linear predictor, supervised information-imbalance selection and Lasso reach $N$8 already at $N$9 and saturate near $r_{ij}^A$0 by $r_{ij}^A$1, whereas random feature selection lags until $r_{ij}^A$2–$r_{ij}^A$3. The paper also reports that no single descriptor outperforms the dynamic propensity itself, while combinations of descriptors do (Sharma et al., 2024).

In clinical prediction, the same idea is adapted to mixed-type data with extensive missingness. The COVID-19 study considers $r_{ij}^A$4 patients, approximately $r_{ij}^A$5 admission features, approximately $r_{ij}^A$6 missing inputs, and 14 binary outcome features organized into a severity tree with 8 leaves. To handle class imbalance, it defines a weighted imbalance $r_{ij}^A$7 with per-patient weights $r_{ij}^A$8 and a normalizing constant chosen so that random nearest-neighbour assignments yield $r_{ij}^A$9. To handle missing data, it avoids imputation and instead restricts each candidate feature tuple $j$0 to the subset $j$1 of patients with no missing values on that tuple, discarding tuples if $j$2 with $j$3 or if the class distribution on $j$4 differs too much from the full data, operationalized by Jensen–Shannon divergence $j$5. The search is exhaustive for small tuple sizes and beam-based thereafter, with beam width $j$6; the optimal number of features is reported to be between 10 and 15, and the selected 13-tuple has mean pairwise Pearson-$j$7 and mean pairwise $j$8, indicating low redundancy. The reported leave-one-out 10-NN severity prediction yields approximately $j$9 exact severity-class accuracy and approximately $i$0 accuracy on the correct side of the severity tree (Wild et al., 2024).

In molecular systems, DII is used to align units, assign relative importance, and determine reduced dimensionality. For the CLN025 $i$1-hairpin peptide, 1,429 frames with 4,278 pairwise heavy-atom distances serve as ground truth, and an exhaustive search over all $i$2 nonempty subsets of 10 candidate collective variables identifies the best 3-plet as $i$3 with learned weights $i$4. For machine-learning force fields, DII selects among 176 ACSF symmetry functions using 546-dimensional SOAP descriptors as ground truth; with 20–50 ACSFs chosen by DII, the reported force-RMSE is $i$5 meV, on par with the full 176-ACSF representation, while evaluation time is reduced by approximately $i$6 (Wild et al., 2024).

These applications collectively show that the rank-based formulation is not limited to a single scientific domain. A plausible implication is that its main methodological attraction is the same across fields: it compares representations directly through neighbourhood structure, without density estimation, explicit generative modeling, or a requirement that the variables be linearly related.

3. Causality and temporal predictability

In high-dimensional dynamical systems, information imbalance is used in a variational causality test based on predictive improvement. The basic comparison is between the self-prediction imbalance $i$7 and the joint-past imbalance

$i$8

where $i$9 scales the putative driver $A$0. If $A$1 is non-causal for $A$2, adding $A$3 does not improve prediction and typically worsens it; if $A$4 is present, there exists $A$5 such that $A$6. The resulting imbalance gain is

$A$7

Benchmark tests on coupled chaotic systems report false-positive rates of $A$8 for identical 3D Rössler flows, $A$9 for different Rössler flows, and $r_{ij}^B$0 for 40-dimensional Lorenz–96 when all coordinates are used, in contrast to approximately $r_{ij}^B$1 false-positive rates for several alternative model-free methods in the same absent-direction tests. The same framework is applied to EEG, where it detects post-offset stimulus-duration effects and a directed POz $r_{ij}^B$2 Fz signal with a peak around 90 ms after onset and a second peak around 160 ms after offset (Tatto et al., 2023).

The DII-based financial literature reformulates the same idea as feature-weight optimization and exclusion testing. With a vector of positive weights $r_{ij}^B$3, DII is minimized on the full predictor set and on the predictor set with variable $r_{ij}^B$4 removed, producing the Imbalance Gain

$r_{ij}^B$5

If $r_{ij}^B$6 for some forecast horizon $r_{ij}^B$7, the variable is deemed non-linearly causal for the target. On synthetic data, DII/IG detects both a linear driver and a quadratic driver in a process where VAR(1)+Granger misses the nonlinear one, and it avoids a false positive in a common-driver scenario where linear Granger spuriously flags the proxy variable. On European Union Allowances returns from January 2013 to April 2024 with $r_{ij}^B$8, both VAR(1)+Granger and DII+IG identify coal futures prices and the IBEX35 index as top drivers; DII additionally flags non-linear relationships involving copper futures and certain uncertainty indices, while some variables with non-zero Granger $r_{ij}^B$9 yield near-zero IG (Salvagnin et al., 21 Aug 2025).

The shared theme is directional predictability without explicit density estimation. This suggests that information imbalance is especially suited to settings in which the predictor–target relationship may be nonlinear, high-dimensional, or only partially observed.

4. Learned representations, intrinsic dimension, and contrastive alignment

In variational autoencoders, information imbalance is used to analyze hidden-layer geometry jointly with Intrinsic Dimension (ID). The measure compares each layer to a reference layer such as the input and reveals whether successive transformations preserve or distort neighbourhood structure. On CIFAR-10, whose input ID is reported as approximately 30, the paper identifies a transition when the bottleneck size $B$0 exceeds the data ID. For $B$1 below the ID, encoder profiles show monotonic compression and the decoder shows no information expansion; once $B$2 exceeds the ID, $B$3 increases with $B$4, indicating that the bottleneck becomes more abstract, while the decoder exhibits a dip in II, i.e. an information expansion relative to the input. Training dynamics also separate into a rapid fit phase from epochs 0 to approximately 10, during which $B$5 rapidly decreases, and a slower generalization phase from epochs 10 to 200, during which it slowly increases again while the KL term decreases. The characteristic double-hunchback ID profile emerges only after the first 10 epochs (Camboulin et al., 2024).

In contrastive vision-LLMs, information imbalance is not defined by a closed-form information-theoretic functional but by a mismatch in shared latent factors between images and captions. The analysis argues that images contain many latent factors while captions mention only a sparse subset, and that this imbalance drives both the modality gap and object bias in CLIP-style models. The modality gap is quantified through the mean-separation metric

$B$6

and through Relative Modality Gap (RMG), while object bias is quantified through the Matching Object–Attribute Distance (MOAD). The synthetic MAD dataset varies information imbalance by always including the digit class in the caption but randomly including only $B$7 of five attributes. As $B$8 increases from 1 to 5, both L2M and RMG decline, image and text MOAD shrink toward zero, and zero-shot attribute accuracy rises from near-chance, below $B$9, to above $A$00. The paper also reports that only a few embedding dimensions drive the gap, and interprets the phenomenon through the alignment–uniformity decomposition of InfoNCE: when alignment is limited by sparse captions, the model reduces loss by increasing uniformity, which separates the modalities in a small number of dimensions (Schrodi et al., 2024).

Taken together, these works treat information imbalance as a geometric property of learned representations. In the VAE case it tracks abstraction and recoverability across layers; in the contrastive vision-language case it explains why joint spaces may separate modalities or over-emphasize factors that are more consistently verbalized.

5. Multimodal contribution imbalance and the search for relative balance

In multimodal classification, information imbalance is often formalized as uneven modality contribution to the fused decision. BalanceBenchmark defines a vanilla late-fusion model by

$A$01

and measures the contribution of modality $A$02 through a Shapley value

$A$03

with $A$04 the accuracy obtained when only modalities in $A$05 are unmasked. The bimodal imbalance degree is then $A$06, and the trimodal version averages all pairwise absolute differences. Lower $A$07 means more balanced modality contributions, with $A$08 and $A$09 only when all Shapley contributions are equal (Xu et al., 15 Feb 2025).

The benchmark groups 17 mitigation methods into four categories—data-level adjustments, feed-forward-level modifications, objective-level redesign, and optimization-level modulation—and evaluates them on seven datasets spanning audio–video, RGB–optical-flow, image–text, and tri-modal settings. Its average training FLOPs by category are reported as follows (Xu et al., 15 Feb 2025):

Category	Avg. FLOPs $A$10
Objective	8.38
Optimization	16.90
Forward	6.94
Data	11.40

The experimental findings establish a three-way trade-off among performance, balance degree, and computational complexity. Objective-based methods perform best on highly imbalanced datasets such as CREMA-D; optimization-based methods excel on low-imbalance datasets such as BalancedAV but incur the highest FLOPs, approximately $A$11; feed-forward methods have the lowest overhead, approximately $A$12 FLOPs, while still delivering gains under moderate imbalance. The paper further argues that accuracy is maximized at a relative-balance point rather than at absolute balance: as $A$13 decreases from a high gap, accuracy improves, but pushing $A$14 all the way toward zero can hurt performance because naturally stronger modalities carry more information (Xu et al., 15 Feb 2025).

A later decision-layer analysis sharpens this point. In a multimodal classifier with final linear layer $A$15, partitioned into modality-specific blocks $A$16, the contribution size of modality $A$17 is measured by the average $A$18 norm

$A$19

Experiments on CREMAD and Kinetic-Sounds show that audio retains systematically larger decision-layer weight norms and larger logit ranges than video, even when encoders are pretrained separately and only the fusion layer is fine-tuned. On CREMAD, unimodal means are reported as $A$20 and $A$21, with logit ranges $A$22 and $A$23; on Kinetic-Sounds, the corresponding means are $A$24 versus $A$25 and $A$26 versus $A$27. The paper reports that the audio decision-weight line remains above the video line even for classes where video unimodal accuracy is higher, and attributes the bias to intrinsic disparities in feature-space and decision-weight distributions rather than optimization dynamics alone. It argues for adaptive decision-layer weight allocation aimed at relative balance according to modality capability (Ma et al., 16 Oct 2025).

6. Graph, strategic, and federated forms of imbalance

In graph learning, topology can itself create information imbalance. One formulation, topology-imbalance, is defined as the uneven topology positions of labeled nodes and is analyzed through two graph-wide coefficients: the Reaching Coefficient (RC), which measures how close unlabeled nodes are to same-class labeled nodes, and the Squashing Coefficient (SC), which averages Ollivier–Ricci curvature along shortest paths from reachable same-class labels. Larger RC means under-reaching is alleviated; larger SC means less over-squashing. The PASTEL framework addresses this by learning a position-aware structure through anchor-based position encodings, a cosine-similarity affinity matrix, and a class-wise conflict measure based on Group-PageRank label influence. On Cora with only 20 labels per class, the reported effect is an increase of RC from 0.40 to 0.55 and SC from $A$28 to $A$29, with a $A$30 point gain in Weighted-F$A$31 over a vanilla GCN; on synthetic stochastic block models, the advantage grows from $A$32 points to over $A$33 points in Weighted-F$A$34 as the inter-community probability $A$35 shrinks (Sun et al., 2022).

A second GNN formulation treats information imbalance as class imbalance transformed into neighbourhood-level heterophily. The label difference index (LDI) is defined from the difference between a node’s one-hot true label vector and the empirical label-proportion vector in its 1-hop neighbourhood. Minority-class nodes have larger average LDI, nodes with higher LDI are more likely to be misclassified, and oversmoothing tends to newly misclassify high-LDI nodes. The paper introduces an improved focal loss and four LDI-aware methods—graph re-sampling, graph re-weighting, graph metric learning, and the graph bilateral-branch network. On transductive Cora with a GCN base, GBBN with label-based weighting improves accuracy/G-mean from 0.339/0.449 for baseline GCN to 0.586/0.710; on inductive Cora with an SGC base, GBBN with combined label+LDI weighting raises Macro-F1 from 0.329 to 0.662 (Wang et al., 2021).

In strategic learning under non-transparency, information imbalance arises because different groups infer different proxies for an opaque decision rule. With population groups supported on subspaces $A$36, group $A$37 learns $A$38, the projection of the principal’s scoring vector onto its observable subspace, and therefore responds strategically according to group-specific information. The resulting disparity is summarized by the information overlap proxy

$A$39

The theory shows that the welfare-maximizing rule can generate negative externalities, so that the true quality of some groups deteriorates, and also characterizes conditions under which all groups improve. When cost matrices are proportional, both groups benefit; when subspaces are orthogonal, no harm also follows. A hybrid rule $A$40 preserves do-no-harm under decomposable costs (Bechavod et al., 2021).

In federated learning, imbalance is analyzed at three levels: inter-case, inter-class, and inter-client. FedBB introduces Positive Negative Balanced (PNB) loss to reweight positives versus negatives within each class and classes against one another through smoothed inverse-frequency terms, and Client Balanced Reweighting (CBR) to assign larger aggregation weights to clients with more balanced local data. On NIH CXR14 with 5 clients, FedBB reaches AUC $A$41 versus FedAvg $A$42 at Dirichlet skew $A$43, and $A$44 versus $A$45 at $A$46; on CIFAR-10 at $A$47, it reaches $A$48 versus $A$49. Ablations show that PNB and CBR both contribute, while overhead remains roughly equal to FedAvg (Chung et al., 8 Jun 2026).

These graph, strategic, and federated formulations share a common pattern: imbalance is not merely a class-count issue but a distortion of the routes through which information becomes available, whether through topology, opacity, or client heterogeneity.

7. Information completeness, infrastructure, and public information systems

A broader public-information interpretation replaces the representation-to-representation question with a coverage question. In internet search, a query $A$50 and each result $A$51 are embedded in a common vector space, a domain-weighted corpus vector is formed as $A$52, and the completeness of result $A$53 is defined by

$A$54

After viewing the top $A$55 results, cumulative completeness is

$A$56

which defines a completeness curve from 0 at $A$57 to 1 at $A$58; its area under the curve quantifies how complete the viewed subset is on average. The empirical validation uses 6.5 trillion raw search-result texts from daily trending Google queries across 48 countries over one year, with 57.6 million searches per day, median 320 results per query, and approximately 18 billion text snippets per day. A randomized experiment with 876 U.S. adults shows that exposing a live completeness score reduces Fact Resistance by $A$59 standard deviations, increases the farthest clicked rank by $A$60 positions, and increases the aggregate completeness of clicked results by $A$61 percentage points, while the overall AOT17 effect is not statistically significant (Khanna, 12 Oct 2025).

In telecommunication infrastructure, imbalance is formalized as a local service shortfall between population and deployed base-station capacity. For geographic granule $A$62, with population $A$63, $A$64 base stations, and capacities $A$65, the imbalance index is

$A$66

The average imbalance $A$67 over a region is area-weighted, or simply the mean on a uniform grid. The design is bounded in $A$68, smooth, differentiable, tunable through $A$69 and $A$70, and depends only on local quantities. The paper then minimizes $A$71 under a base-station budget, obtaining a quasi-convex relaxation and, for $A$72, a water-filling-style solution followed by integer rounding and fill-in. Against the GSMA Mobile Connectivity Index on 90 countries, the country-average $A$73 achieves Pearson’s $A$74 with $A$75, while simulation shows that the proposed deployment strategy outperforms a naive population-proportional baseline whenever resources are not overly scarce (Zhang et al., 2021).

Across these public-system formulations, information imbalance becomes a question of what fraction of a relevant spectrum is actually accessible: search users may see only the “tip of a pre-ranked information iceberg,” and rural granules may receive only a fraction of the service capacity implied by local population. This suggests a broader usage in which imbalance is not restricted to machine-learning representations but also measures how informational or infrastructural resources are distributed across a population.

Information imbalance therefore has no single universal definition. It names a cluster of related quantitative ideas: nearest-neighbour asymmetry between spaces, modality-contribution skew, topology-induced supervision distortion, subgroup-specific opacity, client heterogeneity, and incomplete coverage of an information spectrum. What unifies these uses is methodological rather than ontological. In each case, imbalance is operationalized by comparing what is available in one channel, group, or representation to what is needed to reconstruct, predict, or fairly exploit another.