Redundancy Reduction: Methods & Applications

• Redundancy reduction term is a mathematical mechanism that minimizes statistical overlap among feature vectors, neural activations, or association rules to boost uniqueness.
• It is applied in diverse areas such as self-supervised representation learning, network pruning, and token compression to prevent collapse and improve efficiency.
• Empirical implementations, including Barlow Twins and graph-based pruning, show significant gains in speed, model compactness, and accuracy.

Redundancy reduction term refers to an explicit or implicit mathematical mechanism for minimizing statistical or informational overlap within a structured set of objects—such as feature vectors, neural activations, codewords, association rules, or textual elements. The exact instantiation varies by context: in self-supervised representation learning, redundancy reduction penalizes correlations among embedding dimensions; in compression, it quantifies bit savings achieved by grouping similar blocks; in pruning, it identifies structurally overcomplete layers; in rule mining, it eliminates logically subsumed rules. The overarching aim is to increase efficiency, uniqueness, and/or informativeness by discouraging duplication, dependence, or overlap in the entities under consideration.

1. Redundancy Reduction in Representation Learning

The Barlow Twins framework (Zbontar et al., 2021) introduced a concrete redundancy-reduction term for self-supervised learning. The approach involves constructing, for each batch of size $N$, a $D\times D$ cross-correlation matrix between normalized embedding vectors from two stochastic "views" (data augmentations) of the same samples: $$C_{ij} = \frac{ \sum_{b=1}^N z^a_{b,i} z^b_{b,j} }{ \sqrt{ \sum_b (z^a_{b,i})^2 } \sqrt{ \sum_b (z^b_{b,j})^2 } }$$ The complete Barlow Twins loss is

$$\mathcal{L}_{BT} = \sum_{i=1}^D (1 - C_{ii})^2 + \lambda \sum_{i=1}^D \sum_{j\neq i} C_{ij}^2$$

where the first (invariance) term aligns same-dimension features across views, and the second term is the redundancy-reduction penalty: the sum of squared off-diagonal entries of $C$, penalizing linear dependencies across different embedding dimensions. At the optimum, $C \approx I_D$: features are decorrelated and unique.

The same principle underlies subsequent works such as RRTN for emotion regression (Jing et al., 2022) and RedMotion in self-supervised motion prediction (Wagner et al., 2023), typically varying only in normalization details. In all such frameworks, the redundancy-reduction term is key to avoiding representational collapse and encouraging statistically independent codes.

2. Redundancy Reduction in Model Pruning and Structural Compression

Redundancy reduction as a principle has also informed architectural pruning strategies. In network pruning, redundancy metrics select overparameterized or structurally similar sets of parameters (e.g., filters) for removal, with the goal of maintaining representational diversity post-pruning.

Two canonical criteria arise:

• Filter count: Directly quantifying redundancy as the number of filters in a given layer, and pruning from the layer with the highest count. Li et al. demonstrate that randomly removing filters from the most redundant layer is as or more effective than global saliency-based pruning (Li et al., 2019).
• Graph-based redundancy: Wang et al. (Wang et al., 2021) define, for each convolutional layer $i$, a graph $G_i$ connecting sufficiently similar filters and measure redundancy as

$R_i = \frac{N_i}{w_1 k_i + w_2 N_1^c(X_i)}$

where $N_i$ is filter count, $k_i$ is number of graph components, and $N_1^c(X_i)$ is the 1-covering number (the minimal set of filters covering all others within a graph-radius of 1). Layers with maximal $R_i$ are pruned preferentially, yielding superior compression/accuracy trade-offs by targeting structural redundancy per se.

Regularization-based approaches enforce redundancy reduction via direct penalties on parameter correlation matrices (e.g., filter orthogonality) (Kahatapitiya et al., 2019), with explicit $\ell_1$ or $\ell_2$ norm penalties on off-diagonal entries, favoring mutually-orthogonal convolutional bases.

3. Redundancy Reduction in Token and Data Compression

Compression frameworks exploit redundancy-reduction both implicitly (through grouping of similar components) and explicitly (through redundancy-aware merging).

In image coding, quadtree-based schemes (Faundez-Zanuy et al., 2022) recursively merge neighboring blocks if their attributes (e.g., mean, variance, binary map) are sufficiently similar:

• Blocks are merged if all differences (mean, variance, bit-plane) fall below set thresholds.
• Rate savings per merge quantify local redundancy reduction:

$$\Delta R = 4 \cdot R_{\text{desc}} - (R_{\text{desc}} + R_{\text{tree}})$$

where $R_{\text{desc}}$ is the descriptor bit cost and $R_{\text{tree}}$ the tree overhead. Aggressive merging in homogeneous regions adapts block size to local signal complexity.

In token-based vision/LLMs, strategies such as PyramidDrop (Xing et al., 2024) and FPET (Kim et al., 26 Mar 2025) eliminate superfluous tokens in transformer layers through data-dependent similarity/scoring mechanisms, compressing internal representations adaptively. These mechanisms do not always correspond to an explicit loss penalty; instead, pruning or merging operators are introduced structurally, directly reducing computational footprint by eliminating redundant data pathways.

4. Redundancy Reduction in Association Rule Mining

Redundancy among association rules arises when some rules can be logically entailed by others, rendering them superfluous for data mining or inference purposes. Balcázar (Balcazar, 2010) formalizes multiple variants of redundancy:

• Standard redundancy: $X_0 \rightarrow Y_0$ is redundant w.r.t.\ $X_1 \rightarrow Y_1$ if for all datasets, $$s_D(X_0 \rightarrow Y_0) \geq s_D(X_1 \rightarrow Y_1)$$ and $$\operatorname{conf}_D(X_0 \rightarrow Y_0) \geq \operatorname{conf}_D(X_1 \rightarrow Y_1)$$.
• Plain redundancy: Only the confidence requirement is enforced.

Deductive calculi are presented—specific inference schemes are sound and complete for each redundancy notion. To minimize redundancy, compact bases of representative rules are constructed, such that every high-confidence rule is redundant with respect to some rule in the basis, and the basis is of provably minimal size. The systematic elimination of redundant rules reduces the size and complexity of association rule outputs without sacrificing completeness.

5. Redundancy Reduction in Universal Coding and Rate Penalty

In information-theoretic universal coding, the redundancy reduction term captures the decrease in worst-case excess code length (regret or redundancy) due to risk-averse coding objectives.

Haussler's and Campbell's settings consider minimax redundancy under Rényi divergence $D_\alpha$ (Yagli et al., 2017). The asymptotic minimax Rényi redundancy is

$$R_\lambda(n) = \frac{k-1}{2} \log \frac{n}{2\pi (1+\lambda)^{1/\lambda}} + \log \left( \frac{\Gamma(1/2)^k}{\Gamma(k/2)} \right) + o(1)$$

The difference $-\frac{k-1}{2\lambda} \log(1+\lambda)$ is precisely the redundancy reduction (relative to the ordinary minimax redundancy at $\lambda=0$), quantifying the rate gain from increased risk aversion. The final $o(1)$ term is a vanishing redundancy-reduction term, decaying with block length.

6. Empirical and Practical Implications

Redundancy-reduction terms are consistently associated with:

• Improvements in efficiency, memory usage, and computational cost (e.g., 40%–55% acceleration in PyramidDrop (Xing et al., 2024), state-of-the-art FLOP-accuracy tradeoffs in pruning (Wang et al., 2021)).
• In representation learning, prevention of trivial collapse and enforced dimension specialization (Zbontar et al., 2021, Jing et al., 2022, Wagner et al., 2023).
• More compact, interpretable, and complete rule or code bases in data mining and coding (Balcazar, 2010, Yagli et al., 2017).
• Performance improvements in parameter efficiency and downstream task accuracy—e.g., LinearConv’s 50% parameter reduction with ≤2% test accuracy loss (Kahatapitiya et al., 2019), and FPET’s 19–23% inference speedup and 24–48% memory reduction at negligible accuracy cost (Kim et al., 26 Mar 2025).

Careful hyperparameterization (e.g., selection of redundancy-reduction weight $\lambda$) is required to balance between collapse avoidance, information retention, and efficiency. In empirical analyses, the effect of redundancy-reduction is generally robust to moderate hyperparameter variation.

7. Summary Table: Representative Redundancy Reduction Terms

Domain	Formulation	Source [arXiv]
SSL Embeddings	$$\mathcal{L}_{BT} = \sum_{i}(1-C_{ii})^2 + \lambda\sum_{i\ne j} C_{ij}^2$$	(Zbontar et al., 2021)
CNN Pruning	$R_i = N_i/(w_1 k_i + w_2 N_1^c)$	(Wang et al., 2021)
Token Merging (ViT/PET)	Adapter-based bipartite matching and merging, fully differentiable, no explicit penalty	(Kim et al., 26 Mar 2025)
Document Summarization	$L = \beta L_{ce} + (1-\beta) L_{rd}$, with $L_{rd}$ penalizing sentence similarity	(Xiao et al., 2020)
Association Rules	Rule bases and deductive calculi minimizing logical redundancy	(Balcazar, 2010)
Universal Coding	Redundancy-reduction as negative shift in minimax redundancy, $-\frac{k-1}{2\lambda} \log(1+\lambda)$	(Yagli et al., 2017)

The redundancy reduction term, across its diverse instantiations, operationalizes the principle of information efficiency: promoting orthogonality, diversity, or logical minimality among code elements, features, or rules, thus enhancing compactness, interpretability, and computational tractability.