Budgeted Broadcast: An Activity-Dependent Pruning Rule for Neural Network Efficiency

Abstract: Most pruning methods remove parameters ranked by impact on loss (e.g., magnitude or gradient). We propose Budgeted Broadcast (BB), which gives each unit a local traffic budget (the product of its long-term on-rate $a_i$ and fan-out $k_i$). A constrained-entropy analysis shows that maximizing coding entropy under a global traffic budget yields a selectivity-audience balance, $\log\frac{1-a_i}{a_i}=\beta k_i$. BB enforces this balance with simple local actuators that prune either fan-in (to lower activity) or fan-out (to reduce broadcast). In practice, BB increases coding entropy and decorrelation and improves accuracy at matched sparsity across Transformers for ASR, ResNets for face identification, and 3D U-Nets for synapse prediction, sometimes exceeding dense baselines. On electron microscopy images, it attains state-of-the-art F1 and PR-AUC under our evaluation protocol. BB is easy to integrate and suggests a path toward learning more diverse and efficient representations.

• The paper introduces a novel pruning method that enforces a local traffic budget, linking neuron activity and fan-out for efficient network operation.
• It employs local statistics and entropy maximization via SP-in and SP-out mechanisms to protect rare-feature detectors and optimize connectivity.
• Empirical results across ASR, face recognition, change detection, and synapse prediction demonstrate improved performance over conventional pruning methods.

Budgeted Broadcast: An Activity-Dependent Pruning Rule for Neural Network Efficiency

Introduction and Motivation

The Budgeted Broadcast (BB) framework introduces a biologically inspired, activity-dependent pruning rule for neural networks, formalizing the notion of a neuron's metabolic cost as its "traffic"—the product of its long-term activation rate ($a_i$) and fan-out ($k_i$). Unlike conventional pruning methods that focus on weight magnitude or gradient-based importance, BB enforces a local traffic budget, $t_i = a_i k_i$, and prunes connections when this budget is exceeded. This approach is motivated by principles observed in biological neural circuits, particularly the selectivity–audience balance, which posits a linear relationship between a unit's inactivity log-odds and its fan-out: $\log\frac{1-a_i}{a_i} \approx \beta k_i$. The BB rule is designed to protect rare-feature detectors and promote efficient, decorrelated representations, addressing the redundancy and lack of robustness in standard deep networks.

Methodology: Local Traffic Control and Entropy Maximization

BB operates via two complementary actuators: SP-in (fan-in pruning) and SP-out (fan-out pruning). Each unit tracks its activity using an EMA and periodically updates its binary mask to enforce the traffic budget. The target degree for each unit is computed as $$k = d_0 + \beta^{-1}\log\frac{1-\tilde a}{\tilde a}$$, clipped to a valid range. Masks are refreshed at fixed intervals, and regrowth is enabled by reselecting the top-$k$ connections by magnitude. This mechanism is applied to FFN blocks and convolutional layers, with minimal overhead due to channel-wise statistics and binary masks.

The theoretical underpinning is a constrained-entropy maximization, where the network seeks to maximize coding entropy $H(h)$ subject to a global traffic constraint. The Lagrangian yields the selectivity–audience balance as the stationary condition. Information-theoretic analysis further shows that total traffic serves as an upper bound on mutual information between layers, justifying the BB rule as a principled descent step on a composite objective.

Figure 1: SP-out (Axonal pruning) enforces per-unit traffic budgets by masking outgoing connections, with high-activity units shedding more edges and low-activity units retaining more.

Didactic Validation: Mechanism, Safety, and Optimization

Controlled experiments on MLPs demonstrate the emergence of the selectivity–audience balance, with BB-trained networks exhibiting a robust linear relationship between fan-out and inactivity log-odds, absent in standard SGD-trained networks. BB protects rare features by maintaining their traffic below the pruning threshold, while actively curbing over-active units. In DNF tasks designed to induce optimization barriers, BB consistently escapes lazy-learning traps by structurally pruning ambiguous units, enabling specialization and efficient convergence scaling as $O(W \log W)$.

Empirical Results Across Domains

Automatic Speech Recognition (ASR)

BB was evaluated on LibriSpeech with a Transformer architecture. Under matched sparsity budgets, BB (SP-in) consistently outperformed magnitude and top-$k$ pruning in overall WER reduction, with pronounced gains on long-tail (rare) word buckets.

Figure 2: ASR on LibriSpeech—BB achieves superior WER reduction, especially for rare words, compared to magnitude and top-$k$ pruning.

Face Identification

On VGGFace2-7k with ResNet-101, BB (SP-in) was applied to $1\times1$ bottleneck projections. Across a range of sparsity levels, BB formed or matched the upper envelope of Pareto fronts for both Top-1 classification and verification accuracy, often exceeding dense baselines at moderate sparsity.

Figure 3: Pareto fronts on VGGFace2-7k—BB matches or exceeds dense accuracy across budgets, using fewer active parameters.

Change Detection

For bi-temporal change detection on LEVIR-CD, BB (SP-in) improved mean IoU and F1-score over dense models, with enhanced recovery of true positives in ground truth regions, particularly for small and scattered changes.

Figure 4: Change detection on LEVIR-CD—BB recovers more true positives, especially for subtle changes, compared to dense models.

Synapse Prediction (EM)

BB (SP-in) was integrated into a 3D U-Net for synapse segmentation on volumetric EM data. It achieved state-of-the-art PR-AUC and Best F1 under the evaluation protocol, outperforming both dense and magnitude-pruned baselines.

Figure 5: Synapse prediction overlays—BB (green) detects more true synapses than dense (red) and magnitude (blue), with consensus marked in yellow.

Theoretical and Practical Implications

BB establishes a new axis for structural plasticity, shifting the focus from utility-based pruning to metabolic cost-based allocation. The selectivity–audience balance provides a predictive equilibrium linking network structure to function, with implications for efficient information broadcast and homeostatic resilience. BB's local, label-free statistics and periodic mask updates introduce modest overhead, but its scalability and empirical performance make it suitable for large-scale and foundation models, particularly for protecting rare or long-tail knowledge.

Future Directions

Potential extensions include applying BB to lateral connections and attention mechanisms, where per-token traffic budgets could further enhance efficiency and diversity. Mapping BB-induced unstructured sparsity to hardware-friendly patterns remains an open engineering challenge. The framework also invites exploration of dynamic budget allocation and integration with other forms of structural homeostasis.

Conclusion

Budgeted Broadcast offers a principled, biologically motivated approach to neural network pruning, enforcing local traffic budgets that yield efficient, decorrelated, and robust representations. Empirical results across diverse domains validate its effectiveness, particularly in protecting rare features and improving performance at matched sparsity. The selectivity–audience balance emerges as a unifying principle for network organization, with broad implications for future research in efficient and resilient AI systems.