Hierarchical Reinforcement Learning for Neural Network Compression (HiReLC): Pruning and Quantization

Abstract: We present HiReLC, a hierarchical ensemble-reinforcement learning framework for automated joint quantization and structured pruning of deep neural networks. The framework decomposes the compression search across two levels of abstraction: low-level agents (LLAs) operate independently per block, selecting per-kernel configurations over a multi-discrete action space spanning bitwidth, pruning keep-ratio, quantization type, and granularity, while high-level agents (HLAs) coordinate global budget allocation via ensemble voting guided by Fisher Information-based sensitivity estimates. To mitigate the computational cost of policy evaluation, an iterative active learning loop interleaves surrogate-guided RL optimization with post-compression fine-tuning, using a lightweight MLP surrogate to amortize expensive evaluations and a logit-MSE proxy during cold-start. The surrogate is used for reward shaping rather than as a replacement for final post-compression evaluation. The controller is architecture-agnostic by design, with a modular layer abstraction decoupling the RL environment from the underlying network topology. Experiments across Vision Transformer and CNN benchmarks demonstrate effective parameter-storage compression ratios of 5.99 - 6.72$\times$ with a 3.83 % gain in one setting and 0.55 - 5.62 % accuracy drops elsewhere, supporting hierarchical policy decomposition and sensitivity-aware guidance as practical design choices for joint neural network compression.

• The paper demonstrates a modular hierarchical RL framework that jointly optimizes structured pruning and mixed-precision quantization using high-level and low-level agents.
• It integrates sensitivity awareness via Fisher Information scores, guiding budget allocation and reward shaping to preserve critical network regions.
• Empirical evaluations on ViTs and CNNs show notable compression ratios (up to 6.72x) with modest accuracy drops, validating the framework's efficiency and robustness.

Hierarchical Reinforcement Learning for Joint Neural Network Compression: An Expert Analysis

Framework Design and Motivation

The paper "Hierarchical Reinforcement Learning for Neural Network Compression (HiReLC): Pruning and Quantization" (2606.26002) introduces HiReLC, a modular hierarchical ensemble-RL system for simultaneously applying structured pruning and mixed-precision quantization to deep networks. The motivating challenge is efficient model deployment on resource-constrained platforms, especially as architectures such as Vision Transformers (ViTs) and CNNs scale in complexity. Traditional heuristics or static assignment approaches are insufficient, lacking sensitivity-awareness and adaptability.

HiReLC decomposes the compression search across two abstraction levels:

• High-Level Agents (HLAs) allocate global per-block compression budgets, guided by Fisher Information–based sensitivity scores.
• Low-Level Agents (LLAs) operate independently per block, searching a multi-discrete action space: bitwidth, pruning keep-ratio, quantization type, and granularity.

The pipeline employs an active learning loop alternating RL-based candidate search, surrogate-guided reward shaping, fine-tuning, and surrogate model updates. This controller is architecture-agnostic, with a layer parsing abstraction decoupling RL from network topology.

Figure 1: Overview of the HiReLC compression pipeline, highlighting hierarchical budget allocation and iterative active learning.

Compression Modeling and RL Formulation

HiReLC formalizes compression as a constrained multi-objective optimization over joint kernel configurations. Each action selects:

• Bitwidth ($b_{i,k}$, within configurable bounds)
• Pruning keep-ratio ($\rho_{i,k}$, fraction of output channels retained)
• Quantization type (INT, FLOAT)
• Granularity (uniform, log, per-channel, learned scheme)

The objective maximizes a weighted sum of accuracy retention, parameter-storage compression, and stability regularization, subject to explicit accuracy drop and compression constraints. The hierarchical decomposition into HLAs and LLAs is motivated both by the high-dimensional action space and cross-layer dependency structure.

Figure 2: Reward-flow schematic outlining compression primitive evaluation and reward mapping for RL updates.

The RL environments operate as MDPs, with state vectors including block indices, sensitivity scores, compression metrics, budget deviation, estimated accuracy drop, and agent preference codes. Actions are decoded into kernel-level configuration primitives. The HLA ensemble (PPO/A2C, heterogeneous reward weights) issues per-block LayerBudgets; LLA ensembles (PPO/A2C, diverse reward weights) conduct per-kernel search within those budgets; ensemble votes stabilize final assignments.

Figure 3: Two-tier hierarchical controller architecture: HLA ensemble allocates budgets, LLA ensembles explore per-kernel configurations.

Sensitivity-Aware Guidance and Surrogate Modeling

A key design is the integration of Fisher Information–based sensitivity estimation. Each block's sensitivity score informs both reward shaping and deterministic action correction, ensuring critical regions are compressed conservatively. The sensitivity penalty in the LLA reward is proved bounded and monotone, strongly discouraging aggressive compression of sensitive blocks.

HiReLC amortizes exhaustive post-compression evaluation via a lightweight MLP surrogate model. The surrogate is activated once a replay buffer threshold is met; in earlier cycles, a logit-MSE proxy approximates accuracy degradation. This hybrid reward modeling is empirically effective, especially for ViT-scale architectures where logit-MSE is a reliable quality metric.

Active Compression Loop

The system iterates through hierarchical agent feedback adaptation, ensemble-driven budget allocation, blockwise per-kernel RL search, configuration extraction by weighted vote, compression application, fine-tuning, surrogate update, and global best tracking. This loop is explicit:

1. HLA updates states with empirical accuracy/compression feedback.
2. HLA ensemble allocates budgets.
3. LLA ensembles optimize configurations within budget constraints.
4. Compression is applied and fine-tuned.
5. Surrogate model is updated with new configuration-performance pairs.
6. Best configuration is tracked over cycles.

Empirical Evaluation

Seven experiments are reported across ViT (DeiT, CLIP ViT) and CNN (ResNet18, MobileNetV2) on benchmark tasks. All use INT uniform quantization except the last two (mixed INT/FLOAT + log granularity). Strong numerical results include:

• DeiT-Small/CIFAR-100: 6.23$\times$ compression ratio (CR), 1.72% accuracy drop; sensitivity-aware allocation preserves critical blocks.
• CLIP ViT-B/32/CIFAR-10: 6.64$\times$ CR, 3.83% accuracy gain (compression-as-regularization in over-parameterized regime).
• MobileNetV2/TinyImageNet: 6.59$\times$ CR, 5.62% drop.
• DeiT-Base/CIFAR-100 (Mixed/Log): 6.72$\times$ CR, 2.11% drop (highest CR achieved).

  Figure 4: Per-block compression allocation heatmap reveals sensitivity-driven distribution of compression pressure across transformer blocks and kernel types.

Sensitivity ablation conclusively shows significant improvement (Figure 5: 1.3 percentage points higher accuracy retention, reduced variance), and allocation heterogeneity matches theoretical expectations (Figure 6). LLA training reward curves demonstrate convergence to high-reward configurations (Figure 7).

Figure 5: Multi-seed sensitivity ablation quantifies accuracy improvement and robustness from Fisher-guided allocation.

Figure 6: Sensitivity guidance induces greater blockwise allocation variance, emphasizing differentiated treatment.

Figure 7: LLA reward curve illustrates learning dynamics: improvement and stabilization before fine-tuning.

Contextual Comparison and Contradictory Claims

HiReLC asserts joint optimization of structured pruning and mixed-precision quantization in a unified architecture-agnostic controller, which is contradicts single-axis compression protocols seen in AMC, HAQ, HAWQ-V2, and I-ViT. Contextual Pareto analysis (Figure 8) situates HiReLC trade-offs at the frontier of parameter-storage reduction vs. accuracy drop, with the caveat that datasets and evaluation protocols vary.

Figure 8: Contextual Pareto frontier comparison among HiReLC and contemporary compression baselines.

Ablations further demonstrate that omitting HLA budgets or agent ensemble diversity yields substantial accuracy degradation or instability. Logit-MSE is validated as a practical reward proxy.

Practical and Theoretical Implications

From a deployment perspective, HiReLC's effective parameter-storage compression ratios ($\approx6\times$) with moderate accuracy drops (0.55–5.62%) across diverse architectures can substantially reduce memory and storage requirements, facilitating mobile or edge deployment. The modularity allows extension to hardware-in-the-loop reward modeling and adaptation to latency or energy objectives.

Theoretically, hierarchical RL decomposition is validated for high-dimensional combinatorial search in network compression, with sensitivity-aware guidance demonstrably improving retention and allocation heterogeneity. The surrogate-proxy hybrid approach is practical for amortizing evaluation cost in settings where full fine-tuning is expensive.

Furthermore, the positive accuracy change reported for CLIP ViT-B/32 contradicts dominant assumptions that compression universally degrades performance; the regularization effect is empirically clear.

Speculative Future Directions

Anticipated future work includes:

• Integration of hardware latency or energy metrics for deployment-aware compression.
• Extension of sensitivity estimation (e.g., Hessian trace, SNIP, GraSP) and ensemble agent types.
• Matched benchmarking on standard datasets (ImageNet-1K), precise latency/speedup reporting.
• Tighter constraint handling (deterministic post-processing for accuracy drop compliance).

The framework's reproducibility-ready design and parameterization enable immediate experimentation across backbone and task regimes.

Conclusion

HiReLC establishes a robust hierarchical RL paradigm for joint pruning and quantization, achieving competitive parameter-storage compression with modest accuracy degradation across transformer and CNN architectures. Sensitivity-aware budgeting and active learning via surrogate modeling are demonstrated as practical, effective design choices. The methodology provides a foundation for future AI compression frameworks targeting broader deployment objectives and more stringent practical constraints.