COREY: A Prototype Study of Entropy-Guided Operator Fusion with Hadamard Reparameterization for Selective State Space Models

Abstract: State Space Models (SSMs), represented by the Mamba family, provide linear-time sequence modeling and are attractive for long-context inference. Yet practical deployments remain memory-bandwidth limited because selective state updates are often decomposed into fragmented kernels with repeated intermediate tensor materialization. We present COREY, a prototype framework that combines memory-aware operator fusion with Hadamard-based feature reparameterization. Activation entropy, estimated with fixed-width histograms, is used as a runtime scheduling statistic to place fusion boundaries and choose tile sizes. To regularize heavy-tailed activations, we absorb normalized Hadamard transforms into linear projections, preserving functional equivalence while reducing peak-coordinate concentration. In a controlled prototype study over heavy-tailed SSM activations, COREY consistently reduces proxy latency, improves throughput, and lowers DRAM traffic relative to unfused and fixed-depth baselines. Low-bit results are reported only through a hand-crafted stability proxy and are intended as diagnostic evidence rather than checkpoint-level quality claims. Code repository: https://github.com/mabo1215/COREY_Transformer.git.

• The paper demonstrates that using runtime activation entropy and Hadamard reparameterization improves fusion feasibility and quantization stability in selective state space models.
• The methodology dynamically selects fusion tile sizes based on measured entropy, effectively reducing DRAM traffic and stabilizing heavy-tailed activations.
• Empirical results from prototype tests show enhanced throughput and lower latency, validating the approach for memory-bound, long-context sequence models.

Entropy-Guided Operator Fusion with Hadamard Reparameterization in Selective State Space Models: An Expert Analysis of COREY

Introduction and Motivation

State Space Models (SSMs) exemplified by the Mamba architecture offer compelling linear-time ($\mathcal{O}(L)$) sequence modeling, mitigating the quadratic bottlenecks of Transformer attention for long-context scenarios. However, hardware deployments observe a divergence between algorithmic asymptotics and realized efficiency due to fragmented operator chains and excessive DRAM traffic from intermediate materialization. Furthermore, heavy-tailed activation distributions—common in SSMs—destabilize low-bit quantization and limit fusion, impeding both hardware efficiency and numerical robustness.

The paper "COREY: A Prototype Study of Entropy-Guided Operator Fusion with Hadamard Reparameterization for Selective State Space Models" (2604.10597) addresses these interlocked challenges. The authors propose COREY, a framework that utilizes runtime activation entropy as a proxy for fusion boundary selection and tile sizing, co-optimizing fusion feasibility, outlier suppression, and quantization stability. The core mechanism leverages Hadamard-based feature reparameterization, which is fused into projection blocks to regularize activations while retaining exact functional equivalence.

Methodological Framework

Entropy-Guided Fusion Scheduling

The main contribution is a scheduling rule where the merging of operator chains is guided not only by traditional hardware metrics (arithmetic intensity and memory traffic) but also by local empirical entropy of activations. For contiguous operator regions, a fusion score $S(\mathcal{R})$ is computed via a weighted combination of normalized entropy, arithmetic intensity, and estimated memory footprint. Fusion occurs when this score surpasses a tunable threshold, provided resource constraints (registers, shared memory) are satisfied. Entropy is efficiently estimated via histogram binning and exponential moving average tracking to ensure online feasibility.

Figure 1: Schematic of entropy-guided selective operator fusion leveraging Hadamard-based reparameterization and dynamic tile allocation.

Hadamard Reparameterization

To mitigate heavy-tailed activation distributions, a normalized Hadamard transform is fused into projection layers, producing a more isotropic activation landscape. Explicitly, a projection $y = Wx$ is recast as $y = W H^\top H x = W'Hx$ where $H$ is the Hadamard matrix. This operation increases empirical entropy without altering the representational content—enabling deeper fusion and more aggressive quantization due to attenuated peak-channel deviations. Theoretical analysis establishes that, under doubly-stochastic mixing, this transform provably increases histogram entropy and contracts the maximum coordinate norm, conferring strictly tighter clipping bounds.

Figure 2: Empirical increase in measured activation entropy post-Hadamard reparameterization across varying sequence lengths.

Entropy-Tiling Co-Design

Tile sizes for fused kernels are dynamically selected based on measured entropy. High-entropy regions—indicative of uniformly distributed activations—are processed in larger tiles to maximize reuse and minimize kernel launches. Conversely, low-entropy (outlier-prone) regions are assigned smaller tiles to limit register and shared-memory pressure, reducing the risk of instability.

Theoretical Analysis

• Histogram Entropy Growth: The Hadamard-based mixing increases entropy as long as the underlying mixing is doubly stochastic and the pre-transform distribution is non-uniform. This reconciles activation smoothing with schedule feasibility in the presence of activation outliers.
• Fusion Depth Bound: Fusion group depth is inherently bounded by hardware shared memory limits; entropy-aware scheduling ensures that only resource-feasible segments are fused.
• Quantization Robustness: The fused Hadamard transformation preserves $\ell_2$ norm while strictly contracting the $\ell_\infty$ norm, reducing sensitivity to outliers which is especially significant for aggressive W4A8 quantization.

Empirical Results

The authors perform a controlled prototype study using synthetic heavy-tailed activation workloads, stratified by sequence length. The principal findings are summarized as follows:

• Throughput and Latency: Entropy-guided fusion consistently improves throughput and reduces latency relative to unfused and static-fusion baselines as sequence length increases (particularly in memory-bound regimes). The reduction in DRAM bytes per token confirms that gains derive from lowered memory traffic rather than from compute-side acceleration.

  Figure 3: Latency and throughput as a function of sequence length bucket for FP16 prototype execution.

• Quantization Stability: The diagnostic proxy for low-bit quantization demonstrates clear improvements under entropy-guided fusion with Hadamard smoothing, especially at W8A8 and W4A8 bit widths.
• Robustness: Gains are more pronounced for longer contexts and heavier-tailed activations, validating the theoretical linkage between entropy, fusion feasibility, and quantization stability.

Ablation studies further corroborate the mechanism: (1) tuning the entropy threshold allows one to trade off latency for quality, (2) entropy-guided tiling outperforms static tiling across all buckets, and (3) Hadamard absorption is necessary for the observed quantization robustness.

Practical Implications and Future Directions

Practically, the integration of entropy-guided scheduling into SSM inference pipelines offers a path to closing the gap between theoretical linear-time complexity and hardware-limited realization. The framework is device- and backend-agnostic, admitting integration into future Triton or CUDA fused-kernel implementations. The authors’ experiments on real SSM checkpoints (Mamba-370M, Mamba-1.4B, 2.8B) with LongBench and WikiText-103 demonstrate reproducibility and cross-device robustness, although the evaluation remains at the prototype level. These results suggest that runtime entropy signals could be exploited in broader classes of sequence models where activation outliers and memory traffic dominate inference costs.

Notably, hardware accelerators exposing native Hadamard primitives and entropy-addressable tiling may further amplify the benefits of this approach, pushing the frontier toward co-designed algorithm-system interfaces.

Limitations

The principal limitations of this work arise from its prototype nature: (1) all cost and quality metrics are derived from surrogate models, not direct fused-kernel deployment; (2) entropy, while effective as a global proxy, cannot capture complex inter-channel dependencies and may require extensive tuning per deployment; (3) the efficacy of Hadamard smoothing is maximal for heavy-tailed distributions—models with more uniform activations may see attenuated benefits; and (4) downstream task accuracy and deployment-level perplexity preservation remain open validation targets. The methodology for quantization analysis relies on a custom diagnostic proxy, which must eventually be validated against application-grade benchmarks.

Conclusion

COREY systematically unites entropy-guided fusion scheduling, dynamic tiling, and Hadamard-based reparameterization within selective SSMs, yielding reliable reductions in memory traffic, latency, and quantization-induced degradation in the heavy-tailed regime. This methodology is mathematically justified, empirically validated at the prototype level, and is modular for integration into modern high-performance sequence model inference pipelines. Immediate next steps include broader benchmark coverage, real fused-kernel deployments, and deployment-grade task evaluation to translate these controlled improvements into real-world system impact.

---

Reference:

"COREY: A Prototype Study of Entropy-Guided Operator Fusion with Hadamard Reparameterization for Selective State Space Models" (2604.10597)