Robustifying Sparse Matrix Multiplication

Abstract: In the seminal sparse matrix multiplication problem the goal is to compute the product of two $n \times n$ matrices when the matrices are sparse, i.e., when the number of nonzeros in the input matrices $m_{in}$ and/or the number of nonzeros in the output matrix $m_{out}$ are much smaller than $n<sup>2$. In this paper, we explore the generalized problem of (approximately) computing the $k$ largest output entries, with an approximation error dependent solely on the smaller entries -- from the viewpoint of sparse recovery, this can be seen as a robust variant of sparse matrix multiplication. Despite the substantial research dedicated to sparse matrix multiplication, almost no existing algorithms are robust in this sense. The one exception is Pagh's algorithm in time $\widetilde O(m_{in} + nk)$ [ITCS'12], and it remained open whether other algorithms can be similarly made robust. Our principal contribution is a black-box reduction from robust sparse matrix multiplication to conventional sparse matrix multiplication with only polylogarithmic overhead. Specifically, we show that any sparse matrix multiplication algorithm with running time $T(n, m_{in}, m_{out})$ can be transformed into a robust algorithm running in time $\widetilde O(T(n, m_{in}, k))$. This reduction leverages an extensive toolkit from sparse recovery, and intriguingly, also involves solving a knapsack-type problem. By plugging in the state-of-the-art algorithm for sparse matrix multiplication by Abboud, Bringmann, Fischer, and Künnemann [SODA'24], we achieve significantly improved bounds such as $O((m_{in} + k)<sup>{1.346})$. Notably, in the regime where $k \geq m_{in}<sup>{1.762}$, our reduction culminates in an almost-optimal $k<sup>{1+o(1)}$-time algorithm.

• The paper introduces a black-box reduction that robustifies any sparse matrix multiplication algorithm to achieve strong ℓ∞/ℓ2 recovery guarantees.
• It leverages budget allocation via ℓ2-tail estimation and heavy-hitter recovery to efficiently isolate the k largest entries amid noise.
• The framework significantly improves runtime bounds over previous methods, especially in regimes with moderate k relative to input nonzeros.

Robust Black-Box Reduction for Sparse Matrix Multiplication

Problem Context and Motivation

Sparse matrix multiplication ($AB$ for $n \times n$ $A,B$ with sparse structure) is central in numerical linear algebra, theoretical computer science, IR, and pattern recognition. Classical approaches focus on achieving optimal $\widetilde{O}(IN + OUT)$ time, where $IN$ and $OUT$ are input and nonzero output counts. However, many practical applications, including similarity search and pattern matching, are inherently robust: relevant outputs manifest as $k \ll n^2$ entries with significant magnitude, while the remainder contain “noise” due to intrinsic signal properties or observational errors. This motivates computing only the $k$ largest (potentially noisy) entries of $AB$ with additive error dependent on the “tail mass” of the small entries, formalized using an $\ell_\infty / \ell_2$ recovery guarantee aligned with the robust sparse recovery perspective.

While robust sparse recovery and compressed sensing are extensively studied for vectors, robustification of sparse matrix multiplication received almost no prior algorithmic attention outside Pagh, who designed an FFT-based method with $n \times n$0 complexity. The paper fills this gap via a unified framework that elevates any output-sensitive sparse matrix multiplication algorithm to one obeying robust recovery guarantees, with only polylogarithmic overhead in time.

Definitions, Robust Recovery Model, and Formal Guarantees

Given sparse $n \times n$1 and integer $n \times n$2, the goal is to compute $n \times n$3 so that

$n \times n$4

where $n \times n$5 denotes zeroing out the largest $n \times n$6 entries. This coincides with the best-known $n \times n$7-style robust guarantees from compressed sensing and is strictly stronger than $n \times n$8 forms. The robustification generalizes exact output-sensitive multiplication: setting $n \times n$9 reduces the error to $A,B$0. Thus, robust sparse matrix multiplication interpolates between exact and approximate computation, maintaining relevance across regimes.

Main Technical Contribution: A Black-Box Robustification Reduction

The principal result is a black-box reduction: any algorithm for (exact) sparse matrix multiplication with runtime $A,B$1 can be robustified for the above guarantee at cost $A,B$2, i.e., with only a polylogarithmic slowdown and replacing $A,B$3 by $A,B$4. The reduction is randomized and succeeds with high probability. The reduction leverages key primitives from robust sparse recovery, efficiently computing “budget allocations” (columnwise tail sizes), using a toolkit based on the $A,B$5-tail estimation, heavy-hitter recovery, and a new efficient bi-criteria solution to a minimum-cost Multiple-Choice Knapsack problem for budget assignment. This framework also inherits the “rectangular” fast matrix multiplication benefits of recent state-of-the-art output-sensitive algorithms such as Abboud et al. (SODA ‘24), yielding improved bounds:

• $A,B$6 randomized time for robust sparse matrix multiplication.
• If $A,B$7, one obtains $A,B$8 time—optimal up to subpolynomial factors, matching the obvious output lower bound.

This improves upon the best previous robust result (i.e., Pagh’s $A,B$9 algorithm) in all but the very sparse regimes ($\widetilde{O}(IN + OUT)$0).

Algorithmic Insights and Structural Overview

The reduction unfolds as follows:

1. Budget Allocation Scheme: For a matrix $\widetilde{O}(IN + OUT)$1, compressive sketches estimate, for every column, the $\widetilde{O}(IN + OUT)$2-tail energy after removing its $\widetilde{O}(IN + OUT)$3 largest components for candidate budgets $\widetilde{O}(IN + OUT)$4. The budgets are chosen to satisfy $\widetilde{O}(IN + OUT)$5 and per-column tail energy normalized by $\widetilde{O}(IN + OUT)$6 bounded by the global tail $\widetilde{O}(IN + OUT)$7, thus preventing columns with large tail energy from receiving insufficient attention.
2. Columnwise Robust Sparse Recovery: For each group of columns with equal $\widetilde{O}(IN + OUT)$8, independently robust recovery is performed using state-of-the-art heavy-hitter algorithms (with blockwise sketching). Large output entries are accurately recovered in $\widetilde{O}(IN + OUT)$9-norm, while error on the others is proportional to the tail mass, as per the robust guarantee.
3. Efficiency via Minimum-Cost Multiple-Choice Knapsack: Assigning the $IN$0 budgets reduces to a Multiple-Choice Knapsack instance, efficiently solved ($IN$1) using advanced FPTAS techniques, thus avoiding $IN$2-factor blowup from naïve guessing.

The entire process is inherently modular, decoupling robustification from the underlying multiplication routine. This enables leveraging any advances in “exact” sparse matrix multiplication as new state-of-the-art robust algorithms.

Quantitative Improvements and Comparison

Strong improvements over previous work (notably Pagh ‘13):

• Any output-sensitive algorithm with $IN$3 can, via robustification, handle output up to $IN$4 large entries at essentially the same cost, replacing OUT with $IN$5 (plus polylogarithmic factors).
• In typical regimes with $IN$6, obtain time $IN$7, exponentially faster in $IN$8 than $IN$9 methods.
• When $OUT$0, time is optimal, i.e., the dominant cost comes from simply reporting the output.

Connections and Implications to Approximate Matrix Multiplication

The analysis also delineates the contrast with classical approximate matrix multiplication (AMM): In AMM, error is scaled by global norms (e.g., $OUT$1), suiting settings where all entries matter roughly equally. In robust sparse multiplication, the error is instead scaled by the mass outside the “large” entries, making it more suitable for the increasingly dominant “spiky” or heavy-tailed matrices encountered in modern data analysis. The reduction even recovers and strictly improves guarantees for large-scale AMM under robust norms, with the same runtime as classical sampling- or projection-based approaches but with an error scaled to the signal tail rather than the global norm.

Broader Theoretical and Practical Impact

Theoretically, the robustification framework unifies sparse matrix multiplication and compressed sensing paradigms, demonstrating that robust guarantees are achievable at the cost of their classical analogs. The reduction is especially notable for being black-box: any future improvement in sparse matrix multiplication for the exact model immediately lifts to the robust setting. The budget allocation primitives also appear transferable to distributed and streaming settings, as demonstrated in applications such as efficient distributed heavy-hitter protocols.

Practically, the advances enable robust and efficient processing of large sparse matrices prevalent in graph analytics, large-scale IR, and ML, where accurate identification of “significant” interactions dominates over reconstructing all (mostly negligible) entries. The robustness to noise is algorithmically guaranteed, rather than heuristic. The powerful asymptotics render the robustification procedure applicable well beyond the sizes encountered by earlier robust methods.

Future directions include deterministic robust sparse multiplication (current methods are randomized), extension to asymmetric and broader matrix classes, and deployment in dynamic/streaming scenarios where robust error is essential for reliability.

Conclusion

The work resolves the central question of robustifying sparse matrix multiplication, showing that robust recovery is not inherently more expensive than output-sensitive multiplication of the same output sparsity. The introduced reduction is black-box, modular, and enables robust signal processing even as theoretical and engineering advances in output-sensitive algorithms continue. The results expand the landscape of both matrix multiplication and robust recovery, and provide structural and algorithmic primitives likely to be broadly applicable in sparse/approximate algebra, compressed sensing, and distributed computing.

Reference: "Robustifying Sparse Matrix Multiplication" (2607.01427)