Compressed Outer-Product Construction

• Compressed Outer-Product Construction is a family of techniques that approximates bilinear and multilinear operations by expressing outputs as sums of rank-1 outer products with reduced computational and storage demands.
• These methods are applied in randomized linear algebra, tensor networks, and parameter-efficient model adaptation, employing strategies like sampling, projection, and hashing to optimize performance.
• Theoretical guarantees ensure unbiased estimations with error bounds tied to stable ranks and sketch sizes, making the techniques scalable and practical for large-scale data and complex model tasks.

A compressed outer-product construction refers to a family of algorithmic, algebraic, and combinatorial techniques that approximate or encapsulate a target bilinear or multi-linear operation—such as matrix multiplication, tensor contraction, or direct-product organization—by expressing the output as a sum (or structured set) of rank-1 outer products, but with a reduced computational, storage, or sample complexity versus the naïve expansion. These constructions are central to randomized linear algebra, tensor networks, low-parameter model adaptation (e.g., prompt tuning), sketching algorithms, as well as combinatorial design, each with distinct mathematical frameworks that exploit compression either via sampling, projection, randomized mapping, or algebraic reorganization of product terms.

1. Probabilistic Compression via Outer-Product Sampling

A prominent instantiation is the spectral-norm-accurate approximation of a matrix product $AB^\top$ via weighted sampling of outer products. Given $A=[a_1|\cdots|a_n]\in\mathbb{R}^{d_1\times n}$, $B=[b_1|\cdots|b_n]\in\mathbb{R}^{d_2\times n}$, the product admits the decomposition $AB^\top = \sum_{i=1}^n a_ib_i^\top$. The compressed outer-product estimator samples $m$ terms $a_i b_i^\top$, scaling each by $1/p_i$ where

$$p_i = \frac{1}{2}\,\frac{\|a_i\|^2}{\|A\|_F^2} + \frac{1}{2}\,\frac{\|b_i\|^2}{\|B\|_F^2}.$$

Averaging $m$ such draws yields an unbiased estimate $\widehat{AB^\top}$. Theorem: $A=[a_1|\cdots|a_n]\in\mathbb{R}^{d_1\times n}$0 suffices to achieve spectral-norm error $A=[a_1|\cdots|a_n]\in\mathbb{R}^{d_1\times n}$1 with high probability, where $A=[a_1|\cdots|a_n]\in\mathbb{R}^{d_1\times n}$2 denotes the stable rank. This method achieves stable-rank dependence without reference to the ambient dimensions, and improves upon previous Frobenius-norm-based sampling via a balanced leverage construction (Hsu, 2014).

2. Sketching and Hash-Based Outer-Product Compression

Compressed outer-product techniques are also foundational in sketching algorithms for large-scale matrix multiplication. In Pagh's algorithm and subsequent engineering over the Fast Walsh-Hadamard Transform (FWHT), matrix multiplication $A=[a_1|\cdots|a_n]\in\mathbb{R}^{d_1\times n}$3 is reframed as $A=[a_1|\cdots|a_n]\in\mathbb{R}^{d_1\times n}$4, and each rank-1 component is "compressed" by randomly hashing its rows and columns into $A=[a_1|\cdots|a_n]\in\mathbb{R}^{d_1\times n}$5 buckets using pairwise independent hash and sign functions. The linearity of the sketching procedure then ensures that the aggregate sketch is an unbiased estimator for the full product entries. FFT or FWHT-based convolution accelerates the aggregation step. The variance of the estimator for any $A=[a_1|\cdots|a_n]\in\mathbb{R}^{d_1\times n}$6 is bounded above by $A=[a_1|\cdots|a_n]\in\mathbb{R}^{d_1\times n}$7, and using $A=[a_1|\cdots|a_n]\in\mathbb{R}^{d_1\times n}$8 repetitions with median aggregation yields error guarantees with probability $A=[a_1|\cdots|a_n]\in\mathbb{R}^{d_1\times n}$9. The FWHT variant attains reduced constant factors due to real arithmetic and in-place memory layout, achieving notable practical speedups over state-of-the-art DGEMM in favorable sparsity regimes (Andersson et al., 14 Jan 2026).

3. Compression in Tensor and Tensor-Network Products

In high-order multilinear algebra, compressed outer-product constructions arise in both structured tensor products and tensor network contractions. The projected tensor–tensor product framework replaces a full invertible transform on tensor modes (which incurs $B=[b_1|\cdots|b_n]\in\mathbb{R}^{d_2\times n}$0 cost in tube-length $B=[b_1|\cdots|b_n]\in\mathbb{R}^{d_2\times n}$1) with a tall-skinny unitary matrix $B=[b_1|\cdots|b_n]\in\mathbb{R}^{d_2\times n}$2, $B=[b_1|\cdots|b_n]\in\mathbb{R}^{d_2\times n}$3. The projected product

$B=[b_1|\cdots|b_n]\in\mathbb{R}^{d_2\times n}$4

preserves associativity, distributivity, and transpose properties tube-wise, while reducing both computational and storage complexity by an $B=[b_1|\cdots|b_n]\in\mathbb{R}^{d_2\times n}$5 factor. Decompositions under this algebra yield rank-1 terms with respect to $B=[b_1|\cdots|b_n]\in\mathbb{R}^{d_2\times n}$6, and truncated SVDs (via "star-P" SVD) achieve optimality in the Frobenius norm modulo a projection error term (Keegan et al., 2024).

In tensor networks, specifically for compressed matrix product operator–matrix product state (MPO–MPS) products, the successive randomized compression (SRC) algorithm constructs a compressed MPS approximation of $B=[b_1|\cdots|b_n]\in\mathbb{R}^{d_2\times n}$7 with target bond dimension $B=[b_1|\cdots|b_n]\in\mathbb{R}^{d_2\times n}$8 via a single randomized right-to-left sweep. At each site, a random projection (Khatri–Rao product of local test matrices) compresses the contracted core, and QR extraction provides orthonormal basis updates. This approach matches or surpasses prior deterministic and explicit methods in both speed and—given moderate oversampling—accuracy, especially for large $B=[b_1|\cdots|b_n]\in\mathbb{R}^{d_2\times n}$9 and substantial compression ratios (Camaño et al., 8 Apr 2025).

4. Low-Rank Compressed Outer Products for Parameter-Efficient Adaptation

Compressed outer-product constructions underpin parameter-efficient modules in large-model adaptation contexts, such as prompt tuning for LLMs. In LAMP, the soft-prompt matrix $AB^\top = \sum_{i=1}^n a_ib_i^\top$0 is approximated as $AB^\top = \sum_{i=1}^n a_ib_i^\top$1, where $AB^\top = \sum_{i=1}^n a_ib_i^\top$2, $AB^\top = \sum_{i=1}^n a_ib_i^\top$3, using the top-$AB^\top = \sum_{i=1}^n a_ib_i^\top$4 truncated singular vectors. This decomposes $AB^\top = \sum_{i=1}^n a_ib_i^\top$5 into a sum of $AB^\top = \sum_{i=1}^n a_ib_i^\top$6 rank-1 outer products, which are efficiently recombined (either by explicit sum or by the matrix product $AB^\top = \sum_{i=1}^n a_ib_i^\top$7) in each forward pass. The approach induces a parameter count of $AB^\top = \sum_{i=1}^n a_ib_i^\top$8, providing order-of-magnitude reductions (e.g., $AB^\top = \sum_{i=1}^n a_ib_i^\top$9K $m$0 $m$1K) in resource requirements versus full-parameter soft prompts, with experimentally measured improvements in both accuracy and run-time efficiency over baseline prompt tuning and LoRA adaptation (Lan et al., 16 Feb 2025).

5. Combinatorial Compression: Outer-Product Constructions in Design Theory

Combinatorial analogues of compressed outer-product construction arise in the context of mutually orthogonal Latin squares (MOLS). The classical Kronecker (outer) product constructs $m$2 MOLS of order $m$3 from $m$4 MOLS of orders $m$5 and $m$6. The three-factor or triple-product construction of Dukes–Ling compresses two successive product steps into one, utilizing repeated applications of Rolf Rees's transversal-design resolution. This, under mild hypotheses ($m$7), yields one additional MOLS beyond the naïve bound: $m$8. The procedure is entirely combinatorial, reorganizing block resolutions and families of parallel classes to extract the extra square, thereby slightly strengthening MacNeish’s classical product theorem for orders with at least three large prime-power factors (Dukes et al., 2014).

6. Implementation and Algorithmic Considerations

The following summarizes representative algorithmic implementations of compressed outer-product techniques:

Construction	Compression Mechanism	Complexity (relative terms)
Operator-norm sampling (Hsu, 2014)	Weighted random sampling	$m$9 for $a_i b_i^\top$0 samples
Sketching (FFT/FWHT) (Andersson et al., 14 Jan 2026)	Random hashing + convolution	$a_i b_i^\top$1
Projected tensor product (Keegan et al., 2024)	Tall-skinny unitary projection	$a_i b_i^\top$2
Truncated SVD expansion (Lan et al., 16 Feb 2025)	Low-rank sum of outer products	$a_i b_i^\top$3
Tensor network SRC (Camaño et al., 8 Apr 2025)	Random projection + contraction	$a_i b_i^\top$4

Practical impact stems from the ability to decouple sample or computation complexity from the full data dimension, instead linking accuracy to effective rank, sketch size, or dimension of the compressed representation. For prompt adaptation, compressed outer-product approaches eliminate memory and parameter bottlenecks. In large-scale tensor contraction, they underpin tractable simulation and learning algorithms.

7. Theoretical Guarantees and Limitations

Each construction provides both unbiasedness and rigorous error bounds under appropriate independence or low-rank assumptions. In matrix sampling (Hsu, 2014), the estimator’s spectral-norm error can be tightly controlled by the stable rank of inputs, and the sample complexity is logarithmic in the minimum stable rank. Hash-based sketching (Andersson et al., 14 Jan 2026) yields unbiased entrywise recovery with variance and tail probabilities scaling inversely with bucket count and repetition. In tensor compression, optimality conditions hold in projected algebra, with approximation error split into truncated-rank and projection loss (Keegan et al., 2024). The combinatorial triple-product’s main limitation is the necessity of sufficiently large families of base MOLS to trigger the “+1” gain; otherwise, improvements over direct-product constructions are not achieved. A plausible implication is that for datasets or models with dominant low-rank or low-complexity structure, compressed outer-product constructions provide near-optimal trade-offs between efficiency and representation quality.