Stochastic Pattern-Completion Systems

• Stochastic pattern-completion systems are computational frameworks that reconstruct incomplete tensors using nonnegative constraints and stochastic gradient methods.
• They employ alternating optimization to decompose the tensor completion task into manageable nonnegative matrix subproblems, enhancing scalability.
• The approach leverages parallel accelerated stochastic gradients with adaptive step-sizes for rapid convergence and efficient large-scale performance.

Stochastic pattern-completion systems constitute a class of computational methods designed to reconstruct incomplete multidimensional arrays (“tensors”), particularly under nonnegativity constraints and in the presence of stochasticity introduced via sampling, noise, or algorithmic randomness. In the context of tensor completion, these systems are distinguished by their use of stochastic optimization schemes and pattern-exploiting decompositions to efficiently and scalably infer dense structure from sparse, partially observed, or corrupted data. The accelerated stochastic system proposed in "Accelerated Stochastic Gradient for Nonnegative Tensor Completion and Parallel Implementation" exemplifies this approach by integrating alternating optimization over CPD decomposed tensor factors with a parallel and locally adaptive stochastic gradient framework (Siaminou et al., 2021).

1. Mathematical Foundation of Nonnegative Tensor Completion

Let ${\mathbfcal X}^o \in \mathbb{R}_+^{I_1 \times \cdots \times I_N}$ be an unknown nonnegative tensor of canonical polyadic decomposition (CPD) rank $R$, represented as

${\mathbfcal X}^o = \llbracket U^{o(1)}, \dots, U^{o(N)} \rrbracket = \sum_{r=1}^R u_r^{o(1)} \circ \cdots \circ u_r^{o(N)},$

with each factor $U^{o(i)} \in \mathbb{R}_+^{I_i \times R}$. Only a noisy, partially observed version is accessible:

${\mathbfcal X} = {\mathbfcal X}^o + {\mathbfcal E}, \qquad {\mathbfcal M}(i_1, \dots, i_N) = \begin{cases} 1 & (i_1, \dots, i_N) \in \Omega \ 0 & \text{otherwise} \end{cases}$

where $\Omega$ indexes observed entries and ${\mathbfcal M}$ is the binary mask.

The nonnegative tensor completion (NTC) task is then formalized as:

$$\min_{\{ U^{(i)} \geq 0 \}_{i=1}^N} F(U^{(1)}, \dots, U^{(N)}) + \frac{\lambda}{2} \sum_{i=1}^N \|U^{(i)}\|_F^2,$$

where

$F(U^{(1)}, \dots, U^{(N)}) = \frac{1}{2} \| {\mathbfcal M} \circledast ( {\mathbfcal X} - \llbracket U^{(1)}, \dots, U^{(N)} \rrbracket ) \|_F^2.$

Mode-wise unfolding yields an equivalent nonnegative matrix completion (NMC) subproblem for each factor.

2. Alternating Optimization and Subproblem Decomposition

The tensor completion system operates via an outer alternating optimization (AO) loop cycling through all tensor modes $i = 1, \dots, N$. At each step $R$0, the system computes the Khatri–Rao product $R$1 of all modes except $R$2, splitting the full problem into a sequence of nonnegative matrix completion subproblems:

$R$3

Here, $R$4 and $R$5 are the unfolded tensor and mask in mode $R$6, and $R$7 denotes the stochastic matrix solver.

This approach enables scalability for high-dimensional and large-scale tensor data by leveraging separability across modes and subspace optimizations tailored to the NMC context.

3. Stochastic Accelerated Gradient for NMC

The nonnegative matrix completion subproblem addressed at each AO step is defined as:

$R$8

where the optimization variables and data are $R$9, ${\mathbfcal X}^o = \llbracket U^{o(1)}, \dots, U^{o(N)} \rrbracket = \sum_{r=1}^R u_r^{o(1)} \circ \cdots \circ u_r^{o(N)},$0, ${\mathbfcal X}^o = \llbracket U^{o(1)}, \dots, U^{o(N)} \rrbracket = \sum_{r=1}^R u_r^{o(1)} \circ \cdots \circ u_r^{o(N)},$1.

At each inner iteration ${\mathbfcal X}^o = \llbracket U^{o(1)}, \dots, U^{o(N)} \rrbracket = \sum_{r=1}^R u_r^{o(1)} \circ \cdots \circ u_r^{o(N)},$2, a subset ${\mathbfcal X}^o = \llbracket U^{o(1)}, \dots, U^{o(N)} \rrbracket = \sum_{r=1}^R u_r^{o(1)} \circ \cdots \circ u_r^{o(N)},$3 is generated by subsampling a fraction ${\mathbfcal X}^o = \llbracket U^{o(1)}, \dots, U^{o(N)} \rrbracket = \sum_{r=1}^R u_r^{o(1)} \circ \cdots \circ u_r^{o(N)},$4 of nonzeros within each row. The corresponding stochastic cost and gradient estimates are computed:

${\mathbfcal X}^o = \llbracket U^{o(1)}, \dots, U^{o(N)} \rrbracket = \sum_{r=1}^R u_r^{o(1)} \circ \cdots \circ u_r^{o(N)},$5

executed row-wise.

Row-wise local smoothness constants ${\mathbfcal X}^o = \llbracket U^{o(1)}, \dots, U^{o(N)} \rrbracket = \sum_{r=1}^R u_r^{o(1)} \circ \cdots \circ u_r^{o(N)},$6 are determined via the spectral properties of each block Hessian:

${\mathbfcal X}^o = \llbracket U^{o(1)}, \dots, U^{o(N)} \rrbracket = \sum_{r=1}^R u_r^{o(1)} \circ \cdots \circ u_r^{o(N)},$7

defining individual row step-sizes and momentum parameters:

${\mathbfcal X}^o = \llbracket U^{o(1)}, \dots, U^{o(N)} \rrbracket = \sum_{r=1}^R u_r^{o(1)} \circ \cdots \circ u_r^{o(N)},$8

with extrapolation:

${\mathbfcal X}^o = \llbracket U^{o(1)}, \dots, U^{o(N)} \rrbracket = \sum_{r=1}^R u_r^{o(1)} \circ \cdots \circ u_r^{o(N)},$9

Theoretically, the update exhibits an $U^{o(i)} \in \mathbb{R}_+^{I_i \times R}$0 convergence rate in the strongly convex, smooth regime, with rapid empirical convergence in practical settings (Siaminou et al., 2021).

4. Parallelization and Computational Scalability

The independence of row-wise stochastic accelerated gradient updates enables highly efficient parallelization. Within each inner iteration, each thread processes a distinct set of rows, with no overlap in writes to $U^{o(i)} \in \mathbb{R}_+^{I_i \times R}$1 or $U^{o(i)} \in \mathbb{R}_+^{I_i \times R}$2. Synchronization is relegated to an implicit barrier at the end of each inner loop over $U^{o(i)} \in \mathbb{R}_+^{I_i \times R}$3.

OpenMP is the parallelization framework utilized, supporting nearly linear speedup up to the physical core count. The table below summarizes scaling efficiency observed on benchmark datasets:

# Threads	Dataset	Speedup Factor
1	--	1×
16	Uber pickups (4th order, real)	10-15×
16	Uniform synthetic	>15×

On a $U^{o(i)} \in \mathbb{R}_+^{I_i \times R}$4 Uber pickups tensor ($U^{o(i)} \in \mathbb{R}_+^{I_i \times R}$5M nonzeros), using 1 vs. 16 threads yields a $U^{o(i)} \in \mathbb{R}_+^{I_i \times R}$6–$U^{o(i)} \in \mathbb{R}_+^{I_i \times R}$7 reduction in per-epoch time, with stronger scaling on load-balanced synthetic data (Siaminou et al., 2021).

5. Empirical Performance and Benchmarks

Extensive empirical evaluation demonstrates the system’s effectiveness in several application regimes:

• Image completion: RGB image tensor $U^{o(i)} \in \mathbb{R}_+^{I_i \times R}$8, $U^{o(i)} \in \mathbb{R}_+^{I_i \times R}$9 missing, rank ${\mathbfcal X} = {\mathbfcal X}^o + {\mathbfcal E}, \qquad {\mathbfcal M}(i_1, \dots, i_N) = \begin{cases} 1 & (i_1, \dots, i_N) \in \Omega \ 0 & \text{otherwise} \end{cases}$0, subsampling ${\mathbfcal X} = {\mathbfcal X}^o + {\mathbfcal E}, \qquad {\mathbfcal M}(i_1, \dots, i_N) = \begin{cases} 1 & (i_1, \dots, i_N) \in \Omega \ 0 & \text{otherwise} \end{cases}$1, and ${\mathbfcal X} = {\mathbfcal X}^o + {\mathbfcal E}, \qquad {\mathbfcal M}(i_1, \dots, i_N) = \begin{cases} 1 & (i_1, \dots, i_N) \in \Omega \ 0 & \text{otherwise} \end{cases}$2 epochs result in visually successful reconstructions even under minimal stochastic sampling.
• Large-scale recommendation: MovieLens 10M tensor ${\mathbfcal X} = {\mathbfcal X}^o + {\mathbfcal E}, \qquad {\mathbfcal M}(i_1, \dots, i_N) = \begin{cases} 1 & (i_1, \dots, i_N) \in \Omega \ 0 & \text{otherwise} \end{cases}$3 and synthetic analogs show that, in high-noise environments, larger stochastic fractions ${\mathbfcal X} = {\mathbfcal X}^o + {\mathbfcal E}, \qquad {\mathbfcal M}(i_1, \dots, i_N) = \begin{cases} 1 & (i_1, \dots, i_N) \in \Omega \ 0 & \text{otherwise} \end{cases}$4 yield faster reduction in relative reconstruction error (RRE) over epochs, while in low-noise regimes, smaller ${\mathbfcal X} = {\mathbfcal X}^o + {\mathbfcal E}, \qquad {\mathbfcal M}(i_1, \dots, i_N) = \begin{cases} 1 & (i_1, \dots, i_N) \in \Omega \ 0 & \text{otherwise} \end{cases}$5 suffices:

${\mathbfcal X} = {\mathbfcal X}^o + {\mathbfcal E}, \qquad {\mathbfcal M}(i_1, \dots, i_N) = \begin{cases} 1 & (i_1, \dots, i_N) \in \Omega \ 0 & \text{otherwise} \end{cases}$6

• Scalability: Per-epoch computation time diminishes nearly linearly as threads increase from ${\mathbfcal X} = {\mathbfcal X}^o + {\mathbfcal E}, \qquad {\mathbfcal M}(i_1, \dots, i_N) = \begin{cases} 1 & (i_1, \dots, i_N) \in \Omega \ 0 & \text{otherwise} \end{cases}$7 to the core limit, for all tested ranks ${\mathbfcal X} = {\mathbfcal X}^o + {\mathbfcal E}, \qquad {\mathbfcal M}(i_1, \dots, i_N) = \begin{cases} 1 & (i_1, \dots, i_N) \in \Omega \ 0 & \text{otherwise} \end{cases}$8.

6. System Architecture Summary and Significance

The stochastic pattern-completion system integrates an AO outer loop centered on CPD-based tensor factorization with an inner, accelerated stochastic per-row solver that leverages local smoothness constants to adaptively set step-sizes. Full row-wise independence is harnessed via OpenMP, leading to parallel efficiency and scaling on extremely large, sparse tensors. This construction enables recovery of missing tensor structure in high-noise, high-dimensional settings with practical computational expenditure (Siaminou et al., 2021).

A plausible implication is that such systems will be critical for domain applications where data is both large and highly incomplete, such as recommendation, remote sensing, and scientific data fusion, given their robust empirical convergence for ill-posed and strongly regularized completion tasks.