Q3R: Quadratic Reweighted Rank Regularizer

• Q3R is a non-convex rank surrogate that replaces the discontinuous rank objective with a smoothed log-determinant function, efficiently promoting low-rank structure.
• It underpins IRLS algorithms that employ quadratic majorization, ensuring rapid, locally quadratic convergence and superior statistical estimation in matrix recovery and completion.
• The method leverages SVD-based reweighting and adaptive smoothing to balance computational efficiency with robust performance in structured recovery and deep model training.

The Quadratic Reweighted Rank Regularizer (Q3R) is a non-convex rank surrogate and majorization technique designed for inducing and controlling low-rank structure in matrix optimization problems. Q3R underpins a family of iteratively reweighted least squares (IRLS) algorithms for matrix recovery, structured matrix completion, and parameter-efficient training of large-scale deep models. The approach centers on majorizing a smoothed log-determinant function of singular values with a quadratic surrogate, achieving scalable, accurate, and locally quadratic convergence properties essential for both statistical estimation and modern machine learning applications (Ghosh et al., 6 Nov 2025, Kümmerle et al., 2023, Kümmerle et al., 2018).

1. Smoothed Log-Determinant Surrogate for Rank

Q3R replaces the discontinuous matrix rank objective with the smooth but non-convex $\varepsilon$-smoothed log-determinant surrogate. For a matrix $X$ with singular values $\sigma_1(X),..., \sigma_d(X)$, and $\varepsilon>0$, the surrogate takes the form

$$\varphi_{\varepsilon}(X) = \sum_{i=1}^{d} \log(\sigma_i(X) + \varepsilon),$$

or, in alternative Q3R formulations,

$$F_\varepsilon(X) = \sum_{i=1}^d f_\varepsilon(\sigma_i(X)),$$

where $f_\varepsilon(t)$ is piecewise defined: quadratic near zero ($\frac{1}{2}t^2$ for $|t| \leq \varepsilon$), and logarithmic outside ($$\frac{1}{2}\varepsilon^2 \log(e t^2 / \varepsilon^2)$$ for $|t| > \varepsilon$). This surrogate heavily penalizes small singular values, efficiently promoting low-rank structure for optimization and learning objectives (Ghosh et al., 6 Nov 2025, Kümmerle et al., 2023, Kümmerle et al., 2018).

2. Quadratic Majorization and IRLS Schemes

Q3R deploys a quadratic majorizer for the smoothed log-determinant, making the non-convex problem tractable with iterative updates.

At each IRLS iteration $k$, given a current point $X^{(k)}$ (or $W^{(k)}$), the algorithm computes a quadratic expansion at $X^{(k)}$, majorizing the surrogate: $$Q_\varepsilon(X\,|\,X^{(k)}) = \varphi_\varepsilon(X^{(k)}) + \langle \nabla \varphi_\varepsilon(X^{(k)}), X-X^{(k)} \rangle + \frac12 \langle X-X^{(k)},\, U^{(k)}(X-X^{(k)}) \rangle,$$ where $U^{(k)}$ is the reweighting operator built from the leading $r$ singular vectors of $X^{(k)}$. The explicit form exploits the SVD structure to provide tight local curvature information. Constants and linear terms can be omitted for minimization, yielding a simple weighted least squares update centered at the current iterate (Ghosh et al., 6 Nov 2025, Kümmerle et al., 2023).

This procedure is closely aligned with majorization–minimization (MM) principles and Newton-type methods, resulting in provable monotonic decrease of the surrogate and rapid (locally quadratic) convergence under suitable conditions.

3. Algorithmic Structure and Pseudocode

The Q3R-IRLS methodology iteratively alternates between:

• Computation of the quadratic weight operator: Extract leading singular vectors of the current iterate to construct $U^{(k)}$, which encodes per-mode penalties as functions of current singular values and the smoothing parameter $\varepsilon_k$.
• Solving a quadratic subproblem: Minimize the quadratic surrogate, optionally subject to additional constraints (e.g., $A(X)=y$ in recovery, or adding observed loss for deep learning).
• Updating the smoothing parameter: $$\varepsilon_{k+1} = \min(\varepsilon_k, \sigma_{r+1}(X^{(k+1)}))$$, progressively shrinking the penalization region and enforcing rank targets.
• Repeating until convergence.

A representative pseudocode for AdamQ3R (for low-rank neural network training) involves periodically recomputing $U^{(k)}$ using truncated SVD, and interleaving this with adaptive moment optimization steps (Ghosh et al., 6 Nov 2025). For matrix recovery, the algorithm involves alternating quadratic minimization and reweighting, with monotonic convergence guarantees (Kümmerle et al., 2023, Kümmerle et al., 2018).

4. Theoretical Properties: Convergence and Sample Complexity

Q3R-IRLS methods possess strong theoretical guarantees:

• Majorization–minimization property: Each IRLS iteration decreases the value of the smoothed surrogate, ensuring monotonic progress toward a stationary point (Ghosh et al., 6 Nov 2025, Kümmerle et al., 2018).
• Locally quadratic convergence: If the measurement operator $A$ satisfies a restricted isometry property (RIP) and the iterate approaches the true solution $X_*$, the error at each step contracts quadratically: $$\| X^{(k+1)} - X_* \|_2 \leq \mu \| X^{(k)} - X_* \|_2^2,$$ with explicit constants depending on problem details (Kümmerle et al., 2023).
• Near-optimal sample complexity: For random Gaussian measurements, recovery is possible with

$m \gtrsim r(s+n_2) \log(e n_1 / s),$

matching, up to logarithmic factors, the information-theoretic limits for simultaneous rank- and row-sparsity (Kümmerle et al., 2023).

A key advantage is local quadratic convergence rate, which is not attained by convex relaxations such as the nuclear norm.

5. Applications: Structured Recovery and Deep Model Training

The Q3R methodology has demonstrated strong empirical and statistical performance across diverse domains:

• Simultaneous low-rank and sparse matrix recovery: Accurate reconstruction from minimal measurements; Q3R-IRLS achieves superior empirical success rates relative to nuclear-norm or log-sum relaxations, particularly in regimes with heterogeneous structures (Kümmerle et al., 2023).
• Structured low-rank matrix completion: StrucHMIRLS, a structured Q3R-IRLS variant, efficiently recovers low-rank Hankel and Toeplitz matrices for tasks such as spectral super-resolution, achieving high success rates near the minimal sampling limit (Kümmerle et al., 2018).
• Low-rank training of deep models: AdamQ3R applies the Q3R principle to large-scale neural network training, retaining high predictive accuracy while enforcing low target ranks. For example, ViT-Tiny models pruned to 40% parameter retention incur less than 1.3% accuracy loss on CIFAR-10. Comparable benchmarks show superior or comparable task accuracy to dense or other parameter-efficient baselines, including LoRA and LoRITa, across transformers and LLMs (Ghosh et al., 6 Nov 2025).

6. Computational Aspects and Implementation

Q3R-based IRLS methods feature efficient implementations:

• Structured matrix scenarios: Fast Hankel/Toeplitz products yield per-iteration complexity linear in the size of the input vector and quadratic in rank estimate $R$, suitable for large-scale problems (Kümmerle et al., 2018).
• Unstructured cases and deep learning: Memory requirements for the reweighting matrices scale as $O(r(d_1+d_2))$, with each IRLS update for weight matrices of size $d_1 \times d_2$ incurring $O(d_1d_2 r + (d_1+d_2) r^2 + r^3)$ cost, where $r$ is the effective rank (Ghosh et al., 6 Nov 2025).
• Compatibility: The quadratic reweighted regularizer can be applied to any linear or convolutional weight matrix; bias terms are typically not regularized (Ghosh et al., 6 Nov 2025).
• Optimization: The quadratic surrogate integrates seamlessly with first-order optimizers (e.g., Adam), enabling practical training of large, modern architectures with small periodic overhead for recomputing SVD-based weights.

7. Comparative Performance and Empirical Outcomes

Empirical evaluations underscore the statistical efficiency and scalability of Q3R-IRLS schemes. In Hankel matrix completion, StrucHMIRLS recovers the ground truth with nearly minimal samples, outperforming atomic norm and structured nuclear-norm relaxations (Kümmerle et al., 2018). For deep learning tasks, AdamQ3R demonstrates retention of performance under aggressive parameter pruning; e.g., at 40% retention on ViT-Base for CIFAR-100, AdamQ3R matches or exceeds dense and LoRITa models in accuracy. Similar superiority is observed in GLUE benchmark fine-tuning of RoBERTa-base models (Ghosh et al., 6 Nov 2025).

In summary, Q3R regularization achieves practical, scalable, and theoretically grounded low-rank control across both signal processing and modern machine learning settings, furnishing a unifying IRLS-based mechanism with provable and empirical performance advantages (Ghosh et al., 6 Nov 2025, Kümmerle et al., 2023, Kümmerle et al., 2018).