---
title: Lossless Compression for Convex ERM
url: https://www.emergentmind.com/topics/lossless-compression-framework-for-convex-erm
type: topic
---

# Lossless Compression for Convex ERM

A lossless compression framework for convex empirical risk minimization (ERM) enables computational reductions in optimization tasks without loss of solution accuracy. The framework introduced by Zhu & Chen is built on color refinement—a combinatorial technique from graph theory—generalized to operate on data or kernel matrices associated with convex, differentiable ERMs. It yields exact instance compression for problems including linear and polynomial regression, (multi)class logistic regression, elastic-net regularization, and kernelized methods, while preserving all global optima and producing measurable practical speedups [2602.00437].

## 1. Convex ERM: Formulation and Lossless Compression

Given feature vectors $x_i \in \mathbb{R}^D$, labels $y_i$, and positive weights $v_i$, convex ERM in standard (primal) form seeks to minimize:
$$
F(w, b) = \sum_{i=1}^n v_i\,f(x_i^T w + b;\;y_i) + R(w),
$$
where $f$ is convex and differentiable in its first argument, and $R$ is a convex regularizer (e.g., ridge or elastic-net). The kernelized ERM form utilizes:
- Kernel matrix $K_{ij}=k(x_i,x_j)\succeq0$
- Dual variables $\alpha \in \mathbb{R}^n$, bias $b \in \mathbb{R}$

Objective:
$$
F(\alpha, b) = \sum_{i=1}^n v_i\,f((K\alpha + b 1_n)_i ; y_i) + \frac{\lambda}{2} \alpha^T K \alpha
$$
A compression mapping $\varphi:(X, y) \mapsto (\tilde X, \tilde y)$ is *lossless* if the compressed ERM has exactly the same set of solutions as the original problem.

## 2. Color Refinement and Equitable Partitions

Color refinement operates by representing the data or kernel matrix $A \in \mathbb{R}^{m \times n}$ as a weighted bipartite graph, where rows ($m$) represent samples and columns ($n$) features/kernels. Color refinement seeks a pair of partitions $(\mathcal{P}, \mathcal{Q})$ (rows, columns) that are *equitable*:
- For each sample-block $S \in \mathcal{P}$ and feature-block $T \in \mathcal{Q}$:
  - $\sum_{j \in T} A_{ij}$ is constant over $i \in S$
  - $\sum_{i \in S} A_{ij}$ is constant over $j \in T$

The algorithm iteratively refines blocks, initialized by grouping by label, feature statistics, or other coarse signatures, and splits along dimensional aggregates until the finest equitable partition is obtained. The process has time complexity $O(\mathrm{nnz}(A)(\log m+\log n))$ for sparse $A$ or $O(mn(\log m+\log n))$ when $A$ is dense.

## 3. Compression Mapping and Theoretical Properties

Partition matrices $\Pi_\mathcal{Q} \in \{0,1\}^{n \times q}$ encode feature blocks in $\mathcal{Q}$, with their row-scaled transposes $\Pi_\mathcal{Q}^{\mathrm{scaled}}$ averaging variables within each block. The *compression mapping* for the primal ERM is:
- $\tilde X = \Pi_\mathcal{P}^{\mathrm{scaled}} X \Pi_\mathcal{Q}$
- $\tilde y = \Pi_\mathcal{P}^{\mathrm{scaled}} y$
- $\tilde v = \Pi_\mathcal{P}^T v$
- Regularizer $R(\Pi_\mathcal{Q}w')$ in terms of compressed $w' \in \mathbb{R}^{|\mathcal{Q}|}$

**Main Losslessness Theorem:** If $(\mathcal{P}, \mathcal{Q})$ is a reduction coloring (i.e., the gradients of $F$ and constraints are constant on color-blocks), then:
- Any optimum $x$ in the original problem compresses to $x' = \Pi_\mathcal{Q}^{\mathrm{scaled}}x$ in the reduced problem.
- Any $x'$ optimal for the reduced problem lifts to $x = \Pi_\mathcal{Q} x'$ optimal for the original.

This construction strictly refines any partition from automorphism symmetry ("lifted inference"), guaranteeing at least as much (or more) compression. Proofs leverage convexity (averaging in blocks lowers $F$) and preservation of feasibility via partition structure.

## 4. Specialization to Models and Algorithmic Implementation

The framework admits efficient algorithms for several widely used convex models:

| Model                     | Equitability Conditions                | Additional Conditions                       |
|---------------------------|----------------------------------------|---------------------------------------------|
| Linear regression         | $(\mathcal{P}, \mathcal{Q})$ on $X$   | $(X^T y)_j$ is constant on each $T \in \mathcal{Q}$ |
| Binary logistic regression| $(\mathcal{P}, \mathcal{Q})$ on $X$   | $\sum_i v_i X_{ij} y_i$ const on $T$; $v_i$ const on $S$ |
| Multiclass logistic       | $(\mathcal{P}, \mathcal{Q})$ on $X$   | Replace $y_i$ term with $\mathbf{1}\{y_i=c\}$ counts |
| Elastic-net regression    | Same as linear                         | Convexity (\(\ell_1\) term) via Jensen argument |
| Kernel methods            | $(\mathcal{Q}, \mathcal{Q})$ on $K$   | $\sum_i v_i K_{ij} y_i$ const on $T$; $v_i$ const in block |

Algorithmic steps consist of a color-refinement loop over $(\mathcal{P}, \mathcal{Q})$, with model specialization via suitable signature choices to initiate partitions. The closed-form regression solution $w = (X^T X)^{-1} X^T y$ is preserved upon reduction, and similar properties hold for kernel and logistic models.

## 5. Empirical Evaluation and Compression Performance

Empirical evaluation on five binary classification datasets (Titanic, skin_nonskin, phishing, a7a, breast-cancer) demonstrates:

- Substantial reduction in sample and/or feature count without loss of accuracy:

| Dataset       | Samples: Pre→Post | Features: Pre→Post |
|---------------|------------------|--------------------|
| Titanic       | 1309 → 1147      | 378 → 367          |
| skin_nonskin  | 245057 → 51444   | unchanged          |
| phishing      | 11055 → 5849     | unchanged          |
| a7a           | 16100 → 13900    | 122 → 120          |
| breast-cancer | 683 → 675        | unchanged          |

- Cumulative runtime (reduction plus training) as a fraction of baseline training:

| Dataset       | Runtime (% of baseline) |
|---------------|------------------------|
| Titanic       | 16                     |
| skin_nonskin  | 28                     |
| phishing      | 21                     |
| a7a           | 91                     |
| breast-cancer | 90                     |

- Prediction and objective values of compressed solutions are within numerical solver tolerance of the full problem.

## 6. Key Conditions and Complexity Analysis

The framework's theoretical underpinnings are anchored by precise criteria:
- Gradient equivalence: $\nabla_{j_1} F(\hat x) = \nabla_{j_2} F(\hat x)$ for all $j_1, j_2$ in the same block
- Equitable sums: $\sum_{j\in T} X_{ij}$ constant over $i \in S$; $\sum_{i\in S} X_{ij}$ constant over $j \in T$
- Compression runtime: sparse $A$ yields $O(\mathrm{nnz}(A)(\log m+\log n))$
- Compressed problem dimension: $|\mathcal{P}|$ samples by $|\mathcal{Q}|$ features (or $|\mathcal{Q}|$ dual variables in kernel methods)

By computing the coarsest equitable partition via color refinement, lossless instance compression is achieved for a general class of differentiable convex ERM problems. The approach is strictly at least as strong as group-theoretic symmetry identification and yields measurable end-to-end computational benefits while maintaining exact optimality [2602.00437].

Source: https://www.emergentmind.com/topics/lossless-compression-framework-for-convex-erm