---
title: Constraint-Aligned Loss with Manifold Lifting
url: https://www.emergentmind.com/topics/constraint-aligned-loss-with-manifold-lifting-caml
type: topic
---

# Constraint-Aligned Loss with Manifold Lifting

Searching arXiv for the specified CAML-related papers and closely related work to ground the article in current literature.
Constraint-Aligned Loss with Manifold Lifting (CAML) denotes a loss-construction strategy in which an optimization problem is reformulated by introducing an auxiliary lifted variable so that otherwise conflicting constraints become more compatible during training. In the PINN setting, CAML was proposed to mitigate gradient pathology by reformulating zeroth-order terms into aligned constraints, lifting the network onto an affine manifold through an additive scalar offset, and combining this with a time-dependent delay factor on the PDE-residual term [2605.25001]. In a separate, geometrically related usage, the same label has been applied to the Log-Euclidean extension of Deep CORAL for unsupervised domain adaptation, where covariance matrices are lifted from the manifold of symmetric positive definite matrices to the vector space of symmetric matrices via the matrix logarithm [1705.08180]. The shared theme is manifold-aware loss design, but the two formulations address different optimization pathologies and belong to different application domains.

## 1. Conceptual scope and problem setting

In the 2026 PINN formulation, CAML is introduced in response to a specific training failure mode: gradients from PDE residuals and boundary constraints oppose each other, trapping the model in local minima [2605.25001]. The standard PINN objective combines a residual loss and a boundary-condition loss,
$$
L(\theta)=w_\mathrm{res}L_\mathrm{res}+w_\mathrm{bc}L_\mathrm{bc},
$$
with the residual defined over interior collocation points and the boundary term defined over boundary collocation points. The paper states that, empirically and theoretically, the PDE-residual term alone admits a connected flat valley of zero-residual solutions in function space, which maps to a highly elongated, distorted valley in parameter space, while the boundary-condition gradients typically point across this valley and conflict with the residual gradients. This conflict is summarized by
$$
\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0
$$
over large portions of training [2605.25001].

The 2017 domain-adaptation formulation addresses a different problem: the sensitivity of machine-learning systems to domain shift under unsupervised domain adaptation. There the loss aligns second-order statistics of source and target hidden features, but the key observation is that covariance matrices are symmetric positive definite and therefore lie on a Riemannian manifold of non-positive curvature rather than in a Euclidean vector space [1705.08180]. The data explicitly describes this as a manifold-lifting procedure implemented as a Log-Euclidean extension of Deep CORAL.

A plausible implication is that CAML is best understood not as a single canonical algorithm across all of machine learning, but as a family resemblance among methods that alter the geometry of the objective so that optimization follows a better-conditioned path.

## 2. PINN formulation: aligned constraints and lifted parameterization

The PINN version of CAML is formulated for problems of the form
$$
N[u](x)=f(x), \qquad x\in\Omega,
$$
subject to mixed boundary conditions
$$
a u + B\nabla u\cdot n = g \quad \text{on } \Gamma
$$
[2605.25001]. The trial space is a Sobolev space $F(\Omega)$, for example $H^2(\Omega)$, and the network approximation is $u_\theta(x)$. The PDE operator is written as $N[u]:F(\Omega)\to L^2(\Omega)$, and the boundary operator $B[u]:F(\Omega)\to L^2(\Gamma)$ collects Dirichlet, Neumann, and Robin terms.

The aligned-constraint construction begins from the observation that every zeroth-order term $a\cdot u$ can be shifted by a constant $c\in\mathbb{R}$ without affecting derivative terms. CAML therefore introduces $c$ only into those zeroth-order terms:
$$
r_i(c)=N[u_\theta](x_i)\big|_{u_\theta\to u_\theta+c}-f(x_i),
$$
$$
b_j(c)=a\bigl(u_\theta(\tilde x_j)+c\bigr)+B\nabla u_\theta(\tilde x_j)\cdot n_j-g(\tilde x_j).
$$
The corresponding aligned losses are
$$
L_\mathrm{res}^{\rm align}(\theta,c),\qquad L_\mathrm{bc}^{\rm align}(\theta,c),
$$
and at each iteration CAML selects
$$
c^*(\theta)=\arg\min_{c\in\mathbb{R}}\;w_\mathrm{res}L_\mathrm{res}^{\rm align}(\theta,c)+w_\mathrm{bc}L_\mathrm{bc}^{\rm align}(\theta,c).
$$
For the linear case, where $a$ and $B$ are constant, the data states that this is a one-dimensional quadratic with the closed-form solution
$$
c^*=-\frac{w_\mathrm{res}\sum_i a\,r_i(0)+w_\mathrm{bc}\sum_j a\,b_j(0)}
{w_\mathrm{res}\sum_i a^2+w_\mathrm{bc}\sum_j a^2},
$$
whereas in the nonlinear case a few Newton steps or a simple line search are used [2605.25001].

After eliminating $c$, the final loss is
$$
L_\mathrm{CAML}(\theta;t)=w_\mathrm{res}\,\alpha(t)\,L_\mathrm{res}^{\rm align}\bigl(\theta,c^*(\theta)\bigr)+w_\mathrm{bc}\,L_\mathrm{bc}^{\rm align}\bigl(\theta,c^*(\theta)\bigr).
$$
The paper specifies that $c^*(\theta)$ is recomputed at each step but treated as constant during back-propagation [2605.25001].

## 3. Manifold lifting as geometric realignment

In the PINN formulation, manifold lifting is the transformation
$$
\widetilde u_\theta(x,c)=u_\theta(x)+c,
$$
which expands the expressible solution family to
$$
\{\widetilde u_\theta(x,c):\theta\in\mathbb{R}^P,\;c\in\mathbb{R}\}.
$$
The paper characterizes this as an affine extension of the original parameter-to-function map [2605.25001]. Its geometric motivation is that the PDE-residual zero set
$$
S_\mathrm{res}=\{u:N[u]=f\}
$$
is a flat, connected manifold because of additive invariance, and by adding $c$ the boundary-constraint set is realigned so that the overlap between PDE-admissible and boundary-admissible sets becomes enlarged rather than effectively point-like. The stated consequence is that this “thickens” the overlap and reduces the distance the optimizer must traverse along the narrow PDE valley [2605.25001].

The theoretical claims reported in the data are specific. Theorem 4.1 shows that the PDE-residual alone admits a continuum of minimizers $S_\mathrm{res}$ with at least one flat direction. Lemma 4.2 shows that standard boundary-condition gradients point across this valley and therefore conflict with residual gradients. Proposition C.1, labeled Sublevel-Set Enlargement, proves the pointwise inequality
$$
L_\mathrm{CAML}(\theta)\le L_\mathrm{std}(\theta),
$$
and Proposition C.3 states that the intersection of PDE- and boundary-admissible sets becomes thicker [2605.25001].

This suggests that in the PINN setting the phrase “manifold lifting” is not merely heuristic. It refers to a deliberate change in the geometry of the feasible set, with the auxiliary scalar acting as an additional coordinate that relaxes incompatibilities induced by the original parameterization.

## 4. Delay factor and optimization dynamics

CAML supplements aligned constraints with a delay factor on the residual term,
$$
\alpha(t)=
\begin{cases}
0,&t<t_d,\\[4pt]
\dfrac{t-t_d}{T_r},&t_d\le t<t_d+T_r,\\[6pt]
1,&t\ge t_d+T_r.
\end{cases}
$$
The data states that this suppresses the residual term for the first few hundred steps and then ramps it up [2605.25001]. The stated purpose is to avoid an early, rapid descent into a pathological region of the PDE valley: for $t<t_d$ the model focuses on satisfying boundary conditions first, and only gradually engages the PDE residual.

The algorithmic sketch given in the data consists of four operations per iteration: sampling interior and boundary points, computing residuals in a forward pass, solving the one-dimensional subproblem for $c^*$, and then updating $\theta$ with Adam using the aligned loss [2605.25001]. The paper attributes a distinct role to the two components: aligned constraints reduce directional conflicts by realigning the boundary manifold, while the delay factor smooths the trajectory into the PDE valley and avoids entrapment in high-curvature subregions.

A common misconception would be to treat the delay factor as a generic adaptive weighting rule. The data does not describe it that way. Rather, it is a time-dependent schedule designed to prevent the optimizer from plunging too early into the residual valley. Likewise, the role of the auxiliary constant is not to impose a hard constraint, but to solve a per-iteration subproblem that lowers the combined aligned loss [2605.25001].

## 5. Empirical evaluation on PDE benchmarks

The reported evaluation covers four PDE classes of increasing complexity: Heat with mixed Dirichlet/Neumann boundary conditions, Poisson with Dirichlet plus strong nonlinear boundary conditions, Navier–Stokes with nonlinear convection and incompressibility, and Helmholtz with variable-coefficient elliptic structure on complex geometry [2605.25001]. Architectures include MLP, PirateNets, and PINNsFormer. Baselines are PINN, LO-PINN, SA-PINN, BRDR, and DB-PINN.

The metrics are defined explicitly in the data: **Stp** is the number of iterations until relative $L^2$-error is at most $10^{-3}$, **$L_2$** is the final relative $L^2$-error, and **$\cos(\phi)$** is the fraction of steps with $\langle \nabla L_\mathrm{res},\nabla L_\mathrm{bc}\rangle>0$ [2605.25001].

| Method | Heat: Stp / $L_2$ | Poisson: Stp / $L_2$ |
|---|---:|---:|
| PINN | $10{,}127\pm2{,}839$ / $7.52\times10^{-3}$ | $8{,}005\pm1{,}191$ / $8.02\times10^{-3}$ |
| CAML (ours) | $\mathbf{1{,}577\pm130}$ / $\mathbf{1.16\times10^{-3}}$ | $3{,}868\pm1{,}932$ / $\mathbf{5.00\times10^{-3}}$ |

| Method | $\cos(\phi)_\mathrm{Heat}$ | $\cos(\phi)_\mathrm{Pois}$ |
|---|---:|---:|
| PINN | $5.9\%$ | $3.8\%$ |
| CAML | $39.2\%$ | $52.8\%$ |

The ablation data further separates aligned constraints (AC), delay-residual (DR), and their combination:

| Config | Heat: Stp / $\cos(\phi)$ | Poisson: Stp / $\cos(\phi)$ |
|---|---:|---:|
| PINN | $10{,}127$ / $5.9\%$ | $8{,}005$ / $3.8\%$ |
| AC only | $1{,}491$ / $36.1\%$ | — / $45.2\%$ |
| DR only | $8{,}367$ / $4.9\%$ | $9{,}648$ / $14.6\%$ |
| AC+DR = CAML | $1{,}577$ / $39.2\%$ | $3{,}868$ / $52.8\%$ |

Optimizer compatibility on Heat with an MLP backbone is also reported:

| Optimizer | Stp | $L_2$ |
|---|---:|---:|
| Adam | $1{,}577$ | $1.16\times10^{-3}$ |
| L-BFGS | $55$ | $6.93\times10^{-4}$ |
| DCGD | $991$ | $1.15\times10^{-3}$ |
| ConFIG | $738$ | $1.13\times10^{-3}$ |

The key observations listed in the data are that CAML reduces required iterations by an order of magnitude, lowers final error, raises gradient alignment from roughly $5\%$ to $40$–$55\%$, derives most of its gain from aligned constraints on simple boundary conditions, and relies on delay-residual for strongly nonlinear boundary conditions. The same source also states that CAML is compatible with both first-order and second-order optimizers [2605.25001].

## 6. Relation to Log-Euclidean covariance alignment

A separate use of the same label appears in the detailed exposition attached to “Correlation Alignment by Riemannian Metric for Domain Adaptation” [1705.08180]. There, the object being lifted is not a PINN solution function but an empirical covariance matrix
$$
C=\frac{1}{L-1}X^\top X,
$$
constructed from zero-mean hidden-layer activations $X\in\mathbb{R}^{L\times d}$. Because $C\in \mathrm{Sym}^{++}(d)$, it lies on the Riemannian manifold of symmetric positive definite matrices rather than in a Euclidean subspace. The lifting is the matrix logarithm
$$
\log:\mathrm{Sym}^{++}(d)\to \mathrm{Sym}(d),
$$
implemented by eigendecomposition,
$$
C=U\,\mathrm{diag}(\lambda_1,\dots,\lambda_d)\,U^\top,\qquad
\log C=U\,\mathrm{diag}(\log\lambda_1,\dots,\log\lambda_d)\,U^\top.
$$
The data describes this mapping as a diffeomorphism onto the vector space of symmetric matrices, flattening the curved manifold so that distances in the log domain approximate true geodesic distances [1705.08180].

The loss term is
$$
L_\mathrm{CAML}=\frac{1}{4d^2}\|\log C_S-\log C_T\|_F^2,
$$
which is added to the usual source-domain classification loss:
$$
L=L_\mathrm{CLASS}(X_S,y_S)+\alpha\,L_\mathrm{CAML}(C_S,C_T).
$$
The formulation explicitly contrasts the Log-Euclidean distance
$$
d_{\log E}(C_S,C_T)=\|\log C_S-\log C_T\|_F
$$
with the affine-invariant geodesic distance
$$
d_R(C_S,C_T)=\|\log(C_S^{-1/2}C_TC_S^{-1/2})\|_F,
$$
and states the inequality
$$
d_R(C_S,C_T)\le d_{\log E}(C_S,C_T)\le \sqrt d\cdot d_R(C_S,C_T).
$$
The motivation is that Euclidean covariance alignment ignores curvature and is not invariant under congruence transforms, whereas the Log-Euclidean metric is a true Riemannian metric, cheaper to compute than the affine-invariant alternative, and differentiable everywhere on $\mathrm{Sym}^{++}(d)$ [1705.08180].

The training pipeline reported in the data computes source and target mini-batch covariances with regularization,
$$
C_S=\frac{1}{L_S-1}X_S^\top X_S+\gamma I,\qquad
C_T=\frac{1}{L_T-1}X_T^\top X_T+\gamma I,
$$
with $\gamma\approx10^{-5}$, then eigendecomposes, applies the matrix logarithm, evaluates the loss, and relies on automatic differentiation through eigendecomposition and the matrix logarithm [1705.08180].

## 7. Reported results, extensions, and interpretive boundaries

For the domain-adaptation variant, the reported experiments use AlexNet pre-trained on ImageNet, with the final fc8 re-initialized to output the $31$ classes of the Office dataset, batch size $128$, base learning rate $10^{-3}$ with exponential decay, covariance regularizer $\gamma=10^{-5}$, and loss weights $\lambda\in\{0.05,0.1,1.0\}$ for Deep CORAL and $\alpha\in\{0.1,\dots,10\}$ for the Log-Euclidean variant, typically an order of magnitude larger than $\lambda$ [1705.08180]. The reported accuracies on the six standard unsupervised Office shifts are as follows.

| Method | Average accuracy |
|---|---:|
| AlexNet (no DA) | $58.6$ |
| Deep CORAL ($\lambda$) | $60.5$ $(+1.9)$ |
| CAML ($\alpha$) | $61.4$ $(+2.8)$ |

The detailed splitwise results are also given: AlexNet achieves $57.8, 60.2, 40.0, 95.2, 40.0, 97.8$ on $A\to D$, $A\to W$, $D\to A$, $D\to W$, $W\to A$, and $W\to D$ respectively; Deep CORAL reports $58.9, 65.9, 40.7, 95.6, 41.6, 98.0$; and the Log-Euclidean variant reports $62.0, 68.5, 40.6, 95.3, 40.6, 98.7$ [1705.08180]. The accompanying observations are that aligning second-order statistics yields a clear gain over vanilla AlexNet, that replacing the Euclidean CORAL loss with the Log-Euclidean loss gives an additional improvement of approximately $1\%$, and that Euclidean CORAL distance behaves more noisily during training.

The extension directions named in the PINN paper include applying CAML to highly complex PINN problems and open-sourcing code at the cited repository [2605.25001]. The extension directions named in the domain-adaptation exposition include replacing the Log-Euclidean distance with the full affine-invariant geodesic, aligning covariance operators in infinite-dimensional RKHS via Log-Hilbert-Schmidt metrics, aligning multiple intermediate layers such as conv5, fc6, and fc7, transposing the idea to other modalities including audio spectrogram covariances and text-feature covariances, applying it to deeper architectures such as ResNet and DenseNet, and mixing Riemannian alignment losses with adversarial domain-confusion losses [1705.08180].

A final interpretive boundary is important. In the PINN literature, CAML is the explicit name of a method that combines aligned constraints, manifold lifting, and delay-residual scheduling [2605.25001]. In the domain-adaptation literature, the same acronym is an editorial label attached to a Log-Euclidean Deep CORAL variant rather than the original paper title [1705.08180]. The common structure is manifold-aware loss engineering, but the mathematical objects being lifted, the pathologies being addressed, and the empirical benchmarks are distinct.

Source: https://www.emergentmind.com/topics/constraint-aligned-loss-with-manifold-lifting-caml