Constraint-Aligned Loss with Manifold Lifting

• CAML introduces an auxiliary variable to reformulate conflicting PDE residual and boundary constraints, resulting in improved gradient alignment.
• The method employs manifold lifting to realign the parameter space, which thicken the overlap between PDE-admissible and boundary-admissible sets.
• CAML incorporates a time-dependent delay on the residual term to stabilize training dynamics, significantly reducing iterations and error across benchmarks.

Searching arXiv for the specified CAML-related papers and closely related work to ground the article in current literature. Constraint-Aligned Loss with Manifold Lifting (CAML) denotes a loss-construction strategy in which an optimization problem is reformulated by introducing an auxiliary lifted variable so that otherwise conflicting constraints become more compatible during training. In the PINN setting, CAML was proposed to mitigate gradient pathology by reformulating zeroth-order terms into aligned constraints, lifting the network onto an affine manifold through an additive scalar offset, and combining this with a time-dependent delay factor on the PDE-residual term (Luo et al., 24 May 2026). In a separate, geometrically related usage, the same label has been applied to the Log-Euclidean extension of Deep CORAL for unsupervised domain adaptation, where covariance matrices are lifted from the manifold of symmetric positive definite matrices to the vector space of symmetric matrices via the matrix logarithm (Morerio et al., 2017). The shared theme is manifold-aware loss design, but the two formulations address different optimization pathologies and belong to different application domains.

1. Conceptual scope and problem setting

In the 2026 PINN formulation, CAML is introduced in response to a specific training failure mode: gradients from PDE residuals and boundary constraints oppose each other, trapping the model in local minima (Luo et al., 24 May 2026). The standard PINN objective combines a residual loss and a boundary-condition loss,

$$L(\theta)=w_\mathrm{res}L_\mathrm{res}+w_\mathrm{bc}L_\mathrm{bc},$$

with the residual defined over interior collocation points and the boundary term defined over boundary collocation points. The paper states that, empirically and theoretically, the PDE-residual term alone admits a connected flat valley of zero-residual solutions in function space, which maps to a highly elongated, distorted valley in parameter space, while the boundary-condition gradients typically point across this valley and conflict with the residual gradients. This conflict is summarized by

$$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$

over large portions of training (Luo et al., 24 May 2026).

The 2017 domain-adaptation formulation addresses a different problem: the sensitivity of machine-learning systems to domain shift under unsupervised domain adaptation. There the loss aligns second-order statistics of source and target hidden features, but the key observation is that covariance matrices are symmetric positive definite and therefore lie on a Riemannian manifold of non-positive curvature rather than in a Euclidean vector space (Morerio et al., 2017). The data explicitly describes this as a manifold-lifting procedure implemented as a Log-Euclidean extension of Deep CORAL.

A plausible implication is that CAML is best understood not as a single canonical algorithm across all of machine learning, but as a family resemblance among methods that alter the geometry of the objective so that optimization follows a better-conditioned path.

2. PINN formulation: aligned constraints and lifted parameterization

The PINN version of CAML is formulated for problems of the form

$N[u](x)=f(x), \qquad x\in\Omega,$

subject to mixed boundary conditions

$a u + B\nabla u\cdot n = g \quad \text{on } \Gamma$

(Luo et al., 24 May 2026). The trial space is a Sobolev space $F(\Omega)$, for example $H^2(\Omega)$, and the network approximation is $u_\theta(x)$. The PDE operator is written as $N[u]:F(\Omega)\to L^2(\Omega)$, and the boundary operator $B[u]:F(\Omega)\to L^2(\Gamma)$ collects Dirichlet, Neumann, and Robin terms.

The aligned-constraint construction begins from the observation that every zeroth-order term $a\cdot u$ can be shifted by a constant $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$0 without affecting derivative terms. CAML therefore introduces $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$1 only into those zeroth-order terms:

$$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$2

$$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$3

The corresponding aligned losses are

$$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$4

and at each iteration CAML selects

$$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$5

For the linear case, where $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$6 and $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$7 are constant, the data states that this is a one-dimensional quadratic with the closed-form solution

$$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$8

whereas in the nonlinear case a few Newton steps or a simple line search are used (Luo et al., 24 May 2026).

After eliminating $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$9, the final loss is

$N[u](x)=f(x), \qquad x\in\Omega,$0

The paper specifies that $N[u](x)=f(x), \qquad x\in\Omega,$1 is recomputed at each step but treated as constant during back-propagation (Luo et al., 24 May 2026).

3. Manifold lifting as geometric realignment

In the PINN formulation, manifold lifting is the transformation

$N[u](x)=f(x), \qquad x\in\Omega,$2

which expands the expressible solution family to

$N[u](x)=f(x), \qquad x\in\Omega,$3

The paper characterizes this as an affine extension of the original parameter-to-function map (Luo et al., 24 May 2026). Its geometric motivation is that the PDE-residual zero set

$N[u](x)=f(x), \qquad x\in\Omega,$4

is a flat, connected manifold because of additive invariance, and by adding $N[u](x)=f(x), \qquad x\in\Omega,$5 the boundary-constraint set is realigned so that the overlap between PDE-admissible and boundary-admissible sets becomes enlarged rather than effectively point-like. The stated consequence is that this “thickens” the overlap and reduces the distance the optimizer must traverse along the narrow PDE valley (Luo et al., 24 May 2026).

The theoretical claims reported in the data are specific. Theorem 4.1 shows that the PDE-residual alone admits a continuum of minimizers $N[u](x)=f(x), \qquad x\in\Omega,$6 with at least one flat direction. Lemma 4.2 shows that standard boundary-condition gradients point across this valley and therefore conflict with residual gradients. Proposition C.1, labeled Sublevel-Set Enlargement, proves the pointwise inequality

$N[u](x)=f(x), \qquad x\in\Omega,$7

and Proposition C.3 states that the intersection of PDE- and boundary-admissible sets becomes thicker (Luo et al., 24 May 2026).

This suggests that in the PINN setting the phrase “manifold lifting” is not merely heuristic. It refers to a deliberate change in the geometry of the feasible set, with the auxiliary scalar acting as an additional coordinate that relaxes incompatibilities induced by the original parameterization.

4. Delay factor and optimization dynamics

CAML supplements aligned constraints with a delay factor on the residual term,

$N[u](x)=f(x), \qquad x\in\Omega,$8

The data states that this suppresses the residual term for the first few hundred steps and then ramps it up (Luo et al., 24 May 2026). The stated purpose is to avoid an early, rapid descent into a pathological region of the PDE valley: for $N[u](x)=f(x), \qquad x\in\Omega,$9 the model focuses on satisfying boundary conditions first, and only gradually engages the PDE residual.

The algorithmic sketch given in the data consists of four operations per iteration: sampling interior and boundary points, computing residuals in a forward pass, solving the one-dimensional subproblem for $a u + B\nabla u\cdot n = g \quad \text{on } \Gamma$0, and then updating $a u + B\nabla u\cdot n = g \quad \text{on } \Gamma$1 with Adam using the aligned loss (Luo et al., 24 May 2026). The paper attributes a distinct role to the two components: aligned constraints reduce directional conflicts by realigning the boundary manifold, while the delay factor smooths the trajectory into the PDE valley and avoids entrapment in high-curvature subregions.

A common misconception would be to treat the delay factor as a generic adaptive weighting rule. The data does not describe it that way. Rather, it is a time-dependent schedule designed to prevent the optimizer from plunging too early into the residual valley. Likewise, the role of the auxiliary constant is not to impose a hard constraint, but to solve a per-iteration subproblem that lowers the combined aligned loss (Luo et al., 24 May 2026).

5. Empirical evaluation on PDE benchmarks

The reported evaluation covers four PDE classes of increasing complexity: Heat with mixed Dirichlet/Neumann boundary conditions, Poisson with Dirichlet plus strong nonlinear boundary conditions, Navier–Stokes with nonlinear convection and incompressibility, and Helmholtz with variable-coefficient elliptic structure on complex geometry (Luo et al., 24 May 2026). Architectures include MLP, PirateNets, and PINNsFormer. Baselines are PINN, LO-PINN, SA-PINN, BRDR, and DB-PINN.

The metrics are defined explicitly in the data: Stp is the number of iterations until relative $a u + B\nabla u\cdot n = g \quad \text{on } \Gamma$2-error is at most $a u + B\nabla u\cdot n = g \quad \text{on } \Gamma$3, $a u + B\nabla u\cdot n = g \quad \text{on } \Gamma$4 is the final relative $a u + B\nabla u\cdot n = g \quad \text{on } \Gamma$5-error, and $a u + B\nabla u\cdot n = g \quad \text{on } \Gamma$6 is the fraction of steps with $a u + B\nabla u\cdot n = g \quad \text{on } \Gamma$7 (Luo et al., 24 May 2026).

Method	Heat: Stp / $a u + B\nabla u\cdot n = g \quad \text{on } \Gamma$8	Poisson: Stp / $a u + B\nabla u\cdot n = g \quad \text{on } \Gamma$9
PINN	$F(\Omega)$0 / $F(\Omega)$1	$F(\Omega)$2 / $F(\Omega)$3
CAML (ours)	$F(\Omega)$4 / $F(\Omega)$5	$F(\Omega)$6 / $F(\Omega)$7

Method	$F(\Omega)$8	$F(\Omega)$9
PINN	$H^2(\Omega)$0	$H^2(\Omega)$1
CAML	$H^2(\Omega)$2	$H^2(\Omega)$3

The ablation data further separates aligned constraints (AC), delay-residual (DR), and their combination:

Config	Heat: Stp / $H^2(\Omega)$4	Poisson: Stp / $H^2(\Omega)$5
PINN	$H^2(\Omega)$6 / $H^2(\Omega)$7	$H^2(\Omega)$8 / $H^2(\Omega)$9
AC only	$u_\theta(x)$0 / $u_\theta(x)$1	— / $u_\theta(x)$2
DR only	$u_\theta(x)$3 / $u_\theta(x)$4	$u_\theta(x)$5 / $u_\theta(x)$6
AC+DR = CAML	$u_\theta(x)$7 / $u_\theta(x)$8	$u_\theta(x)$9 / $N[u]:F(\Omega)\to L^2(\Omega)$0

Optimizer compatibility on Heat with an MLP backbone is also reported:

Optimizer	Stp	$N[u]:F(\Omega)\to L^2(\Omega)$1
Adam	$N[u]:F(\Omega)\to L^2(\Omega)$2	$N[u]:F(\Omega)\to L^2(\Omega)$3
L-BFGS	$N[u]:F(\Omega)\to L^2(\Omega)$4	$N[u]:F(\Omega)\to L^2(\Omega)$5
DCGD	$N[u]:F(\Omega)\to L^2(\Omega)$6	$N[u]:F(\Omega)\to L^2(\Omega)$7
ConFIG	$N[u]:F(\Omega)\to L^2(\Omega)$8	$N[u]:F(\Omega)\to L^2(\Omega)$9

The key observations listed in the data are that CAML reduces required iterations by an order of magnitude, lowers final error, raises gradient alignment from roughly $B[u]:F(\Omega)\to L^2(\Gamma)$0 to $B[u]:F(\Omega)\to L^2(\Gamma)$1–$B[u]:F(\Omega)\to L^2(\Gamma)$2, derives most of its gain from aligned constraints on simple boundary conditions, and relies on delay-residual for strongly nonlinear boundary conditions. The same source also states that CAML is compatible with both first-order and second-order optimizers (Luo et al., 24 May 2026).

6. Relation to Log-Euclidean covariance alignment

A separate use of the same label appears in the detailed exposition attached to “Correlation Alignment by Riemannian Metric for Domain Adaptation” (Morerio et al., 2017). There, the object being lifted is not a PINN solution function but an empirical covariance matrix

$B[u]:F(\Omega)\to L^2(\Gamma)$3

constructed from zero-mean hidden-layer activations $B[u]:F(\Omega)\to L^2(\Gamma)$4. Because $B[u]:F(\Omega)\to L^2(\Gamma)$5, it lies on the Riemannian manifold of symmetric positive definite matrices rather than in a Euclidean subspace. The lifting is the matrix logarithm

$B[u]:F(\Omega)\to L^2(\Gamma)$6

implemented by eigendecomposition,

$B[u]:F(\Omega)\to L^2(\Gamma)$7

The data describes this mapping as a diffeomorphism onto the vector space of symmetric matrices, flattening the curved manifold so that distances in the log domain approximate true geodesic distances (Morerio et al., 2017).

The loss term is

$B[u]:F(\Omega)\to L^2(\Gamma)$8

which is added to the usual source-domain classification loss:

$B[u]:F(\Omega)\to L^2(\Gamma)$9

The formulation explicitly contrasts the Log-Euclidean distance

$a\cdot u$0

with the affine-invariant geodesic distance

$a\cdot u$1

and states the inequality

$a\cdot u$2

The motivation is that Euclidean covariance alignment ignores curvature and is not invariant under congruence transforms, whereas the Log-Euclidean metric is a true Riemannian metric, cheaper to compute than the affine-invariant alternative, and differentiable everywhere on $a\cdot u$3 (Morerio et al., 2017).

The training pipeline reported in the data computes source and target mini-batch covariances with regularization,

$a\cdot u$4

with $a\cdot u$5, then eigendecomposes, applies the matrix logarithm, evaluates the loss, and relies on automatic differentiation through eigendecomposition and the matrix logarithm (Morerio et al., 2017).

7. Reported results, extensions, and interpretive boundaries

For the domain-adaptation variant, the reported experiments use AlexNet pre-trained on ImageNet, with the final fc8 re-initialized to output the $a\cdot u$6 classes of the Office dataset, batch size $a\cdot u$7, base learning rate $a\cdot u$8 with exponential decay, covariance regularizer $a\cdot u$9, and loss weights $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$00 for Deep CORAL and $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$01 for the Log-Euclidean variant, typically an order of magnitude larger than $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$02 (Morerio et al., 2017). The reported accuracies on the six standard unsupervised Office shifts are as follows.

Method	Average accuracy
AlexNet (no DA)	$$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$03
Deep CORAL ($$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$04)	$$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$05 $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$06
CAML ($$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$07)	$$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$08 $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$09

The detailed splitwise results are also given: AlexNet achieves $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$10 on $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$11, $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$12, $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$13, $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$14, $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$15, and $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$16 respectively; Deep CORAL reports $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$17; and the Log-Euclidean variant reports $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$18 (Morerio et al., 2017). The accompanying observations are that aligning second-order statistics yields a clear gain over vanilla AlexNet, that replacing the Euclidean CORAL loss with the Log-Euclidean loss gives an additional improvement of approximately $$\langle \nabla_\theta L_\mathrm{res},\nabla_\theta L_\mathrm{bc}\rangle<0$$19, and that Euclidean CORAL distance behaves more noisily during training.

The extension directions named in the PINN paper include applying CAML to highly complex PINN problems and open-sourcing code at the cited repository (Luo et al., 24 May 2026). The extension directions named in the domain-adaptation exposition include replacing the Log-Euclidean distance with the full affine-invariant geodesic, aligning covariance operators in infinite-dimensional RKHS via Log-Hilbert-Schmidt metrics, aligning multiple intermediate layers such as conv5, fc6, and fc7, transposing the idea to other modalities including audio spectrogram covariances and text-feature covariances, applying it to deeper architectures such as ResNet and DenseNet, and mixing Riemannian alignment losses with adversarial domain-confusion losses (Morerio et al., 2017).

A final interpretive boundary is important. In the PINN literature, CAML is the explicit name of a method that combines aligned constraints, manifold lifting, and delay-residual scheduling (Luo et al., 24 May 2026). In the domain-adaptation literature, the same acronym is an editorial label attached to a Log-Euclidean Deep CORAL variant rather than the original paper title (Morerio et al., 2017). The common structure is manifold-aware loss engineering, but the mathematical objects being lifted, the pathologies being addressed, and the empirical benchmarks are distinct.