Deep Tangent Bundle Method

• Deep Tangent Bundle (DTB) method is a framework that uses the tangent spaces of manifolds to provide local linear approximations and preserve intrinsic geometric structures.
• It integrates differential geometry, deep learning, and numerical analysis to improve manifold learning, sampling accuracy, and high-dimensional PDE solutions.
• DTB methods enable efficient, mesh-free computations applicable in probabilistic numerics, manifold estimation, and nonlinear PDE solvers.

The Deep Tangent Bundle (DTB) method is a class of algorithms and mathematical frameworks that leverage the tangent bundle—the structure associating to each point of a manifold its tangent space—in order to endow learning, sampling, and numerical solution methods with local linear approximations that preserve, exploit, or estimate the manifold’s local geometry. In both deep learning and applied mathematics, DTB methods are distinguished by their explicit use of tangent bundles: for upsampling and sampling on manifolds, manifold estimation, and as the numerical backbone for solving high-dimensional partial differential equations (PDEs) by projecting target functions or PDE right-hand sides into a neural network's tangent space. This multifaceted concept unites geometric learning theory, differential geometry, and numerical PDEs.

1. Mathematical Foundations: Tangent Bundles and Higher-Order Extensions

The tangent bundle $TM$ of a manifold $M$ comprises all tangent vectors at all points of $M$, forming a smooth vector bundle when $M$ is endowed with a linear connection. For a smooth Banach manifold, the $k$‑order tangent bundle $T^k M$ consists of equivalence classes $[y,x]_k$ of curves $y$ at $x$ whose derivatives agree up to order $k$, with local coordinates mapping $[y,x]_k$ to $$(\varphi_a(x), (\varphi_a\circ y)'(0),\dots,(\varphi_a\circ y)^{(k)}(0))$$ (Suri, 2014).

A key structural property: $T^k M$ is always a smooth fiber bundle—its projection $\pi_k:T^k M \to M$ gives the bundle structure—but is a vector bundle if and only if $M$ has a linear connection; the local trivializations become linear via a lifted connection, enabling operations such as metric or Lagrangian lifting to $T^k M$. The infinite-order (deep) tangent bundle $T^\infty M$ arises as the projective limit of the sequence $\cdots\to T^k M\to T^{k-1} M\to\cdots\to TM$, with Fréchet model space, essential for handling objects depending on arbitrarily high derivatives—a situation directly encountered in the design of deep or layered tangent bundle models.

In tangent categories—the categorical abstraction of tangent bundles—differential bundles generalize vector bundles, capturing the tangent bundle concept abstractly for applications in differential geometry, algebraic geometry, and other settings (Cockett et al., 2016, Ching, 9 Jul 2024). Differential bundles are determined by a projection and zero section; under suitable pullbacks, all other structure (addition, vertical lift) is encoded in these, allowing for generalizations to higher categorical and algebraic frameworks relevant for advanced DTB implementations.

2. Tangent Bundle Manifold Learning and Tangent Bundle Learners

Tangent Bundle Manifold Learning (TBML) extends classical non-linear manifold learning. Instead of reconstructing only points of the manifold, TBML approaches require both the pointwise proximity (for each $X$ on the manifold, $X \approx g(h(X))$ for suitable embedding $h$ and reconstruction $g$) and tangent proximity ($L(X) \approx \operatorname{Span}(J_g(h(X)))$) (Bernstein et al., 2012). The method estimates local tangent spaces using weighted local PCA, aligns them via Grassmannian and Procrustes methods, and constructs embedding and reconstruction maps so that both locations and local differential structures (tangents) are preserved.

This foundational work shows—by precise lower bounds on reconstruction error in terms of misalignment between tangent spaces—that preserving local tangent information is critical for generalization and out-of-sample extrapolation. These structural principles are directly translatable into deep settings: for instance, a Deep Tangent Bundle method would penalize both pointwise and tangent misalignments using neural architectures and additional Jacobian-based loss terms.

Tangent bundle learners (TBLs) generalize this idea for manifold estimation. After learning local tangent spaces (e.g., by mixtures of probabilistic principal component analyzers, MoPPCA), the assignment function relates data points to tangent spaces. To obtain bounded manifold estimates, Faithful Neighborhood Estimation methods shrink the unbounded tangent approximations to regions where the tangent space locally fits the data, using convex hulls, density level sets, or intersection with high-density sets, and then reconstruct the manifold as the union of such faithful neighborhoods (Ramachandra et al., 2019).

3. Deep Tangent Bundle (DTB) Methods in Sampling and Inference

DTB methodology appears in sampling from densities supported on submanifolds. Given a density $p(\theta) \propto f(\alpha(\theta))$, where $\alpha$ maps a lower-dimensional parameter space $\Theta$ into a typically higher-dimensional ambient space, the DTB approach constructs mini-samples by projecting ambient-space Gaussians onto the tangent space of the manifold at sampled points and then pulling them back to parameter space via the pseudoinverse Jacobian (Chua, 2018).

Weighted importance sampling is used to produce high-resolution, computationally efficient approximations of $p(\theta)$, with local geometry (metric, curvature) controlling the sampling region. The core steps:

• For each base chain sample $\theta_i$, compute $J_i = D\alpha(\theta_i)$ (the Jacobian) spanning $T_{\alpha(\theta_i)}\mathcal{M}$.
• Sample $m$ points $\beta_{ij}$ from a local Gaussian, project: $$\beta_{ij}^\perp = \alpha(\theta_i) + P_i^\perp(\beta_{ij} - \alpha(\theta_i))$$ (with $P_i^\perp = J_i J_i^+$).
• Pull back: $$\theta_{ij} = \theta_i + J_i^+(\beta_{ij} - \alpha(\theta_i))$$.
• Assign importance weight $w_{ij}$ based on density comparisons and curvature corrections.

The result is a weighted empirical approximation that improves effective resolution and density estimation accuracy while leveraging only the derivatives of the mapping, thus reducing computational expense. The importance of local tangent bundle geometry is especially pronounced in manifolds with significant curvature or nonlinearity.

4. Deep Tangent Bundle Methods for High-Dimensional PDEs

The DTB method, in its most advanced neural application, provides a numerical framework for solving evolutionary PDEs in high dimensions by leveraging the tangent bundle of deep neural networks (Wu et al., 31 Aug 2025). The base idea is to approximate the spatial differential operator $F[u]$ in $\partial_t u = F[u]$ by projecting $F[u]$ into the tangent space of a DNN $f_\theta(z)$ over its parameters $\theta$. The separation of time discretization (performed by classical schemes such as Euler or trapezoidal) from the spatial approximation (conducted in the DNN’s tangent bundle) is a defining feature.

Mathematical Summary:

• For $f_\theta(z)$, define the tangent bundle at parameter value $\theta$ as $$B(f,\theta) = \operatorname{span}\{\partial f_\theta/\partial\theta_i\}$$.
• Obtain the optimal coefficient vector $\alpha$ by solving

$$\alpha = \operatorname{argmin}_{v\in\mathbb{R}^m} \int \|\left(\partial f_\theta(z)/\partial\theta\right)\cdot v - g(z)\|^2\,dz,$$

with projection and metric tensors

$$G(\theta) = \int \left(\frac{\partial f_\theta(z)}{\partial \theta}\right)^T\left(\frac{\partial f_\theta(z)}{\partial \theta}\right) dz,\quad P(\theta;g) = \int \left(\frac{\partial f_\theta(z)}{\partial \theta}\right)^T g(z) dz.$$

• Solve for $\alpha$ via the pseudoinverse: $\alpha = G(\theta)^\dagger P(\theta;g)$.

The time-stepping in the evolutionary PDE is replaced by

$$u^{k+1}(z) = u^k(z) + h \cdot K_\theta^k[F[u^k]](z)$$

where $K_\theta^k$ is the tangent bundle projection operator.

The parameter update strategies (fixed, explicit step, or periodic reset via $L^2$ minimization) manage tangent bundle conditioning, while all approximations reduce to linear problems. This approach avoids nonconvex optimization, mitigates stiffness, and is mesh-free—crucial for scalability in high dimensions. Experimental results include successful solutions for 2D and 5D PDEs (heat, Allen–Cahn, Wasserstein flows) with accuracy competitive with classical numerical methods.

5. Differential Bundles and Abstract Tangent Categories

Theoretical generalizations of tangent bundles—differential bundles in tangent categories—provide the categorical infrastructure underlying DTB principles (Cockett et al., 2016, Ching, 9 Jul 2024). In these settings, the tangent functor $T$ abstracts the usual differential geometrical structure, and a differential bundle is determined by its projection and zero section, with vertical lift and addition induced from these. This conceptual viewpoint ensures that the DTB constructions generalize naturally to more abstract or higher-categorical contexts.

Linear morphisms between such bundles (i.e., those preserving the lift) are necessarily additive, generalizing classical linear maps between vector bundles, reinforcing that tangent bundle structures and their morphisms encode the essential linear and additive properties for differential geometric reasoning in both classical and categorical settings.

6. Applications and Extensions

DTB methods have been deployed in several domains:

• Sampling and inference on manifold-restricted distributions: derivative-based upsampling in gravitational-wave analysis and probabilistic numerics (Chua, 2018).
• Numerical solutions of high-dimensional PDEs: mesh-free solvers for nonlinear and diffusion-dominated problems (Wu et al., 31 Aug 2025).
• Manifold learning and estimation: geometric embeddings that preserve tangent information, essential for structure-preserving feature learning (Bernstein et al., 2012), and manifold reconstruction algorithms using tangent bundle learners (Ramachandra et al., 2019).
• Theoretical abstractions: construction and recognition of tangent bundles in tangent categories and limit-based presentations for higher and infinite order tangent bundles (Suri, 2014, Ching, 9 Jul 2024).

Potential extensions include incorporating higher-order tangent bundles for more accurate local approximations; leveraging categorical tangent structures for new deep learning architectures; and exploring further connections with geometric deep learning, physics-informed networks, and categorical dynamics.

7. Significance and Outlook

The Deep Tangent Bundle method unifies a broad class of geometric and numerical techniques wherein the tangent bundle structure is explicitly exploited to encode local linearity, enable efficient computation, and improve generalization by ensuring differential structure consistency. By embracing tangent bundles as primary objects—rather than passive byproducts—DTB methodologies provide robust, theoretically grounded, and scalable tools for manifold-based data analysis, high-dimensional Bayesian inference, and mesh-free solution of PDEs, with ongoing research extending these ideas deep into differential geometry, category theory, and machine learning (Bernstein et al., 2012, Suri, 2014, Chua, 2018, Ramachandra et al., 2019, Ching, 9 Jul 2024, Wu et al., 31 Aug 2025).