Laplace Gating Functions

• Laplace gating functions are defined via time-power gains in the Laplace domain and L¹-norm based selection in neural mixtures, offering precise control over signal emphasis.
• In waveform inversion, multiplying by tⁿ translates into s-derivative operations that enable multiscale sensitivity and robust model updates.
• Within HMoE architectures, Laplace gating simplifies expert assignment by removing parameter entanglement, leading to faster convergence and improved specialization.

Laplace gating functions are a family of gating mechanisms with foundations in two distinct areas: inverse problems (notably Laplace-domain waveform inversion) and neural network mixtures of experts. They are defined by the use of shift-equivariant, norm-based or time-power weighting schemes that offer precise control over signal emphasis or expert selection, leading to distinct analytic and practical advantages compared to traditional gating approaches. In waveform inversion, Laplace gating refers to specific algebraic manipulations in the Laplace domain using time-power gains; in Mixture of Experts architectures, Laplace gating employs L¹ distance-based partitioning, producing robust and accelerated expert specialization.

1. Mathematical Definition and Instantiations

Laplace gating functions in continuous domains arise from the application of time-power gains in the Laplace transform, and in neural architectures, they arise via L¹-norm–based exponentials centered at learned locations.

Laplace-domain time-power gating (waveform inversion):

• Time-domain function: Apply multiplicative gain $g_n(t) = t^n$ to the trace $d(t)$ for integer $n \geq 0$.
• Laplace transform:

$$\mathcal{L}\{t^n d(t)\}(s) \equiv \int_0^\infty t^n d(t) e^{-s t}\,dt.$$

• Classic Laplace property:

$$\mathcal{L}\{t^n f(t)\}(s) = (-1)^n \frac{d^n}{ds^n} F(s),$$

with $F(s)=\mathcal{L}\{f(t)\}(s)$, directly relating time-power gain to $n$-th order $s$-derivatives (Ha et al., 2014).

Neural mixture of experts (MoE, HMoE):

• For $x \in \mathbb{R}^d$, Laplace gate for cluster/expert $i$:

$$g_i(x; \mu_i, \tau) = \frac{\exp(-\|x - \mu_i\|_1 / \tau)}{\sum_j \exp(-\|x - \mu_j\|_1 / \tau)}$$

with learned centers $\mu_i$ and temperature $\tau$ controlling gate sharpness.

• Hierarchical MoE (HMoE) extends to multi-level Laplace gating, e.g., for outer and inner experts with respective centers and temperatures (Nguyen et al., 3 Oct 2024).

2. Analytic Properties and Functional Consequences

Laplace gating functions possess several theoretically significant properties arising from their definitions.

In waveform inversion:

• Multiplying by $t^n$ and Laplace transforming is equivalent to taking $n$ derivatives with respect to the Laplace damping $s$. Iteratively,

$$\frac{d^n}{ds^n} \tilde{d}(s) = (-1)^n \int_0^\infty t^n d(t) e^{-s t} dt$$

(Ha et al., 2014).

• This equivalence allows direct analytic computation of the gained traces, their derivatives, and gradients by leveraging closed-form Green's functions for specific backgrounds.

In HMoE architectures:

• The Laplace gate replaces affine (dot-product) scoring with norm-based (L¹) localization, which affects parameter dependencies:
  • Removes cross-couplings among parameters inherent to softmax gates, e.g., $$\partial/\partial \eta = \partial^2/\partial a \partial \tau$$.
  • For Laplace-Laplace HMoE, only the classical $$\partial/\partial \nu = (1/2)\partial^2/\partial \tau^2$$ relation remains (Nguyen et al., 3 Oct 2024).
• This structural separation leads to simpler optimization landscapes and tractable polynomial systems governing convergence analysis.

3. Objective Function Integration and Gradient Computation

Waveform inversion (Laplace-domain FWI):

• The standard FWI objective with Laplace gain:

$$E(s, n) = -\sum_{i,j} w_{ij} \ln\left( \frac{\frac{\partial^n}{\partial s^n} \tilde{d}_{ij}^{obs}(s)}{\frac{\partial^n}{\partial s^n} \tilde{u}_{ij}^{mod}(s)} \right)$$

where $w_{ij}$ are amplitude or offset weights.

• The gradient with respect to a model parameter $m_k$:

$$\frac{\partial E(s, n)}{\partial m_k} = \sum_{i,j} \frac{1}{\tilde{u}_{n,ij}} \frac{\partial \tilde{u}_{n,ij}}{\partial m_k} \cdot \ln\left( \frac{\tilde{u}_{n,ij}}{\tilde{d}_{n,ij}} \right)$$

• All required Laplace-domain responses and their derivatives can be constructed analytically for constant backgrounds via explicit Green's functions (Ha et al., 2014).

HMoE architectures:

• Gradients propagate through the Laplace gate using:

$$\frac{\partial g_i(x)}{\partial \mu_j} = \text{(normalizing factor)}/\tau \cdot \text{sign}(x - \mu_j) \cdot g_i(x) g_j(x)$$

and analogously for temperature derivatives.

4. Comparative Analysis with Softmax Gating

Laplace gating directly contrasts with softmax gating in both signal processing and neural architectures.

Structural and convergence implications (HMoE):

• Softmax gating with affine scores introduces parameter entanglement: various partial differential identities ($\partial/\partial \eta$, $\partial/\partial a$, etc.) persist, increasing the dimension of the convergence polynomial system.
• Laplace gating removes these couplings, leaving only scale-related dependencies—simplifying the analysis and enabling faster rates for expert estimation under over-parametrized regimes.
• Empirically, Laplace-Laplace (LL–LL) gating achieves superior convergence and specialization, with observed rates uniformly $O_P(n^{-1/4})$ for all $m$ (expert cell sizes), contrasted with variable and slower rates for Softmax–Softmax (SM–SM) (Nguyen et al., 3 Oct 2024).

5. Application Domains

Initial model building for Laplace-domain full-waveform inversion:

• Laplace gating with increasing $n$ shifts sensitivity from early (shallow) arrivals to late (deep) arrivals.
• Sequentially combining gradients for $n=0,1,\ldots,N$ yields a multiscale update, efficiently constructing robust velocity models starting from homogeneous backgrounds. This multiscale strategy provides improved initialization for subsequent inversions (Ha et al., 2014).

Hierarchical Mixture of Experts (HMoE):

• Laplace gating is used for both expert selection (input partitioning) and specialization acceleration, particularly in high-dimensional or multimodal spaces.
• Empirical studies include:
  • Multimodal fusion (MIMIC-IV): 12-layer Transformer+HMoE (LL–LL) shows improved AUROC and F1 on 48-hr mortality, length-of-stay, and phenotyping versus SM–SM baselines.
  • Latent-domain discovery (eICU, MIMIC-IV): HMoE–SL (outer softmax, inner Laplace) produces best-in-class discovery for readmission/mortality prediction.
  • Image classification (CIFAR-10, ImageNet): LL–LL gating yields top accuracy, with 1–2% gains over SM–SM in one-layer MoE setups (Nguyen et al., 3 Oct 2024).

Application	Laplace gating role	Empirical improvement
Waveform inversion	Multiscale sensitivity via $t^n$ gains	Deep/shallow features
HMoE multimodal fusion	Expert specialization, partitioning	+2–3 AUROC/F1
Image classification	Robust expert assignment	+1–2% accuracy

6. Hyperparameterization and Practical Guidance

Temperature parameters $\tau_1$ (outer) and $\tau_2$ (inner) control the sharpness of expert partitioning:

• Lower $\tau$ sharpens gate assignments, leading to more decisive expert selection; higher $\tau$ smooths decisions, increasing robustness.
• Empirical best practices: $\tau_1 \approx 0.3$, $\tau_2 \approx 0.8$ across diverse domains. Extremely small $\tau$ values ($< 0.01$) induce over-specialization and underfitting (Nguyen et al., 3 Oct 2024).
• Cross-validation over $\{0.1, 0.5, 1.0, 2.0\}$ is effective for tuning.

7. Signal Emphasis and Multiscale Decomposition

By design, Laplace gating provides a natural mechanism for temporally or spatially adaptive weighting:

• In the Laplace domain, $e^{-s t}$ suppresses late arrivals; multiplication by $t^n$ ($n > 0$) compensates, allowing energy from deeper events to be preserved.
• Gradient fields produced for different $n$ can be combined to generate updates that recover both shallow and deep features, establishing a multiscale geometric representation of subsurface structure (Ha et al., 2014).
• In expert models, the center-based gating using L¹ distance partitions the input space geometrically, enhancing the model's capacity to localize experts to well-defined regions without inducing unwanted parameter interactions.