Continuous Cross-Layer Attention Transmission

• CCLAT is a continuous attention mechanism that uses learned nonnegative clock rates to enforce smooth, monotonic, near-diagonal alignments.
• It replaces external positional encodings with a probabilistic meeting-kernel derived from learned clocks, ensuring stability in both parallel and autoregressive decoding.
• The mechanism integrates into Transformer architectures with minimal overhead, showing robust improvements in tasks such as frame-synchronous sequence transduction and text-to-speech synthesis.

Continuous Cross-Layer Attention Transmission (CCLAT) is a rigorous neural attention mechanism formulated to align discrete source sequences with continuous or ordered target sequences, where explicit modeling of temporal or sequential alignment is crucial. Unlike standard scaled dot-product attention, CCLAT—introduced as Stochastic Clock Attention (SCA)—replaces external positional encodings and masks with learned, nonnegative latent “clocks” on source and target sequences. The attention kernel is derived from the meeting probability of these clocks, enforcing built-in continuity, monotonicity, and near-diagonal alignment, central to tasks such as frame-synchronous sequence transduction and text-to-speech synthesis (Soh et al., 18 Sep 2025).

1. Formal Definition and Mathematical Framework

Let $X_s$ for $s \in [0,S]$ be the discrete source indices and $Y_t$ for $t \in [0,T]$ the continuous or ordered target indices. Each index is projected via learned features: $$\eta^X_s = \mathcal{F}(X_s),\quad \eta^Y_t = \mathcal{G}(Y_t)$$ Attention alignment is modeled using learned, nonnegative rate functions: $$\phi_X(s) = \phi(\eta^X_s),\quad \phi_Y(t) = \phi(\eta^Y_t),\quad \phi(\cdot) \ge \varepsilon > 0$$ Two clock variants are defined:

• Normalized clocks (parallel decoding; requires known global length)

$$\lambda^X_s = \frac{\int_0^s \phi_X(u)\,du}{\int_0^S \phi_X(u)\,du},\quad \lambda^Y_t = \frac{\int_0^t \phi_Y(v)\,dv}{\int_0^T \phi_Y(v)\,dv}$$

• Unnormalized clocks (autoregressive decoding; local cumulative only)

$$\tilde\lambda^X_s = \int_0^s \phi_X(u)\,du,\quad \tilde\lambda^Y_t = \int_0^t \phi_Y(v)\,dv$$

Often, these are rescaled for comparability.

The kernel for cross-layer attention, termed the “meeting-kernel,” is given by the marginal probability density that the two clocks coincide at $(s, t)$: $$K_{\mathrm{meet}}(s, t) = \mathbb{E}\left[\delta(\lambda^X_s - \lambda^Y_t)\right]$$ A path-integral treatment (Martin–Siggia–Rose / Onsager–Machlup formalism) linearizes clocks around deterministic means with Gaussian fluctuations, yielding a closed-form kernel: $$K_{\mathrm{meet}}(s,t) = \frac{1}{\sqrt{2\pi\,\Sigma_{s,t}^2}} \exp\left(-\frac{(\lambda^x_s-\lambda^y_t)^2}{2\,\Sigma_{s,t}^2}\right)$$ where

$\Sigma_{s,t}^2 = A_X(s) + A_Y(t)$

and $A_X(s), A_Y(t)$ are variance integrals dependent on learned feature derivatives and covariance surrogates.

2. Structural Properties: Continuity, Monotonicity, and Near-Diagonal Bias

SCA/CCLAT fundamentally enforces three properties critical to alignment tasks:

• Strict monotonicity: The nonnegative rate $\phi \ge \varepsilon$ ensures $\lambda$ is strictly increasing, enforcing order.
• Local continuity: For small changes in $s$ (or $t$), $\lambda^x_s$ (or $\lambda^y_t$) changes smoothly, yielding locally continuous attention alignments.
• Near-diagonal bias: The Gaussian kernel structure with a strong quadratic penalty on $(\lambda^x_s - \lambda^y_t)^2$ inherently aligns attention along the diagonal (i.e., $t \propto s$).

Notably, these properties eliminate the need for external positional encodings, complicated masking, or auxiliary guided loss terms. Causality and smoothness are intrinsic to the construction (Soh et al., 18 Sep 2025).

3. Attention Score Formulation and Comparison to Scaled Dot-Product Attention

The SCA attention score for each $(s, t)$ interaction is: $$S_{s,t} = -\frac{\|\lambda_s - \lambda_t\|_2^2}{2\,\sqrt{d}\,\Sigma_{s,t}^2}$$ (with constants absorbed into a learned $\mathrm{logit\_scale}$), where $d$ is the representation dimension.

For contrast, standard scaled dot-product attention (SDPA) computes: $$S^{\mathrm{SDPA}}_{s,t} = \frac{(\eta^X_s)^\top (\eta^Y_t)}{\sqrt{d}}$$ which requires explicit positional encodings to regularize sequence order. In CCLAT/SCA, alignment kernels derived from learned clocks serve this function directly.

4. Decoding Regimes and Algorithmic Realization

SCA/CCLAT operates in two regimes, reflected in implementation logic via a normalization flag:

• Parallel decoding (normalized clocks):
  • $\lambda_s$ and $\lambda_t$ are computed over the full sequence.
  • Variance surrogate $var_s \propto pos_s(1 - pos_s)$ with $pos_s = (s-0.5)/S$.
  • Softmax is applied row-wise to the score matrix $S_{s,t}$.
• Autoregressive decoding (unnormalized clocks):
  • $z_{0,s} = \sum_{u \leq s} \phi(\eta^X_u)$.
  • Variance surrogate $var_s \propto pos_s$ with $pos_s = s - 0.5$.
  • Scores updated incrementally, enforcing causal masking.

The essential pseudocode is presented in the source (Soh et al., 18 Sep 2025), highlighting the computation of clocks, variance surrogates, pairwise distances, and the final attention score.

5. Integration and Hyperparameterization

CCLAT/SCA integrates directly into Transformer cross-attention layers with minimal adaptation:

• Projections $QW_q, KW_k, VW_v$ into $d$-dimensional query, key, and value spaces.
• Time-step layer normalization (“MaskedTimeNorm”) over non-padded tokens.
• The nonlinearity $\phi(x) = \tfrac{1}{2}(1 + x\frac{1+x+|x|}{1+|x|})$ enforces smooth, nonnegative clock rates.
• Score scaling by $1/\sqrt{d}$ with learned $\mathrm{logit\_scale}$.
• Multi-head structure is preserved: attention computations are independent per head and concatenated.
• Parameter cost is limited to standard QKV projections and the $\mathrm{logit\_scale}$.

A typical architecture used for evaluation consisted of a 6-layer encoder, 4-layer decoder, $d_\mathrm{model}=256$, 4 heads, and dropout 0.1, with no additional model parameters required beyond those for QKV and scaling.

6. Empirical Evaluation: Text-to-Speech Alignment

The SCA/CCLAT mechanism was evaluated on the LJSpeech-1.1 dataset (≈24 hours of speech, 80-dim mel-spectrogram targets). Experiments replaced standard cross-attention with either SCA or SDPA, using the same architecture and evaluation pipeline (ASR: Whisper & wav2vec 2.0, metrics: WER/CER).

Key outcomes:

• Parallel decoding (normalized): SCA produced consistently sharp, near-diagonal alignments for all mel-to-phoneme ratios (MPR $\in [3.0, 10.0]$), whereas SDPA degraded severely at low/high MPR due to truncation/overrun and blurring artifacts. At best MPR $=6.0$, SCA achieved 7.03% WER / 3.66% CER vs. SDPA 7.39% / 3.94%. SCA was notably more stable under global time-scaling.
• Autoregressive decoding (unnormalized): SDPA yielded incoherent alignments (WER/CER ≈ 100%); SCA, however, gave intelligible speech, with WER = 66.5% and CER = 48.5% at MPR = 7.0.

7. Extensions, Use Cases, and Limitations

Potential avenues for extension and application include:

• Multi-scale or hierarchical clock variants (e.g., combining global and local clocks).
• Direct application to video or continuous temporal signals, facilitating smooth frame-synchronous alignment and preventing attention “jumps.”
• Integration into diffusion or flow-based decoders as a form of alignment guidance.

Limitations are documented:

• The mechanism presumes roughly monotonic alignments, i.e., it does not naturally accommodate heavy sequence reordering.
• The normalized-clock regime (used for parallel decoding) demands knowledge or prediction of the sequence’s global length.
• The Brownian-bridge variance approximation simplifies fluctuations, potentially neglecting higher-order effects (Soh et al., 18 Sep 2025).

A plausible implication is that while CCLAT provides alignment benefits in continuous/monotonic tasks with negligible parameter overhead, alternative approaches may be required for applications involving non-monotonic or heavily scrambled alignments.