Hybrid Dual-Path Linear (HDPL)

• Hybrid Dual-Path Linear (HDPL) is an architectural framework that combines block-diagonal local projections with low-rank global context integration to enhance efficiency and model control.
• Its two-path approach replaces traditional dense layers in Transformers and speech pipelines, resulting in reduced parameter counts, faster convergence, and improved validation loss.
• HDPL leverages a variational autoencoder bottleneck to create explicit latent control, enabling enhanced adaptability, interpretability, and multimodal synchronization in neural architectures.

Hybrid Dual-Path Linear (HDPL) is an architectural principle and family of operators that couple two topologically distinct transformation pathways within a linear or affine projection. HDPL-based methods have been proposed as a solution to the inefficiency and lack of structural inductive bias inherent in standard dense linear layers, both in deep sequence models such as Transformers and in multichannel speech enhancement pipelines. HDPL achieves this by explicitly decoupling local, high-rank detail processing from global, low-dimensional context integration, thereby providing gains in parameter efficiency, representational capacity, and architectural affordance for downstream adaptation and control (Khasia, 5 Feb 2026, Qin et al., 24 Jul 2025).

1. Mathematical Formulation in Deep Learning Architectures

In Transformer-style architectures, the HDPL operator replaces the standard dense linear transformation $y = x W^\top$ (with $W \in \mathbb{R}^{D_\text{out}\times D_\text{in}}$) by summing a block-diagonal local projection with a low-rank, variational bottleneck:

$$\mathcal{H}(x) = \Psi_\text{local}(x) + \Phi_\text{global}(x)$$

Local Path – Block-Diagonal Detail Projection:

Input and output features are partitioned into $K$ disjoint groups, yielding $W^{(B)} = \operatorname{diag}(W_1, ..., W_K)$, with $$W_k \in \mathbb{R}^{(D_\text{out}/K) \times (D_\text{in}/K)}$$. This yields high-rank computation within each block, preserving high-frequency features and enforcing no cross-group mixing.

$\Psi_\text{local}(x) = x (W^{(B)})^\top$

Global Path – Low-Rank VAE Context Projection:

A variational autoencoder bottleneck is instantiated as follows:

• Encoder computes

$$\mu(x) = x W_\mu^\top, \quad \log \sigma^2(x) = x W_\sigma^\top$$

where $$W_\mu, W_\sigma \in \mathbb{R}^{R\times D_\text{in}},\ R \ll D_\text{in}$$.

• Latent code (training): $z = \mu(x) + \sigma(x) \odot \epsilon$, $W \in \mathbb{R}^{D_\text{out}\times D_\text{in}}$0; at inference, set $W \in \mathbb{R}^{D_\text{out}\times D_\text{in}}$1.
• Decoder: $W \in \mathbb{R}^{D_\text{out}\times D_\text{in}}$2, with $W \in \mathbb{R}^{D_\text{out}\times D_\text{in}}$3.

Auxiliary KL-divergence regularization is applied over the $W \in \mathbb{R}^{D_\text{out}\times D_\text{in}}$4’s in each HDPL layer to prevent posterior collapse.

$W \in \mathbb{R}^{D_\text{out}\times D_\text{in}}$5

The complete HDPL mapping is: $W \in \mathbb{R}^{D_\text{out}\times D_\text{in}}$6

(Khasia, 5 Feb 2026)

2. Integration into Transformer and Other Architectures

The HDPL operator is incorporated into existing Transformer blocks via a surgical replacement strategy:

• HDPL replaces attention projections $W \in \mathbb{R}^{D_\text{out}\times D_\text{in}}$7 and MLP projections $W \in \mathbb{R}^{D_\text{out}\times D_\text{in}}$8 with $W \in \mathbb{R}^{D_\text{out}\times D_\text{in}}$9.
• Standard dense layers are retained for the attention output $$\mathcal{H}(x) = \Psi_\text{local}(x) + \Phi_\text{global}(x)$$0 and the MLP “down” projection $$\mathcal{H}(x) = \Psi_\text{local}(x) + \Phi_\text{global}(x)$$1.

Pseudocode implementations for both HDPL forward computation and entire Transformer blocks are specified, with all $$\mathcal{H}(x) = \Psi_\text{local}(x) + \Phi_\text{global}(x)$$2 terms summed into the loss. No ablations with “VAE path only” or “block-diag path only” were reported; only the binary comparison of surgical HDPL versus full-dense.

In signal processing, HDPL describes dual-path multichannel prediction, extending multi-channel linear prediction (MCLP) with pathways for both temporal and frequential filtering:

$$\mathcal{H}(x) = \Psi_\text{local}(x) + \Phi_\text{global}(x)$$3

where $$\mathcal{H}(x) = \Psi_\text{local}(x) + \Phi_\text{global}(x)$$4 and $$\mathcal{H}(x) = \Psi_\text{local}(x) + \Phi_\text{global}(x)$$5 are temporal and frequency domain prediction filters, respectively (Qin et al., 24 Jul 2025).

3. Training Procedures and Empirical Benchmarks

Deep Learning Setting:

• Dataset: FineWeb-Edu (SmolLM2-tokenized, $$\mathcal{H}(x) = \Psi_\text{local}(x) + \Phi_\text{global}(x)$$6, $$\mathcal{H}(x) = \Psi_\text{local}(x) + \Phi_\text{global}(x)$$7).
• Model baseline: 4-layer Llama-style Transformer ($$\mathcal{H}(x) = \Psi_\text{local}(x) + \Phi_\text{global}(x)$$8, 8 heads, RMSNorm, RoPE, SwiGLU, all-dense).
• HDPL configuration: Replace $$\mathcal{H}(x) = \Psi_\text{local}(x) + \Phi_\text{global}(x)$$9 with HDPL (rank $K$0, groups $K$1, KL scale $K$2).
• Optimization: AdamW ($K$3), peak LR $K$4, cosine schedule, $K$5 warmup steps, $K$6 total steps, batch size 32.

Model	Parameter Count (M)	Validation Loss	Throughput (k tok/s, TPU v3)
Baseline	67.11	4.3206	480
HDPL	62.53	4.2838	270

HDPL yields a 6.8% parameter reduction and validation loss improvement of approximately 0.0368. The slowdown in throughput is attributed to unoptimized kernels for grouped convolution and VAE. The hybrid model converges faster (e.g. loss $K$7 at step $K$8 versus baseline's $K$9).

Speech Enhancement Setting:

HDPL with dual-path MCLP and multi-norm beamforming outperforms GWPE, GWPE+MVDR, and WPD, especially for high $W^{(B)} = \operatorname{diag}(W_1, ..., W_K)$0 (late reverberation) and low SNR.

$W^{(B)} = \operatorname{diag}(W_1, ..., W_K)$1 (s)	GWPE	GWPE+MVDR	WPD	HDPL
0.2	2.80/8.2	2.95/9.5	3.05/10.2	3.00/10.0
0.6	2.20/4.8	2.40/6.0	2.50/6.8	2.65/7.5
1.0	1.85/3.2	2.05/4.1	2.15/4.5	2.30/5.4

([PESQ/SI-SNR] at SNR=25dB)

The order-selection algorithm—based on Pearson correlation bands—yields robust, near-optimal performance across a variety of reverberation times.

4. Architectural Affordances of the Probabilistic Latent Path

Integration of the VAE bottleneck creates explicit, low-dimensional latent spaces within Transformer backbones. This “materialized” latent path provides a locus for several modes of control:

• Inference-Time Control: Direct manipulation of $W^{(B)} = \operatorname{diag}(W_1, ..., W_K)$2 (e.g. arithmetic, clamping, conditioning) provides more tractable intervention than high-dimensional activation patching; adjusting $W^{(B)} = \operatorname{diag}(W_1, ..., W_K)$3 allows certainty or novelty modulation.
• Continual Adaptation and Interpretability: Latent distributions can be aligned to priors without replay, mitigating catastrophic forgetting and producing disentangled, probe-friendly representations through the KL regularizer.
• Hypernetworks and Meta-Learning: The reduced-size parameterization enables hypernetworks to generate $W^{(B)} = \operatorname{diag}(W_1, ..., W_K)$4 or inject latents, supporting rapid adaptation at lower computational cost.
• Cross-Model and Cross-Modal Synchronization: In federated and multimodal contexts, moment-matching over $W^{(B)} = \operatorname{diag}(W_1, ..., W_K)$5 sidesteps parameter permutation symmetries, and the universal latent manifold $W^{(B)} = \operatorname{diag}(W_1, ..., W_K)$6 provides a fusion interface for heterogeneous modalities, obviating large adapter layers.

These affordances suggest that HDPL’s probabilistic latent path is a foundational primitive for adaptive, tractable, and interpretable neural architectures (Khasia, 5 Feb 2026).

5. Signal Processing Instantiations: Dual-Path MCLP and Beamforming

In the multichannel speech enhancement context, HDPL refers to a two-stage pipeline:

1. Dual-Path MCLP: Simultaneous temporal and frequential prediction using $W^{(B)} = \operatorname{diag}(W_1, ..., W_K)$7 and $W^{(B)} = \operatorname{diag}(W_1, ..., W_K)$8, learned by minimizing an $W^{(B)} = \operatorname{diag}(W_1, ..., W_K)$9-regularized objective via a Proximal Alternating Linearized Minimization (PALM) solver.
2. Multi-Norm Beamforming: Output is denoised using a beamformer optimized for both power $$W_k \in \mathbb{R}^{(D_\text{out}/K) \times (D_\text{in}/K)}$$0 and sparsity $$W_k \in \mathbb{R}^{(D_\text{out}/K) \times (D_\text{in}/K)}$$1 criteria, under a look-direction constraint ($$W_k \in \mathbb{R}^{(D_\text{out}/K) \times (D_\text{in}/K)}$$2), using ADMM.

The order-selection method is data-driven, based on the cross-trial Pearson correlation, and shown to be robust over broad $$W_k \in \mathbb{R}^{(D_\text{out}/K) \times (D_\text{in}/K)}$$3 ranges. Empirical evaluation on 8-mic, reverberant noisy rooms using TIMIT demonstrates state-of-the-art speech enhancement, with high PESQ and SI-SNR even in challenging conditions (Qin et al., 24 Jul 2025).

6. Summary and Outlook

HDPL architectures generalize the concept of a “hybrid” linear transformation, coupling sparse, block-diagonal microstructures with global, low-rank, probabilistic bottlenecks. In Transformer LLMs, this improves efficiency (≈6.8% parameter reduction), generalization (lower validation loss), and unlocks architectural affordances for controllability, adaptation, and synchronization. In signal processing, dual-path MCLP and multi-norm beamforming under the HDPL formalism robustly dereverberate and denoise, outperforming established linear prediction and beamforming baselines in adverse acoustics.

Continued exploration of HDPL-based operators in vision, speech, and general multimodal fusion is suggested by the success and flexibility demonstrated in these applications (Khasia, 5 Feb 2026, Qin et al., 24 Jul 2025).