Inter-Layer Structural Encoders

• ILSE is a fusion module that aggregates summary vectors from all transformer layers to produce a task-specific representation.
• The Cayley-Encoder variant leverages sparse expander Cayley graphs for efficient, bottleneck-free inter-layer message passing.
• Empirical studies show that ILSE can improve classification accuracy by up to 44% and enhance semantic similarity metrics with minimal parameter overhead.

Inter-Layer Structural Encoders (ILSE) are a class of fusion modules designed to distill and integrate the information encoded in all layers of a pre-trained transformer, particularly LLMs, to produce a single, task-specific representation. Distinct from the prevailing approach of utilizing only the final-layer output, ILSE systematically aggregates summary vectors from all layers, leveraging structured graph-based message passing or permutation-invariant pooling. The leading ILSE variant, the Cayley-Encoder, utilizes expander Cayley graphs from the group $SL(2,\mathbb{Z}_n)$ to induce sparse, bottleneck-free inter-layer communication. Empirical studies establish that ILSE yields substantial gains—in some cases up to 44% improvement in classification accuracy and 25 percentage points in semantic similarity metrics—over last-layer and best-layer baselines across a broad range of tasks and model scales, with minimal parameter and compute overhead (Ulanovski et al., 24 Mar 2026).

1. Architectural Foundations of ILSE

Given a frozen transformer with $L$ layers, ILSE operates as an independently trainable module that extracts per-layer summary vectors $z_\ell \in \mathbb{R}^d$, $\ell = 1\ldots L$, where each

$z_\ell = \frac{1}{T} \sum_{t=1}^T H_\ell[t,:]$

with $H_\ell \in \mathbb{R}^{T \times d}$ denoting the $\ell$th layer's token activations over sequence length $T$. ILSE computes a single output representation $h^* \in \mathbb{R}^{d'}$ suitable for a downstream task (e.g., classification, semantic similarity).

ILSE comprises three principal structural variants:

• Set‐Encoder: Treats $\{z_\ell\}_{\ell=1}^L$ as an unordered set; applies DeepSets, with permutation-invariant pooling.
• Fully‐Connected (FC) Encoder: Models full $L$0-node connectivity; runs a message-passing neural network (MPNN) on a complete graph.
• Cayley‐Encoder: Projects layer nodes onto a sparse, 4-regular Cayley expander graph over $L$1, executing MPNN over this topology.

Each variant passes the fused representation $L$2 to a simple linear (classification) or cosine similarity (semantic textual similarity, STS) task head. The parameter overhead scales as $L$3 to $L$4 relative to the base model.

2. The Cayley-Encoder: Graph-Theoretic Mechanisms

The Cayley-Encoder instantiates the core insight that a sparse, highly connected, regular graph can promote robust information integration with minimal parameter cost.

Expander Cayley Graph Construction:

• Base group: $L$5, the group of $L$6 matrices over integers mod $L$7 with determinant one.
• Generator set $L$8 of cardinality 4 chosen to make the Cayley graph non-bipartite, 4-regular, and a spectral expander with diameter $L$9.
• Total nodes $z_\ell \in \mathbb{R}^d$0. Choose the smallest $z_\ell \in \mathbb{R}^d$1 such that $z_\ell \in \mathbb{R}^d$2.

Each real layer $z_\ell \in \mathbb{R}^d$3 is randomly mapped to a unique graph node, and unused nodes are initialized as "virtual" with embedding zero.

Graph Neural Network (GNN) Protocol:

• Node features: $z_\ell \in \mathbb{R}^d$4 (layer nodes); $z_\ell \in \mathbb{R}^d$5 (virtual nodes).
• Message passing over $z_\ell \in \mathbb{R}^d$6 layers ($z_\ell \in \mathbb{R}^d$7 or $z_\ell \in \mathbb{R}^d$8):

$z_\ell \in \mathbb{R}^d$9

$\ell = 1\ldots L$0

where $\ell = 1\ldots L$1, $\ell = 1\ldots L$2 are either small multilayer perceptrons (MLPs, as in Graph Isomorphism Networks, GIN) or linear+ReLU (as in Graph Convolutional Networks, GCN).

• After K steps, collect the "real layer" node states, pool (usually by summation), and apply an MLP readout to derive $\ell = 1\ldots L$3.

The expander properties of Cayley graphs guarantee:

• Regularity: All nodes have degree 4.
• Logarithmic diameter: $\ell = 1\ldots L$4 hops suffice for any two nodes to communicate, preserving throughput.
• Spectral gap: Ensures fast mixing and mitigates over-squashing.

3. Mathematical Formulation and Training

The fusion mechanics in ILSE are precisely defined:

• Layer mean-pooling: For each layer, obtain $\ell = 1\ldots L$5 as above.
• Set-Encoder (DeepSets): $\ell = 1\ldots L$6, with $\ell = 1\ldots L$7 as MLPs.
• FC- and Cayley-Encoders: Both apply the GNN protocol, but differ in underlying graph: complete vs sparse expander.

Objective/Loss Functions:

• Classification: Attach a linear softmax head; optimize cross-entropy:

$\ell = 1\ldots L$8

• STS: For a pair $\ell = 1\ldots L$9, compute representations $z_\ell = \frac{1}{T} \sum_{t=1}^T H_\ell[t,:]$0:

$z_\ell = \frac{1}{T} \sum_{t=1}^T H_\ell[t,:]$1

Optimize mean-squared error:

$z_\ell = \frac{1}{T} \sum_{t=1}^T H_\ell[t,:]$2

Only ILSE parameters (GNN, DeepSet, and task head) are trained; the base LLM remains frozen.

4. Integration with LLMs

ILSE has been evaluated with a diverse array of LLMs, spanning 14M to 8B parameters:

• Pythia-14M through Pythia-2.8B
• Gemma2-2B
• Llama3-8B

Operational workflow:

1. Extract token-level hidden states $z_\ell = \frac{1}{T} \sum_{t=1}^T H_\ell[t,:]$3, form $z_\ell = \frac{1}{T} \sum_{t=1}^T H_\ell[t,:]$4.
2. Pass $z_\ell = \frac{1}{T} \sum_{t=1}^T H_\ell[t,:]$5 into the selected ILSE variant.
3. Use Optuna to select hyperparameters (typical values: $z_\ell = \frac{1}{T} \sum_{t=1}^T H_\ell[t,:]$6, hidden dim 256, dropout 0.0–0.3, learning rate $z_\ell = \frac{1}{T} \sum_{t=1}^T H_\ell[t,:]$7–$z_\ell = \frac{1}{T} \sum_{t=1}^T H_\ell[t,:]$8, weight decay $z_\ell = \frac{1}{T} \sum_{t=1}^T H_\ell[t,:]$9–$H_\ell \in \mathbb{R}^{T \times d}$0).
4. Optimizer: Adam. Batch size: 64 (classification), 256 (STS).
5. Overall parameter increase: 0.015%–0.1% for Cayley-Encoder.

This design achieves scalable inter-layer fusion with minimal computational and memory cost.

5. Empirical Evaluation and Results

ILSE was tested across 13 tasks:

• 5 classification tasks: Banking77, Emotion, MTOP-Domain, MTOP-Intent, PoemSentiment.
• 8 semantic similarity tasks (MTEB): STSBenchmark, and zero-shot STS12, STS13, STS14, STS15, STS16, BIOSSES, SICK-R.

Key findings:

• ILSE variants yield up to +30% improvement over last-layer and +25% over best-layer baselines. On the Emotion task, absolute improvement reaches +44% accuracy.
• Cayley-Encoder achieves up to +25 percentage-point advantage in Pearson $H_\ell \in \mathbb{R}^{T \times d}$1 on STS over last-layer, outperforming depth-wise attention methods on 7 of 8 similarity tasks.
• Few-shot regime (32 labeled examples per class): Cayley-Encoder with Pythia-410M exceeds all full-data baselines.
• Model scaling: 14M-param LLMs with ILSE can match or surpass 2.8B-param LLMs using only last-layer outputs.
• Representative numbers (averaged over three LLMs, five classification tasks):

Baseline	Accuracy (%)
Last-Layer	57
Best-Layer	60
ILSE (Cayley)	87

On STS Benchmark (Pythia-410M):

Method	Pearson $H_\ell \in \mathbb{R}^{T \times d}$2
Last-Layer	0.39
Cayley-Encoder (GIN)	0.56

6. Ablation Analysis and Component Comparisons

Extensive ablation underscores that all structured ILSE topologies (Set, FC, Cayley) surpass scalar and last-layer baselines, demonstrating the value of cross-layer aggregation.

• FC vs Cayley: FC is denser—$H_\ell \in \mathbb{R}^{T \times d}$3 edges—yielding strong but less parameter-efficient gains versus Cayley (4L edges; similar or superior performance, notably on STS).
• GNN variant: GIN (sum+MLP) performs best on STS; GCN (degree-normalized mean) can outperform in classification.
• MPNN depth $H_\ell \in \mathbb{R}^{T \times d}$4: Increasing from 1 to 2 layers offers minor improvements without over-smoothing, due to expander graph properties.

7. Algorithmic Workflow of Cayley-Encoder

1. Compute minimal $H_\ell \in \mathbb{R}^{T \times d}$5 with $H_\ell \in \mathbb{R}^{T \times d}$6.
2. Construct 4-regular Cayley graph $H_\ell \in \mathbb{R}^{T \times d}$7 for $H_\ell \in \mathbb{R}^{T \times d}$8.
3. Assign each layer $H_\ell \in \mathbb{R}^{T \times d}$9 to a node; virtual nodes initialized with zero embeddings.
4. For $\ell$0, carry out:
   • $\ell$1
   • $\ell$2
5. Collect states for real-layer nodes, pool by sum.
6. Apply MLP readout to aggregated vector.
7. Feed $\ell$3 to the task head; backpropagate through ILSE parameters only.

8. Significance and Implications

ILSE provides a mathematically principled, efficient, and modular approach to inter-layer fusion in frozen LLMs (Ulanovski et al., 24 Mar 2026). The Cayley-Encoder, in particular, couples group-theoretic graph construction with message-passing neural networks to achieve effective representation integration with strong regularization properties. This enables small models to attain performance competitive with far larger LLMs, significantly advances data efficiency in few-shot settings, and provides a robust basis for downstream transfer. A plausible implication is that structured inter-layer connectivity, especially with expander graphs, constitutes a powerful inductive bias for layer-wise information fusion in large-scale neural architectures.