---
title: Inter-Layer Structural Encoders
url: https://www.emergentmind.com/topics/inter-layer-structural-encoders-ilse
type: topic
---

# Inter-Layer Structural Encoders

Inter-Layer Structural Encoders (ILSE) are a class of fusion modules designed to distill and integrate the information encoded in all layers of a pre-trained transformer, particularly large language models (LLMs), to produce a single, task-specific representation. Distinct from the prevailing approach of utilizing only the final-layer output, ILSE systematically aggregates summary vectors from all layers, leveraging structured graph-based message passing or permutation-invariant pooling. The leading ILSE variant, the Cayley-Encoder, utilizes expander Cayley graphs from the group \( SL(2,\mathbb{Z}_n) \) to induce sparse, bottleneck-free inter-layer communication. Empirical studies establish that ILSE yields substantial gains—in some cases up to 44% improvement in classification accuracy and 25 percentage points in semantic similarity metrics—over last-layer and best-layer baselines across a broad range of tasks and model scales, with minimal parameter and compute overhead [2603.22665].

## 1. Architectural Foundations of ILSE

Given a frozen transformer with \( L \) layers, ILSE operates as an independently trainable module that extracts per-layer summary vectors \( z_\ell \in \mathbb{R}^d \), \( \ell = 1\ldots L \), where each
\[
z_\ell = \frac{1}{T} \sum_{t=1}^T H_\ell[t,:]
\]
with \( H_\ell \in \mathbb{R}^{T \times d} \) denoting the \( \ell \)th layer's token activations over sequence length \( T \). ILSE computes a single output representation \( h^* \in \mathbb{R}^{d'} \) suitable for a downstream task (e.g., classification, semantic similarity).

ILSE comprises three principal structural variants:
- **Set‐Encoder:** Treats \( \{z_\ell\}_{\ell=1}^L \) as an unordered set; applies DeepSets, with permutation-invariant pooling.
- **Fully‐Connected (FC) Encoder:** Models full \( L \)-node connectivity; runs a message-passing neural network (MPNN) on a complete graph.
- **Cayley‐Encoder:** Projects layer nodes onto a sparse, 4-regular Cayley expander graph over \( SL(2,\mathbb{Z}_n) \), executing MPNN over this topology.

Each variant passes the fused representation \( h^* \) to a simple linear (classification) or cosine similarity (semantic textual similarity, STS) task head. The parameter overhead scales as \( 10^{-4} \) to \( 10^{-3} \) relative to the base model.

## 2. The Cayley-Encoder: Graph-Theoretic Mechanisms

The Cayley-Encoder instantiates the core insight that a sparse, highly connected, regular graph can promote robust information integration with minimal parameter cost.

**Expander Cayley Graph Construction:**
- Base group: \( G = SL(2,\mathbb{Z}_n) \), the group of \( 2 \times 2 \) matrices over integers mod \( n \) with determinant one.
- Generator set \( S \) of cardinality 4 chosen to make the Cayley graph non-bipartite, 4-regular, and a spectral expander with diameter \( O(\log|G|) \).
- Total nodes \( |V_n| = n^3 \prod_{p|n} (1 - 1/p^2) \). Choose the smallest \( n \) such that \( |V_n| \geq L \).

Each real layer \( \ell \) is randomly mapped to a unique graph node, and unused nodes are initialized as "virtual" with embedding zero.

**Graph Neural Network (GNN) Protocol:**
- Node features: \( h_v^{(0)} = z_\ell \) (layer nodes); \( h_v^{(0)} = 0 \) (virtual nodes).
- Message passing over \( K \) layers (\( K = 1 \) or \( 2 \)):
  \[
  m_v^{(k)} = \sum_{u \in N(v)} M^{(k)}(h_u^{(k-1)})
  \]
  \[
  h_v^{(k)} = U^{(k)}(h_v^{(k-1)}, m_v^{(k)})
  \]
  where \( M^{(k)} \), \( U^{(k)} \) are either small multilayer perceptrons (MLPs, as in Graph Isomorphism Networks, GIN) or linear+ReLU (as in Graph Convolutional Networks, GCN).
- After K steps, collect the "real layer" node states, pool (usually by summation), and apply an MLP readout to derive \( h^* \).

The expander properties of Cayley graphs guarantee:
- **Regularity:** All nodes have degree 4.
- **Logarithmic diameter:** \( O(\log L) \) hops suffice for any two nodes to communicate, preserving throughput.
- **Spectral gap:** Ensures fast mixing and mitigates over-squashing.

## 3. Mathematical Formulation and Training

The fusion mechanics in ILSE are precisely defined:
- **Layer mean-pooling:** For each layer, obtain \( z_\ell \) as above.
- **Set-Encoder (DeepSets):** \( h^* = \rho\left( \mathit{Pool}_{\ell=1}^L \phi(z_\ell) \right) \), with \(\phi, \rho\) as MLPs.
- **FC- and Cayley-Encoders:** Both apply the GNN protocol, but differ in underlying graph: complete vs sparse expander.

**Objective/Loss Functions:**
- **Classification:** Attach a linear softmax head; optimize cross-entropy:
  \[
  L_{CE} = -\sum_i y_i \cdot \log \operatorname{softmax}(Wh^*_i + b)
  \]
- **STS:** For a pair \((s_1, s_2)\), compute representations \((h^*_1, h^*_2)\):
  \[
  \hat{y} = \cos(h^*_1, h^*_2) = \frac{h^*_1{}^T h^*_2}{\|h^*_1\|\|h^*_2\|}
  \]
  Optimize mean-squared error:
  \[
  L_{MSE} = (\hat{y} - y)^2
  \]
Only ILSE parameters (GNN, DeepSet, and task head) are trained; the base LLM remains frozen.

## 4. Integration with Large Language Models

ILSE has been evaluated with a diverse array of LLMs, spanning 14M to 8B parameters:
- Pythia-14M through Pythia-2.8B
- Gemma2-2B
- Llama3-8B

**Operational workflow:**
1. Extract token-level hidden states \( H_\ell \), form \( z_\ell \).
2. Pass \( \{z_\ell\} \) into the selected ILSE variant.
3. Use Optuna to select hyperparameters (typical values: \( K \in \{1,2\} \), hidden dim 256, dropout 0.0–0.3, learning rate \(1 \times 10^{-4}\)–\(1 \times 10^{-3}\), weight decay \(1 \times 10^{-4}\)–\(1 \times 10^{-3}\)).
4. Optimizer: Adam. Batch size: 64 (classification), 256 (STS).
5. Overall parameter increase: 0.015%–0.1% for Cayley-Encoder.

This design achieves scalable inter-layer fusion with minimal computational and memory cost.

## 5. Empirical Evaluation and Results

ILSE was tested across 13 tasks:
- **5 classification tasks:** Banking77, Emotion, MTOP-Domain, MTOP-Intent, PoemSentiment.
- **8 semantic similarity tasks (MTEB):** STSBenchmark, and zero-shot STS12, STS13, STS14, STS15, STS16, BIOSSES, SICK-R.

**Key findings:**
- ILSE variants yield up to +30% improvement over last-layer and +25% over best-layer baselines. On the Emotion task, absolute improvement reaches +44% accuracy.
- Cayley-Encoder achieves up to +25 percentage-point advantage in Pearson \( r \) on STS over last-layer, outperforming depth-wise attention methods on 7 of 8 similarity tasks.
- Few-shot regime (32 labeled examples per class): Cayley-Encoder with Pythia-410M exceeds all full-data baselines.
- Model scaling: 14M-param LLMs with ILSE can match or surpass 2.8B-param LLMs using only last-layer outputs.
- Representative numbers (averaged over three LLMs, five classification tasks):

| Baseline         | Accuracy (%) |
|------------------|-------------|
| Last-Layer       | 57          |
| Best-Layer       | 60          |
| ILSE (Cayley)    | 87          |

On STS Benchmark (Pythia-410M):

| Method                  | Pearson \( r \) |
|-------------------------|-----------------|
| Last-Layer              | 0.39            |
| Cayley-Encoder (GIN)    | 0.56            |

## 6. Ablation Analysis and Component Comparisons

Extensive ablation underscores that all structured ILSE topologies (Set, FC, Cayley) surpass scalar and last-layer baselines, demonstrating the value of cross-layer aggregation.

- **FC vs Cayley:** FC is denser—\( L(L-1)/2 \) edges—yielding strong but less parameter-efficient gains versus Cayley (4L edges; similar or superior performance, notably on STS).
- **GNN variant:** GIN (sum+MLP) performs best on STS; GCN (degree-normalized mean) can outperform in classification.
- **MPNN depth \( K \):** Increasing from 1 to 2 layers offers minor improvements without over-smoothing, due to expander graph properties.

## 7. Algorithmic Workflow of Cayley-Encoder

1. Compute minimal \( n \) with \( |SL(2,\mathbb{Z}_n)| \geq L \).
2. Construct 4-regular Cayley graph \( G=(V,E) \) for \( SL(2,\mathbb{Z}_n) \).
3. Assign each layer \( \ell \) to a node; virtual nodes initialized with zero embeddings.
4. For \( k=1\ldots K \), carry out:
   - \( m_v^{(k)} = \sum_{u\in N(v)} W_m^{(k)} h_u^{(k-1)} \)
   - \( h_v^{(k)} = \operatorname{ReLU}( W_u^{(k)} [h_v^{(k-1)} \parallel m_v^{(k)}] ) \)
5. Collect states for real-layer nodes, pool by sum.
6. Apply MLP readout to aggregated vector.
7. Feed \( h^* \) to the task head; backpropagate through ILSE parameters only.

## 8. Significance and Implications

ILSE provides a mathematically principled, efficient, and modular approach to inter-layer fusion in frozen LLMs [2603.22665]. The Cayley-Encoder, in particular, couples group-theoretic graph construction with message-passing neural networks to achieve effective representation integration with strong regularization properties. This enables small models to attain performance competitive with far larger LLMs, significantly advances data efficiency in few-shot settings, and provides a robust basis for downstream transfer. A plausible implication is that structured inter-layer connectivity, especially with expander graphs, constitutes a powerful inductive bias for layer-wise information fusion in large-scale neural architectures.

Source: https://www.emergentmind.com/topics/inter-layer-structural-encoders-ilse