Semantic-Informed Kernels (SIK) in Language Models

• Semantic-Informed Kernels (SIK) are specialized kernel functions that embed linguistic semantic similarity into neural models to enhance interpretability and uncertainty quantification.
• They utilize techniques like weighted clustering of Word2Vec and sentiment features to construct coherent convolutional filters, enabling visualized attention over input tokens.
• SIKs facilitate efficient transfer learning and robust uncertainty estimation by applying methods such as Kernel Language Entropy and positive semidefinite kernel matrices.

Semantic-Informed Kernels (SIK) are kernel functions or convolutional filters with explicit alignment to semantic properties of linguistic units, developed to enhance interpretability, coherence, reusability, and uncertainty quantification in neural models of language. SIKs appear in two principal lines: (1) as semantically coherent and reusable convolutional kernels for sentence classification in convolutional neural networks (CNNs) (Lakshmana et al., 2016); and (2) as positive semidefinite kernels encoding semantic similarity structure among LLM outputs for uncertainty quantification, as in Kernel Language Entropy (KLE) (Nikitin et al., 2024). Both approaches share a core methodology: semantic similarity is embedded directly into the kernel construction, either by clustering and parameterization for CNNs or by defining and normalizing kernel matrices over model outputs.

1. SIK in CNNs for Sentence Classification

Let a sentence of length $n$ be represented as $X\in \mathbb{R}^{n\times d}$, where each row is a pretrained Word2Vec embedding $\mathbf{x}_i\in \mathbb{R}^d$ (typically $d=300$). A CNN model applies $m$ convolutional kernels $\mathbf{v}_j \in \mathbb{R}^{k\,d}$, each spanning $k$ consecutive tokens. The convolutional response is

$$f_{j,i} = \mathrm{ReLU}\left(\mathbf{v}_j^\top\,\mathrm{vec}(X_{i:i+k-1})+b_j\right)$$

with max-pooling

$g_j = \max_{1\leq i\leq n-k+1}\, f_{j,i}.$

The pooled features $\mathbf{g} = [g_1,\dots,g_m]^T$ feed a linear softmax classifier.

To enforce semantic coherence, SIK construction proceeds by (A) selecting a manageable subset of $k$-grams (e.g., by filtering for sentiment lexicon hits, parse constituents, or POS), (B) representing each $k$-gram by its concatenated Word2Vec vectors $\mathbf{p}_l$ and SentiWordNet-based sentiment vector $\mathbf{s}_l$, and (C) forming a joint feature $$\mathbf{z}_l = [\alpha\,\mathbf{p}_l;\,\beta\,\mathbf{s}_l]$$.

$k$-grams are clustered via weighted K-means using

$$d(l,l') = \|\mathbf{p}_l-\mathbf{p}_{l'}\|_2^2 + \gamma\|\mathbf{s}_l-\mathbf{s}_{l'}\|_2^2$$

yielding clusters $\{C_1,\ldots, C_K\}$.

Each kernel is parameterized as a non-negative linear combination of its cluster members:

$$\mathbf{v}_j = \sum_{l\in C_j} \theta_{jl}\,\mathbf{p}_l , \quad \theta_{jl}\geq 0.$$

Centroid values $\theta_{jl}=1/|C_j|$ or learned weights initialized near this value are used. Training optimizes cross-entropy loss plus $\ell_1$ penalty and (optionally) a cluster-coherence regularizer:

$$\mathcal{L} = -\frac{1}{n}\sum_i \log p(y_i \mid X_i;\mathbf{w},\theta) + \lambda(\|\mathbf{w}\|_1+\|\theta\|_1) + \mu\sum_{j}\sum_{l,l'\in C_j} d(l,l').$$

Backpropagation updates $\theta_{jl}$ and classifier weights $\mathbf{w}$.

2. Visualization and Interpretability

SIK-equipped CNNs enable visualization of "attention" over input tokens by tracking, for each filter $j$, the position $\hat i_j = \arg\max_i f_{j,i}$. Define per-token scores

$$a_t = \sum_{j=1}^m \mathbf{1}[t\in \{\hat i_j,\ldots,\hat i_j+k-1\}]\,f_{j,\hat i_j}\,|\mathbf{w}_c^{(j)}|,$$

where $\mathbf{w}_c^{(j)}$ is the classifier weight for kernel $j$ and class $c$. Normalization and visualization (e.g., as a heatmap) yield interpretable model rationales that reflect the contribution of each $k$-gram to the final decision (Lakshmana et al., 2016).

3. Reusability and Transfer Across Domains

The reusability property allows kernels learned on a source domain to be applied to a target domain with minimal retraining. Kernels $\{\mathbf{v}_j\}$ are frozen; only classifier weights $\mathbf{w}$ are retrained, which is computationally efficient (e.g., $2$ hours for full CNN vs $2$ minutes with frozen kernels on SST-1). Adding a small proportion ($10\%-25\%$) of new domain-specific flexible filters can close the remaining gap in accuracy to within $1\%-2\%$ of CNN-Static baselines (Lakshmana et al., 2016). This transferability is a direct result of explicitly encoding reusable semantic structure into the kernels.

4. Semantic-Informed Kernels for Fine-Grained Uncertainty Quantification in LLMs

Beyond convolutional architectures, SIKs serve as the foundation of the Kernel Language Entropy (KLE) framework for uncertainty quantification in LLMs (Nikitin et al., 2024). Given $N$ output samples $O=\{o_1,\dots,o_N\}$, a semantic similarity kernel $k'(o_i,o_j)$ (e.g., cosine similarity or RBF kernel on embeddings) forms the unnormalized matrix $K'_{ij}=k'(o_i,o_j)$. KLE normalizes $K'$ to unit trace:

$$K''_{ij} = \frac{K'_{ij}}{\sqrt{K'_{ii} K'_{jj}}}, \qquad K = \frac{K''}{\mathrm{Tr}(K'')}.$$

$K$ is PSD, symmetric, and $\mathrm{Tr}(K)=1$, qualifying as a quantum density matrix $\rho$. The von Neumann entropy,

$$S(\rho) = -\mathrm{Tr}[\rho \log \rho] = -\sum_{i=1}^N \lambda_i \log \lambda_i$$

(with $\lambda_i$ the eigenvalues of $\rho$), quantifies semantic uncertainty among model outputs, with higher $S(\rho)$ indicating greater spread in the semantic content of those outputs.

Unlike previous methods based on hard clustering (Semantic Entropy, SE), KLE captures graded semantic dependencies, not just equivalence classes. SE arises as a special case: for block-diagonal kernel $K_{SE}$ (each block corresponding to a cluster), $S(K_{SE})$ exactly matches cluster-probability entropy. Any kernel encoding richer pairwise similarity strictly generalizes SE, distinguishing not only between but within semantic clusters (Nikitin et al., 2024).

5. Implementation and Empirical Results

In SIK for CNNs, typical hyperparameters include Word2Vec dimension $d=300$, kernel widths $k\in\{3,4,5\}$, $100$ clusters per $k$-gram-type ($m=300$ total kernels), and regularization parameters ($\lambda$ in $[10^{-6},10^{-8}]$, $\mu\approx 10^{-3}-10^{-4}$). Training uses AdaDelta or Adam, minibatches of size $50$, and feature dropout of $0.5$. Ablation shows that SIK achieves accuracy nearly matching full CNNs—within $1$–$2\%$—on benchmarks such as MR, IMDB, SUBJ, SST-1, and SST-2. Efficient kernel reuse delivers $50\times$ training speedup with only minimal loss in prediction quality (Lakshmana et al., 2016).

For SIK in KLE, kernel choices include Gaussian RBF, cosine similarity, graph-based heat kernels, and block-diagonal mixtures. Hyperparameter selection via cross-validation or entropy convergence diagnostics yields robust performance. Computational costs are dominated by pairwise computations ($O(N^2)$) and eigen-decomposition ($O(N^3)$), but with $N\approx 10-20$, these are negligible relative to LLM inference. Low-rank approximations are feasible for large $N$. Empirically, in 60 model-task pairs, KLE significantly outperforms SE methods in AUROC and AUARC for uncertainty estimation: e.g., on Llama 2 70B Chat and BioASQ, AUROC rises from $0.74$ (SE) to $0.87$ (KLE), with parallel gains across Natural Questions, SQuAD, SVAMP, and TriviaQA (Nikitin et al., 2024).

6. Context, Extensions, and Significance

Semantic-Informed Kernels unify the motivations of interpretability, transferability, and calibrated modeling of linguistic uncertainty. In CNNs, semantic coherence in filters yields models whose decisions can be directly traced to clusters of interpretable phrase features, rather than to opaque, uninterpretable convolutional weights. In LLM output analysis, SIK-based entropies capture semantic variability that bypasses shallow lexical or syntactic equivalence. The proven generalization of KLE over hard-cluster-based SE demonstrates that the SIK approach can capture subtler distinctions, such as paraphrases and variations in answer framing, providing more nuanced confidence signals that are unattainable with discrete equivalence class methods.

A plausible implication is that future research may extend SIK constructions to other architectures and modalities, leveraging their capacity to encode graded semantic similarity in both feature learning and output modeling. The transferability and interpretability delivered by semantic-informed kernel methods suggest new directions for research in controllable generalization, model auditing, and robust deployment across diverse text domains (Lakshmana et al., 2016, Nikitin et al., 2024).