Language Feature Bottleneck

• Language feature bottleneck is a constraint where all predictive signals pass through a low-capacity, interpretable representation such as natural language or discrete codes.
• It is implemented via two-stage mappings and concept bottleneck models that link high-dimensional inputs to compressed representations, enhancing interpretability and sample efficiency.
• Empirical studies in image classification, continual learning, and speech processing demonstrate that these models achieve competitive accuracy while offering clear, human-readable explanations.

A language feature bottleneck, also referred to as a language bottleneck or language bottleneck model, constitutes a formal architectural or cognitive constraint in which all predictive or inferential information passes through an explicitly defined, interpretable, and highly compressed intermediary—most typically natural-language representations, low-dimensional feature layers, or discrete codes—before downstream processing or decision-making. This paradigm arises across domains including deep learning classifiers, continual learning, reinforcement learning, knowledge tracing, and even the cognitive science of language, and is motivated by the quest for interpretability, compression, sample efficiency, and human alignment.

1. Formal Definitions and Mathematical Characterization

A language bottleneck enforces that the passage of information from an arbitrary input space (e.g., images, sequences, behaviors) to final predictions or actions must traverse a low-capacity, interpretable representation—almost always in the form of natural language or a compressed set of discrete features. In deep neural architectures, this is typically instantiated as a two-stage mapping:

• $f_1: x \rightarrow s$, where $x$ is the high-dimensional input and $s$ is a compressed, discrete natural-language sequence or concept vector,
• $f_2: s \rightarrow \hat{y}$, where $\hat{y}$ is the output, for instance, a class label or a policy decision.

In image classification, for instance, $f_{\mathrm{cap}}: x \to (s_1,\dots,s_N)$ (captioning model) generates a textual description, which serves as the only channel for predictive information, followed by a text-to-class mapping, typically via a transformer LLM: $\hat{y} = \mathrm{softmax}(W h_{[CLS]} + b)$ (Udo et al., 2024). The bottleneck constraint is operationalized as a requirement that all downstream predictions, explanations, and actions be strictly functions of the textual or low-dimensional bottleneck representation.

In continual learning and concept bottleneck models (CBMs), this intermediary is a fixed set of concept neurons $c_1,\dots,c_k$ (e.g., each corresponding to a human-interpretable property), and predictive output derives from the concept activations: $\hat{y} = \mathrm{softmax}(W A^+_N(x) + b)$ with regularization to encourage sparsity and interpretability (Sun et al., 2024, Yu et al., 30 Mar 2025).

2. Model Architectures and Approaches

a. Two-Stage and Concept Bottleneck Models

Language bottleneck architectures in vision-LLMs, as exemplified by Udo & Koshinaka (Udo et al., 2024), combine off-the-shelf image captioners—such as BLIP, BLIP-2, or CLIP Interrogator—and transformer-based LLMs to achieve interpretation and classification. The entire predictive signal must be rendered as text before classification, ensuring every model prediction is accompanied by an explicit, human-readable explanation.

Concept bottleneck models (CBMs), both in language and vision, enforce a mapping from high-dimensional data to a k-dimensional, semantically labeled concept vector, which is then the only input to the final classifier or decision module. This framework is central to interpretability and is realized in Concept Bottleneck LLMs (CB-LLM) (Sun et al., 2024), Language-Guided CBMs for continual learning (Yu et al., 30 Mar 2025), and language-guided concept selection using LLMs (Yang et al., 2022).

b. Information Bottleneck and Compression-Based Bottlenecks

Information bottleneck methods, as in Infor-Coef (Tan, 2023), frame the bottleneck as a constraint on mutual information between compressed intermediate representations (e.g., pruned tokens in transformer layers) and both the input and target variables: the objective penalizes $I(s^i; h^i)$ for compression while maintaining label-predictive $I(s^i; y)$. Discrete key-value bottlenecks, employing hard quantization to small codebooks, effectively restrict how much and which aspects of the input are retained during learning (Diera et al., 2024).

c. Sequential, Cognitive, and Streaming Bottlenecks

At the cognitive level, language processing itself is hypothesized to arise from severe information-processing bottlenecks—the now-or-never bottleneck—whereby fleeting memory and incrementality force linguistic signals to be compressed into local, discrete, and systematically recoded representations (e.g., words, phrases) (Ferrer-i-Cancho, 2015, Futrell et al., 2024). The degree of predictive information ($I[X_{<t}; X_{\ge t}]$) quantifies the required memory, positing that human language structure is shaped by the need to minimize this information bottleneck across phonological, morphological, syntactic, and semantic levels (Futrell et al., 2024).

3. Empirical Performance and Application Domains

Language bottleneck models have been extensively evaluated in image classification, sequential decision-making, knowledge tracing, continual learning, and speech/language processing:

• Image classification: In disaster image classification on CrisisNLP, language bottleneck models using advanced captioners (BLIP-2, CLIP Interrogator) and BERT classifiers achieve or surpass the accuracy of strong vision-only models (ViT-Base, ViT-Large), with the text-only CLIP Interrogator pipeline reaching 85.09% and late-fusion with vision models achieving up to 87.3% (Udo et al., 2024).
• Interpretability: All predictions are explainable via natural-language captions or concept activations, as seen in CB-LLM and knowledge tracing LBMs (Sun et al., 2024, Berthon et al., 20 Jun 2025).
• Sample and computational efficiency: Language bottleneck approaches in knowledge tracing attain comparable accuracy to state-of-the-art embedding-based methods using orders of magnitude fewer samples, leveraging the human-interpretable summaries for faster learning (Berthon et al., 20 Jun 2025).
• Generalization and continual learning: Concept-bottleneck and discrete key-value models reduce catastrophic forgetting, with explicit bottleneck constraints ensuring retention and transfer: e.g., LG-CBM on ImageNet-subset improves by +3.06% over best non-interpretable continual learning baselines (Yu et al., 30 Mar 2025), while DKVB achieves up to 96.3% accuracy on 20 Newsgroups incremental tasks (Diera et al., 2024).
• Speech and language identification: Bottleneck features in deep and multilingual neural networks provide robust, transferrable acoustic representations that improve generalization across languages and data regimes (Ma et al., 2018, Padhi et al., 2020, Hermann et al., 2018, Zhang et al., 2022).

4. Theoretical and Cognitive Foundations

Language feature bottlenecks are grounded in both machine learning and cognitive science:

• Memory and incrementality in cognition: The now-or-never bottleneck posits that human forebrain working memory ($M$ bits, time span $\tau$) strictly limits the temporal and informational window for linguistic processing; this leads to chunk-and-pass architectures, real-time compression, and properties such as dependency locality and planarity in human languages (Ferrer-i-Cancho, 2015).
• Information-theoretic minimization: Optimization of predictive information ($I_{\mathrm{pred}}$) in lossy or lossless coding models yields codes with compositional, systematic, and local structure that directly mirror words and morphemes in natural language (Futrell et al., 2024).
• Compression and generalization: As per the information bottleneck principle, restricting the channel capacity at architectural bottlenecks (e.g., low-dimensional concept layers, codebooks) forces models to focus on the most relevant, generalizable features, filtering out idiosyncratic noise (Tan, 2023, Gao et al., 2021).

5. Interpretability, Synergy, and Fusion

A central motivation for language feature bottlenecks is interpretability. Every model prediction can be attributed to explicit concept activations, textual summaries, or human-readable rules:

• CBMs and Concept Alignment: Models such as CB-LLM and LG-CBM map hidden activations to named concepts, validated by human ratings of activation faithfulness (average 4.03 vs. random 3.15) and explanation preference (>80% in favor) (Sun et al., 2024, Yu et al., 30 Mar 2025).
• Complementary feature extraction: Vision- and language-only bottleneck models attend to different aspects (visual texture vs. semantic content), with late fusion yielding higher accuracy than either modality alone (+2.2 points on disaster classification) (Udo et al., 2024).
• Intervention and fairness: The interpretability of the bottleneck allows post-hoc intervention, such as removing or reweighting specific concepts (e.g., neutralizing "overpriced" flips sentiment predictions with 79% entailment) (Sun et al., 2024).

6. Trade-offs, Limitations, and Prospects

Bottleneck models inherently impose a trade-off between informativeness and interpretability:

• Information loss vs. explainability: While early simple captioners severely limited classification accuracy, recent advances (BLIP-2, CLIP Interrogator) have closed or overturned this gap, suggesting next-generation bottleneck models can be both accurate and interpretable (Udo et al., 2024, Yang et al., 2022).
• Dependence on LLM and captioner quality: Performance depends on the fidelity with which relevant source attributes can be verbalized and the capacity of downstream LLMs to utilize these representations.
• Bottleneck tightness and length constraints: Enforcing a length or dimensionality cap on the bottleneck introduces a minimal-description principle, but excessively tight bottlenecks degrade signal; empirical curves demonstrate accuracy plateaus as summary length increases (Berthon et al., 20 Jun 2025).
• Scalability and generalization: Class-specific attribute bottlenecks (ALBM) avoid spurious cue inference and enable zero-shot generalization to unseen classes through unified attribute schemas (Zhang et al., 26 Mar 2025). However, full disentanglement and attribute independence remain open challenges.

Future research in language feature bottlenecks focuses on extending interpretability to new modalities, refining bottleneck organization (class- or attribute-specific), and integrating with humans-in-the-loop for more effective, steerable, and generalizable learning systems.