---
title: Ambiguity-Encoding Neurons in Neural Models
url: https://www.emergentmind.com/topics/ambiguity-encoding-neurons-aens
type: topic
---

# Ambiguity-Encoding Neurons in Neural Models

Searching arXiv for the specified papers to ground the article in the current preprint record.
arXiv search query: 1908.11771 OR 2312.05484 OR 2410.00781 OR 2509.13664 OR 2512.08815 OR 2512.23531
Ambiguity-Encoding Neurons (AENs) denote neurons, hidden dimensions, or closely related representational units whose activity carries information about ambiguity, underspecification, or competition among candidate interpretations. In the literature surveyed here, the term is introduced operationally for large language models (LLMs): an AEN is a neuron, or tiny set of neurons, whose activation is highly predictive of whether a question is ambiguous under a linear probe and whose perturbation strongly degrades ambiguity classification [2509.13664]. Closely related work extends the concept beyond this narrow operational definition. Some papers provide direct sparse neuron-level evidence, whereas others provide representation-level, population-level, dendritic, or normative accounts that are AEN-adjacent rather than definitive. Taken together, this literature frames ambiguity not as a single phenomenon but as a family of coding problems involving lexical sense selection, question underspecification, perceptual competition, contextual disambiguation, and ambiguous-but-efficient population codes [1908.11771][2312.05484][2410.00781][2512.08815][2512.23531].

## 1. Definition and conceptual boundaries

In its strictest current usage, an AEN is not a biologically defined cell type or a neuron class discovered from first principles. It is an operational construct: a sparse subset of hidden dimensions within a model layer whose activations are strongly linearly predictive of ambiguity and whose targeted perturbation has a large functional effect on ambiguity detection or ambiguity-related behavior [2509.13664]. Under this definition, the central empirical question is whether ambiguity is merely distributed somewhere in representation space or whether it is concentrated in a very small number of coordinates.

The literature also uses broader, partly analogical notions of ambiguity encoding. In neural machine translation (NMT), ambiguity is instantiated as source-side lexical ambiguity, specifically ambiguous nouns whose correct translation depends on context [1908.11771]. In perceptual and systems-neuroscience settings, ambiguity appears as competition between alternative stimulus interpretations or as contextual uncertainty about how sensory evidence should be integrated [2312.05484][2410.00781]. In information-theoretic work, ambiguity refers to noninvertible single-unit response functions, where multiple stimulus values can map to the same unit response even though the population code remains jointly informative [2512.23531]. In conceptual neurobiological accounts, ambiguity is related to latent interpretive state, metadata, or hypothesis-dependent routing rather than to an explicit scalar ambiguity variable [2512.08815].

This diversity creates an important boundary condition. Some papers address explicit ambiguity representation, some address ambiguity resolution, and some address coding architectures in which single-unit ambiguity is beneficial or unavoidable. AENs therefore sit at the intersection of probe-based interpretability, sparse feature localization, distributed coding, and context-sensitive computation. The strongest direct evidence concerns sparse, linearly decodable neurons in LLMs; the remaining literature is best understood as supplying prerequisites, mechanisms, or normative rationales rather than a settled unified theory.

## 2. Sparse neuron-level evidence in large language models

The most direct arXiv evidence for AENs comes from the study of question ambiguity in instruction-tuned 7B–8B LLMs [2509.13664]. That work defines question ambiguity as a question being under-specified or contextually incomplete, such that a human would naturally ask for clarification before answering. Ambiguous instances are original underspecified questions, and clear instances are rewritten or context-enriched disambiguated versions drawn from AmbigQA and SituatedQA. The analysis focuses on the model’s pre-filling stage, that is, the forward pass over the prompt before autoregressive generation begins.

For an input sequence \(x=(x_1,\dots,x_T)\), the model produces hidden states
\[
h^{(\ell)}(x)=\big(h_1^{(\ell)},\dots,h_T^{(\ell)}\big)\in\mathbb{R}^{T\times d},
\]
which are summarized by mean pooling,
\[
\bar h^{(\ell)}(x)=\frac{1}{T}\sum_{t=1}^T h_t^{(\ell)}\in\mathbb{R}^d.
\]
At the default probe layer \(\ell=14\), ambiguity is decoded by logistic regression,
\[
\hat y(x)=\sigma\!\left(w^\top \bar h^{(\ell)}(x)+b\right).
\]
Neurons are ranked by \(|w_i|\), and the top-\(k\) coordinates are functionally validated by injecting Gaussian noise only into those coordinates and measuring the accuracy drop. The selected set \(\mathcal S_k\) is treated as the AEN set when perturbing it causes the strongest degradation. This yields a sparse definition grounded in both correlational evidence and intervention.

The reported sparsity is unusually strong. The paper identifies **1 AEN** in Mistral 7B, **1 AEN** in Gemma 7B, and **3 AENs** in LLaMA 3.1 8B. At Layer 14, full probes reach **93.30**, **90.65**, and **95.25** accuracy on AmbigQA for Mistral 7B Instruct v0.3, LLaMA 3.1 8B Instruct, and Gemma 7B IT respectively, and **94.14**, **95.40**, and **97.10** on SituatedQA. AEN-only probes remain close: **90.30**, **88.60**, and **92.00** on AmbigQA, and **92.35**, **94.00**, and **96.90** on SituatedQA. These results support the claim that ambiguity is not only linearly decodable from full hidden states but can be concentrated in an extremely small number of dimensions.

The paper also reports cross-dataset recurrence of top neurons within each model. Shared top neurons across AmbigQA and SituatedQA are **2070** for Mistral, **788, 1384, 4062, 4055** for LLaMA, and **1995** for Gemma [2509.13664]. This does not establish cross-model correspondence, and the paper explicitly states that the representation appears model-specific rather than universal, but it does argue against a purely dataset-artifactual explanation.

Layerwise analysis places these AENs early in the forward computation. Probe accuracy rises rapidly in early layers, performance saturates before Layer 5 in all three models, and in Gemma 7B the AEN-only probe exceeds **90% by Layer 2**. This indicates that ambiguity recognition is formed during prompt processing rather than emerging only during later generation-related computation. The same work further shows that manipulating the AENs can shift behavior from direct answering toward abstention or clarification. On AmbigQA, steering only the AENs yields abstention rates of **18.0%** for Mistral 7B, **52.0%** for LLaMA 3.1 8B, and **13.2%** for Gemma 7B; reverse steering pushes ambiguous cases back toward direct answering, with direct-answer rates of **56.2%** and **52.6%** for LLaMA 3.1 8B on AmbigQA and SituatedQA, respectively. The paper does not claim that these neurons are uniquely or exhaustively responsible for ambiguity processing, but it does provide the strongest neuron-level AEN evidence presently available in this set of works.

## 3. Distributed and component-level precursors

Before the term AEN was introduced, related work in NMT had already shown that ambiguity-resolving information can become highly accessible inside contextual representations, even without identifying specific neurons [1908.11771]. That study asks where disambiguating information resides inside RNNS2S and Transformer systems for lexical word-sense disambiguation, using ambiguous German source nouns in the ContraWSD benchmark. The classifier takes the concatenation \([R_{ambi};R_{sense}]\), where \(R_{ambi}\) is the representation at the ambiguous source noun and \(R_{sense}\) is the target-side embedding of a candidate translation, and predicts whether the candidate is the correct translation in context.

The core result is representational rather than neuron-localized. Static source embeddings reach only about **63–69%** accuracy, whereas top-layer encoder hidden states exceed **91%** in all settings. The reported values are **63.1 \rightarrow 94.2 \rightarrow 97.5** for embedding, encoder, and decoder in DE\(\rightarrow\)EN RNN; **63.2 \rightarrow 97.2 \rightarrow 98.3** in DE\(\rightarrow\)EN Transformer; **68.7 \rightarrow 91.7 \rightarrow 95.1** in DE\(\rightarrow\)FR RNN; and **68.9 \rightarrow 95.6 \rightarrow 96.9** in DE\(\rightarrow\)FR Transformer. The absolute gain from embeddings to encoder states is roughly **23–34 points**. Since the probing classifier is shallow, the paper interprets this as evidence that the encoder has already integrated context needed for word-sense disambiguation into the hidden state at the ambiguous noun.

Layerwise findings sharpen the argument. Even the first encoder layer substantially outperforms embeddings, but performance increases with depth in both architectures. In stacked bidirectional RNNs, forward states outperform backward states at corresponding depths, and concatenating forward and backward states yields **96.8%** for DE\(\rightarrow\)EN and **95.7%** for DE\(\rightarrow\)FR. In Transformers, self-attention analysis shows that ambiguous nouns allocate lower self-weight than nouns in general and, from the second layer onward, exhibit lower attention entropy than nouns in general. Attention entropy is defined as
\[
E_{At}(x_t)=-\sum_{i=1}^{|x|} At(x_i,x_t)\log At(x_i,x_t).
\]
The reported interpretation is that self-attention “recognizes” ambiguous nouns and shifts attention outward toward a smaller set of relevant context words. This is not neuron-level evidence for AENs, but it is strong evidence for ambiguity-encoding subspaces and ambiguity-sensitive contextual routing. A plausible implication is that direct AEN discovery becomes more likely in upper encoder layers, particularly in Transformers, where ambiguity information appears progressively enriched.

## 4. Normative theory: ambiguous single units and informative populations

A distinct theoretical line argues that ambiguity at the single-unit level can be optimal rather than pathological [2512.23531]. The setting comprises an underlying relevant variable \(x\), an intermediate signal \(s\), and multiple sensors or channels \(C_i\), with architecture
\[
x \to s \to \{C_i\}_{i=1}^K.
\]
Individual channels are capacity-limited, conditionally independent given \(s\), and may have noninvertible mean responses \(\bar C_i(s)\). Ambiguity is defined precisely by this noninvertibility: multiple values of \(s\) can produce the same mean output \(C_i\).

The objective is
\[
{\cal F}=I(\{C_i\};x)-\sum_{i=1}^K \lambda_i I(C_i;s),
\]
which maximizes information the population conveys about \(x\) while penalizing channel-wise information costs. In the low-noise regime, the effective local coding variable is
\[
\phi_i(s)=\frac{1}{\sigma_i(s)}\frac{d\bar C_i(s)}{ds},
\]
and the paper derives an optimality condition fixing \(\phi_i^2(s)\) by the upstream noise \(\sigma_s^2(s)\) and the allocation parameters \(\lambda_i\). After optimizing the noise levels, the population information becomes
\[
I(\{C_i\};x)=I(s;x)+\frac{1}{2}\log(1-\Lambda),
\qquad \Lambda=\sum_{j=1}^K \lambda_j.
\]
The central conclusion is that ambiguity or noninvertibility of individual channels does not directly reduce \(I(\{C_i\};x)\) once noise levels are optimized.

Where ambiguity matters is the cost side. For each channel, the entropy deficit \(\rho_i\) is defined by
\[
H[C_i]=H[s]+\left\langle \log \left|\frac{d\bar C_i(s)}{ds}\right|\right\rangle_s-\rho_i.
\]
If \(\bar C_i(s)\) is invertible, then \(\rho_i=0\); if it is folded or noninvertible, then \(\rho_i>0\). The channel information becomes
\[
I(C_i;s)=I(s;x)-\rho_i+\frac{1}{2}\log\!\left(\lambda_i\frac{1-\Lambda}{\Lambda^2}\right).
\]
Thus, for fixed relevant-information performance, ambiguity lowers the information burden on the individual channel by \(\rho_i\). The mechanism is not generic redundancy reduction alone, but complementary folding of the stimulus axis across channels so that single channels are ambiguous while the joint population is unambiguous. The paper states this explicitly: the whole set \(\{C_i\}\) must provide an unambiguous encoding of \(s\), so folding in one response has to be compensated by structure in the others.

For AEN theory, this provides a normative foundation for neurons that appear individually ambiguous but are efficient components of a distributed code. It does not identify AENs in biological or artificial systems. It does, however, justify the possibility that apparently confusing single-unit tuning curves—periodic, folded, non-monotonic, or mixed—can be optimal when single channels have limited information capacity and the population code is jointly decodable.

## 5. Cellular and dynamical mechanisms of ambiguity-sensitive computation

Mechanistic proposals for ambiguity-sensitive coding in biological systems emphasize either compartmental integration, temporal switching, or latent intracellular state. A dendritic account is provided by a model with basal sensory compartments and apical contextual compartments in pyramidal neurons [2312.05484]. The basal vector is \(\mathbf b=f(\mathbf x)\), the apical vector is
\[
\mathbf a=g(\mathbf c \oplus \mathbf U\sigma(\mathbf b)),
\]
and somatic output is computed by
\[
\mathbf h=\sigma(\mathbf b)\odot(\sigma(\mathbf a)+1).
\]
This “apical prior” makes top-down input a thresholded multiplicative gain modulator rather than a driving input. On the five-scenario contextual integration task, the model learns when to use context and when to ignore it. In MNIST with oracle context, ambiguous baseline accuracy is **46.3 \pm 0.3\%**, rising to **98.5 \pm 0.3\%** in ambig.Match, while ambiguous irrelevant context remains at **46.0 \pm 0.8\%** and contradictory context in unambiguous cases still yields **96.0 \pm 0.4\%**. The paper does not show dedicated neurons whose firing rate explicitly encodes ambiguity magnitude, but it does report sparse LRP-defined subpopulations relevant for ambiguous class pairs and states that the apical compartment encodes some information about sensory ambiguity and context relevance.

A different mechanistic picture is temporal alternation between competing stimulus codes [2410.00781]. In that framework, multiplexing arises under dual-stimulus conditions through competition between two latent drift-diffusion-like processes. The key latent variable is
\[
L_{ij}\in\{A,B\},
\]
which labels whether spike \(j\) in trial \(i\) was generated by the \(A\)-like or \(B\)-like latent process. The paper models ambiguity not as a scalar uncertainty code but as competition between representations, expressed as switching. Using inferior colliculus recordings in macaques, it reports that **18.8% (106)** of analyzed triplets are consistent with multiplexing under the strict category where competition is favored, with **16.0% (17/106)** slow switching and **84.0% (89/106)** fast switching. This is best interpreted as evidence for ambiguity-related representational instability rather than explicit AENs.

A more speculative single-cell proposal is the “multi-state neuron” [2512.08815]. There, a neuron has slower-changing internal states that determine which subset of synapses are functional at a given time. Controlling synapses trigger intracellular chemical signaling, those signals alter internal state, and the resulting state selectively deactivates controlled synapses. The paper argues that these state changes can occur on timescales of **seconds to minutes**, and it distinguishes fast spikes as “data” from slower state changes as “metadata.” This does not formalize ambiguity, uncertainty, or posterior distributions, nor does it provide experimental validation. Nevertheless, it offers an indirect mechanistic substrate for AEN-like behavior: the meaning of a neuron’s firing need not be fixed, but can depend on latent intracellular state, context, or current hypothesis.

## 6. Limitations, controversies, and open problems

The current literature does not yet support a single, uniform doctrine of AENs. The LLM evidence is the most direct, but it remains narrow: only **three instruction-tuned open-weight models**, only **two ambiguity datasets**, no token-level localization, no seed-stability study, and incomplete implementation detail for steering on pooled representations [2509.13664]. The strongest supported claim is therefore that certain neurons are highly predictive and causally effective for ambiguity-related behavior in those models and datasets, not that they are universal or monosemantic ambiguity detectors.

The NMT literature provides strong component-level support but weak neuron-level support. Encoder hidden states at ambiguous-token positions encode enough information to predict the correct translation sense with very high accuracy, and attention patterns differ systematically for ambiguous nouns, but the study does **not** identify specific neurons, hidden-state dimensions, sparse directions, or specialized heads [1908.11771]. The paper itself makes the implication explicit: to turn this into direct AEN evidence, one would need per-dimension probing, sparse linear probes, head-level analysis, and causal ablation or intervention.

Normative and mechanistic biological papers leave a different set of gaps. The information-theoretic theory of ambiguous signals is a low-noise continuous-channel model, not a spiking or learning model [2512.23531]. The dendritic model demonstrates ambiguity resolution by compartmental gain modulation but not dedicated explicit ambiguity-coding neurons [2312.05484]. The multiplexing model captures competition-driven switching rather than explicit uncertainty coding and would require ambiguity manipulations independent of stimulus identity, explicit latent uncertainty variables, simultaneous population recording, and behavioral correlation to establish true ambiguity encoding [2410.00781]. The multi-state-neuron proposal remains underformalized: it has no formal state-space model, no tested learning rule, no direct ambiguity variable, and no empirical demonstration [2512.08815].

A plausible synthesis is that AEN research currently spans three evidence levels. The first is **direct sparse neuron-level evidence**, where ambiguity is linearly decodable from a very small number of coordinates and perturbing those coordinates changes behavior. The second is **representation-level evidence**, where ambiguity information is readily available in hidden states or compartmental activations but not localized to individual neurons. The third is **normative or mechanistic support**, where ambiguity-sensitive coding is shown to be efficient or biologically plausible without direct identification of ambiguity-coding units. Future work that combines sparse localization, causal intervention, explicit uncertainty modeling, and population-level behavioral validation would be needed to move from AEN-like interpretations to a mature general theory of ambiguity-encoding neurons.

Source: https://www.emergentmind.com/topics/ambiguity-encoding-neurons-aens