Language Activation Probability Entropy

• LAPE is an entropy-based metric that quantifies the concentration versus dispersion of neuron activations across languages, distinguishing language-specific from language-agnostic units.
• It is applied to dense neurons, sparse autoencoder features, and token activation distributions to assess model specialization and uncertainty during decoding.
• Empirical findings show a U-shaped distribution of language-specific neurons and demonstrate LAPE's utility in steering model behavior and enhancing multilingual performance.

Searching arXiv for papers on Language Activation Probability Entropy and entropy-aligned decoding. Language Activation Probability Entropy (LAPE) is an entropy-based measure for quantifying concentration versus dispersion in activation behavior. In its original and most explicit formulation for multilingual LLMs, LAPE identifies language-specific neurons by converting a neuron’s activation probabilities across languages into a probability distribution and measuring its Shannon entropy: low entropy indicates that the neuron activates predominantly for one or a few languages, while high entropy indicates language-agnostic behavior (Tang et al., 2024). Later work applies the same idea to sparse autoencoder features, programming languages, and language-specific subnetworks, while a separate decoding literature uses the term or an equivalent mapping for the entropy of the LM’s token activation probability distribution at each generation step (Andrylie et al., 15 Jul 2025, Kargaran et al., 1 Jun 2025, Ahmed et al., 5 Jan 2026). Across these literatures, the common principle is unchanged: entropy is used as a compact scalar summary of selectivity, uncertainty, or specialization.

1. Core definition and mathematical form

In the multilingual neuron literature, LAPE is defined over feed-forward network neurons. For a neuron $n \equiv (i,j)$ and language $\ell$, the activation probability is the expected fraction of token positions in that language at which the neuron is active: $$p_{n}(\ell) \equiv p^{\ell}_{i,j} \;=\; \mathbb{E}\big[ \ \mathbb{I}\{\mathrm{act\_fn}\!\big(\tilde{\bm{h}^{i}}\bm{W}^{i}_{1}\big)_{j} > 0\} \ \big| \ \text{language } \ell \ \big].$$ Because the raw $p_n(\ell)$ values do not necessarily sum to 1 across languages, the vector is L1-normalized: $$\bm{p}_{n}' \;=\; \frac{\bm{p}_{n}}{\|\bm{p}_{n}\|_{1}},$$ and the LAPE score is then the entropy of $\bm{p}_n'$: $$\mathrm{LAPE}(n) \;=\; H\!\big(\bm{p}_{n}'\big) \;=\; -\sum_{\ell=1}^{L} p_{n}'(\ell) \, \log \big(p_{n}'(\ell)\big).$$ In implementation, the natural logarithm is used and no additional smoothing or bias correction is applied (Tang et al., 2024).

This same formal pattern recurs in later work. For language-specific neurons in multilingual LLMs, one commonly writes

$$p_l(h) = \mathbb{E}_{x \sim D_l} [ I(a_h(x) > 0) ], \qquad P_l(h) = p_l(h) / \sum_{l'} p_{l'}(h),$$

followed by

$LAPE(h) = -\sum_{l=1}^k P_l(h) \log P_l(h).$

A normalized variant,

$\hat{H}(h) = H(h) / \log k,$

is sometimes used for interpretability, mapping entropy to $\ell$0 (Mondal et al., 21 Mar 2025).

The intuition is stable across papers. A low-LAPE neuron is interpreted as language-specific; a high-LAPE neuron is interpreted as language-agnostic. This same intuition is later transferred from dense neurons to sparse features, and from natural languages to programming languages (Andrylie et al., 15 Jul 2025, Kargaran et al., 1 Jun 2025).

2. Original multilingual-neuron formulation

“Language-Specific Neurons: The Key to Multilingual Capabilities in LLMs” introduced LAPE as a novel detection method to identify language-specific neurons within LLMs (Tang et al., 2024). The study evaluates LLaMA‑2 (7B, 13B, 70B) and BLOOM (7.1B), uses Wikipedia corpora in seven languages—English, Simplified Chinese, French, Spanish, Vietnamese, Indonesian, and Japanese, with Japanese excluded for BLOOM-specific analyses—and samples 100 million tokens per language for LAPE estimation (Tang et al., 2024).

The paper labels neurons language-specific by two criteria. First, it applies low entropy selection: the bottom 1% of neurons by LAPE are selected as candidates. Second, for a neuron to be assigned to a particular language $\ell$1, it must have $\ell$2 above a global threshold $\ell$3, set to the 95th percentile of all activation probabilities across neurons and languages for the model. For LLaMA‑2‑70B, $\ell$4. A neuron can be shared by multiple languages if it exceeds $\ell$5 for each (Tang et al., 2024).

The main empirical pattern is a pronounced U-shaped distribution. Language-specific neurons concentrate at the bottom and top of the network, with middle layers containing far fewer such neurons. In LLaMA‑2‑70B, approximately 23k neurons are flagged as language-specific by LAPE, and the second layer alone accounts for roughly 7,000 language-specific neurons across languages, while the final four layers each exceed 1,000 (Tang et al., 2024). The paper interprets this as bottom layers transforming diverse scripts and segmentations into a shared semantic space, and top layers projecting semantics into language-specific vocabularies.

The paper also reports causal evidence. Deactivating language-specific neurons identified by LAPE induces a strong diagonal pattern of performance degradation: turning off Chinese-specific neurons primarily increases perplexity on Chinese, with much smaller effects on other languages. In open-ended multilingual generation on Vicuna-derived prompts, activating language-specific neurons improves the likelihood that the model responds in the intended language. For LLaMA‑2‑70B, language accuracy improves from $\ell$6 for Chinese, $\ell$7 for French, $\ell$8 for Spanish, $\ell$9 for Vietnamese, $$p_{n}(\ell) \equiv p^{\ell}_{i,j} \;=\; \mathbb{E}\big[ \ \mathbb{I}\{\mathrm{act\_fn}\!\big(\tilde{\bm{h}^{i}}\bm{W}^{i}_{1}\big)_{j} > 0\} \ \big| \ \text{language } \ell \ \big].$$0 for Indonesian, and $$p_{n}(\ell) \equiv p^{\ell}_{i,j} \;=\; \mathbb{E}\big[ \ \mathbb{I}\{\mathrm{act\_fn}\!\big(\tilde{\bm{h}^{i}}\bm{W}^{i}_{1}\big)_{j} > 0\} \ \big| \ \text{language } \ell \ \big].$$1 for Japanese (Tang et al., 2024). The same study notes that Chinese and Japanese exhibit cross-impact, consistent with shared characters, and reports that approximately 25% of Chinese and Japanese neurons overlap (Tang et al., 2024).

3. Variants, thresholds, and extensions beyond dense FFN neurons

Subsequent work preserves the entropy-over-languages template but modifies the activation event, thresholds, or representational basis. In “Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages,” SAE-LAPE applies LAPE to sparse autoencoder features rather than raw neurons (Andrylie et al., 15 Jul 2025). For a feature $$p_{n}(\ell) \equiv p^{\ell}_{i,j} \;=\; \mathbb{E}\big[ \ \mathbb{I}\{\mathrm{act\_fn}\!\big(\tilde{\bm{h}^{i}}\bm{W}^{i}_{1}\big)_{j} > 0\} \ \big| \ \text{language } \ell \ \big].$$2 and language $$p_{n}(\ell) \equiv p^{\ell}_{i,j} \;=\; \mathbb{E}\big[ \ \mathbb{I}\{\mathrm{act\_fn}\!\big(\tilde{\bm{h}^{i}}\bm{W}^{i}_{1}\big)_{j} > 0\} \ \big| \ \text{language } \ell \ \big].$$3,

$$p_{n}(\ell) \equiv p^{\ell}_{i,j} \;=\; \mathbb{E}\big[ \ \mathbb{I}\{\mathrm{act\_fn}\!\big(\tilde{\bm{h}^{i}}\bm{W}^{i}_{1}\big)_{j} > 0\} \ \big| \ \text{language } \ell \ \big].$$4

and

$$p_{n}(\ell) \equiv p^{\ell}_{i,j} \;=\; \mathbb{E}\big[ \ \mathbb{I}\{\mathrm{act\_fn}\!\big(\tilde{\bm{h}^{i}}\bm{W}^{i}_{1}\big)_{j} > 0\} \ \big| \ \text{language } \ell \ \big].$$5

Because TopK SAEs explicitly zero non-top-k values, the indicator $$p_{n}(\ell) \equiv p^{\ell}_{i,j} \;=\; \mathbb{E}\big[ \ \mathbb{I}\{\mathrm{act\_fn}\!\big(\tilde{\bm{h}^{i}}\bm{W}^{i}_{1}\big)_{j} > 0\} \ \big| \ \text{language } \ell \ \big].$$6 coincides with “feature selected in top-k for that token.” The paper identifies 540 language-specific features out of approximately 2 million features across layers, with most low-entropy features concentrating in layers roughly 8–15, and reports that low-LAPE features are more interpretable (Andrylie et al., 15 Jul 2025).

The thresholds had to be modified for sparsity. The original neuron LAPE threshold used by Tang et al. was not suitable for SAEs, so SAE-LAPE first applies a High-Frequency Latents filter: a feature must be active for more than 10% of tokens and at least 98% of examples in at least one language-specific corpus. A feature is then called specific to language $$p_{n}(\ell) \equiv p^{\ell}_{i,j} \;=\; \mathbb{E}\big[ \ \mathbb{I}\{\mathrm{act\_fn}\!\big(\tilde{\bm{h}^{i}}\bm{W}^{i}_{1}\big)_{j} > 0\} \ \big| \ \text{language } \ell \ \big].$$7 if

$$p_{n}(\ell) \equiv p^{\ell}_{i,j} \;=\; \mathbb{E}\big[ \ \mathbb{I}\{\mathrm{act\_fn}\!\big(\tilde{\bm{h}^{i}}\bm{W}^{i}_{1}\big)_{j} > 0\} \ \big| \ \text{language } \ell \ \big].$$8

and language-specific features are selected only when they are specific to just one language (Andrylie et al., 15 Jul 2025).

“Multilingual LLMs Encode Script Over Linguistic Structure” extends LAPE to both dense neurons and sparse features while using it to study romanization and word-order shuffling (Verma et al., 6 Apr 2026). For raw neurons, the paper filters by the global 95th percentile of activation probabilities across all neurons and languages, discards neurons whose activation probability never exceeds this threshold in any language, and selects the top 1% lowest-entropy neurons. For SAE latents, it imposes minimum example rate $$p_{n}(\ell) \equiv p^{\ell}_{i,j} \;=\; \mathbb{E}\big[ \ \mathbb{I}\{\mathrm{act\_fn}\!\big(\tilde{\bm{h}^{i}}\bm{W}^{i}_{1}\big)_{j} > 0\} \ \big| \ \text{language } \ell \ \big].$$9, high-frequency latent rate $p_n(\ell)$0, and a relative top-k membership rule,

$p_n(\ell)$1

The paper finds that romanization induces near-disjoint representations that align with neither native-script inputs nor English, while word-order shuffling has limited effect on unit identity (Verma et al., 6 Apr 2026).

The same entropy logic has also been applied to programming languages. “How Programming Concepts and Neurons Are Shared in Code LLMs” adapts LAPE from natural languages to English and 11 programming languages, defining per-neuron language activation probabilities over code corpora and then measuring entropy across languages (Kargaran et al., 1 Jun 2025). The paper reports that most selected language-specific neurons are concentrated in bottom layers, but neurons exclusive to each programming language tend to appear in the top layers. For programming languages that are highly aligned with multiple other programming languages, identifying language-specific neurons is not feasible (Kargaran et al., 1 Jun 2025).

4. Steering, adaptation, and the limits of neuron-level control

LAPE-based neuron sets have been used for inference-time steering, parameter-efficient adaptation, and causal analyses. “Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation” studies Llama-3.1-8B, Mistral-Nemo-12B, and Aya-Expanse-8B & 32B across 21 languages and reports that these neurons cluster in deeper layers, with non-Latin scripts showing greater specialization (Gurgurov et al., 30 Jul 2025). In Llama-3.1 at top-1% selection, the paper reports $p_n(\ell)$2, $p_n(\ell)$3, $p_n(\ell)$4, versus $p_n(\ell)$5, $p_n(\ell)$6, $p_n(\ell)$7. It then performs “language arithmetics,” defined as systematic activation addition and multiplication, and finds that additive Deactivate+Activate improves Llama-3.1-8B language-forcing success from $p_n(\ell)$8 at top-1% to $p_n(\ell)$9 at top-5%, outperforming replacement-based methods (Gurgurov et al., 30 Jul 2025).

A more adaptation-oriented use appears in “Sparse Subnetwork Enhancement for Underrepresented Languages in LLMs” (Gurgurov et al., 15 Oct 2025). That paper computes LAPE on FFN neurons using a positivity threshold of $$\bm{p}_{n}' \;=\; \frac{\bm{p}_{n}}{\|\bm{p}_{n}\|_{1}},$$0, selects the lowest $$\bm{p}_{n}' \;=\; \frac{\bm{p}_{n}}{\|\bm{p}_{n}\|_{1}},$$1 of LAPE scores, imposes activity thresholds of $$\bm{p}_{n}' \;=\; \frac{\bm{p}_{n}}{\|\bm{p}_{n}\|_{1}},$$2, and then fine-tunes only the weights associated with the selected neurons for a target language. On Llama-3.1-8B, averaged over 12 mid- and low-resource languages, the reported target-language performance average is $$\bm{p}_{n}' \;=\; \frac{\bm{p}_{n}}{\|\bm{p}_{n}\|_{1}},$$3 for target neurons, compared with $$\bm{p}_{n}' \;=\; \frac{\bm{p}_{n}}{\|\bm{p}_{n}\|_{1}},$$4 for the base model, $$\bm{p}_{n}' \;=\; \frac{\bm{p}_{n}}{\|\bm{p}_{n}\|_{1}},$$5 for full fine-tuning, $$\bm{p}_{n}' \;=\; \frac{\bm{p}_{n}}{\|\bm{p}_{n}\|_{1}},$$6 for FFN-only fine-tuning, $$\bm{p}_{n}' \;=\; \frac{\bm{p}_{n}}{\|\bm{p}_{n}\|_{1}},$$7 for LoRA, and $$\bm{p}_{n}' \;=\; \frac{\bm{p}_{n}}{\|\bm{p}_{n}\|_{1}},$$8 for a random subset. At the same time, general capabilities average is $$\bm{p}_{n}' \;=\; \frac{\bm{p}_{n}}{\|\bm{p}_{n}\|_{1}},$$9 for target neurons versus $\bm{p}_n'$0 for the base model, while full fine-tuning drops to $\bm{p}_n'$1 (Gurgurov et al., 15 Oct 2025). The paper states that the approach updates only up to 1% of model parameters.

The most important counterpoint comes from “Language-specific Neurons Do Not Facilitate Cross-Lingual Transfer” (Mondal et al., 21 Mar 2025). That study evaluates LAPE-selected neurons and activation probability-based thresholding on Llama 3.1 and Mistral Nemo for zero-shot XNLI and XQuAD. Its conclusion is negative: test-time interventions based on LAPE or Act Prob 90p generally do not help, and neuron-specific LoRA fine-tuning did not improve zero-shot transfer. The paper attributes this to polysemanticity/superposition, distributed language signals and cross-layer interactions, and methodological sensitivity to the chosen language set (Mondal et al., 21 Mar 2025).

Taken together, these findings do not support a single universal claim about LAPE-selected units. The positive results concern steering of output language, monolingual enhancement, and causal manipulation of language behavior (Tang et al., 2024, Gurgurov et al., 30 Jul 2025, Gurgurov et al., 15 Oct 2025). The negative results concern zero-shot cross-lingual NLU transfer (Mondal et al., 21 Mar 2025). This suggests that LAPE can isolate useful language-associated structure without implying that those units are a reliable lever for cross-lingual generalization.

5. Decoding-time reinterpretation: entropy of token activation probabilities

A separate line of work uses “Language Activation Probability Entropy” to denote the entropy of the model’s conditional distribution over the vocabulary at each generation step. In the context of LLM decoding, “activation probability” refers to the model’s conditional distribution over the vocabulary at generation step $\bm{p}_n'$2, given the input $\bm{p}_n'$3 and past tokens $\bm{p}_n'$4. The corresponding per-step entropy is

$\bm{p}_n'$5

This quantity is exactly “the entropy of the LM’s token activation probability distribution at each step” (Ahmed et al., 5 Jan 2026).

“Entropy-Aligned Decoding of LMs for Better Writing and Reasoning” visualizes this quantity over 15 generation steps and highlights a “typical” entropy band roughly between $\bm{p}_n'$6 and $\bm{p}_n'$7 nats (Ahmed et al., 5 Jan 2026). The figure associates entropy collapse with repetitive output, entropy rising well above the typical band with incoherent output, and moderate, stable entropy with coherent, fluent text. The paper introduces EPIC, a hyperparameter-free decoding approach that incorporates the entropy of future trajectories into LM decoding, explicitly regulates the amount of uncertainty expressed at every step of generation, and aligns the sampling distribution’s entropy to the aleatoric uncertainty. Through Entropy-Aware Lazy Gumbel-Max sampling, EPIC is described as exact and efficient, requiring only a sublinear number of entropy evaluations per step (Ahmed et al., 5 Jan 2026).

A related training-time use appears in “Entropy Regularizing Activation,” where the paper defines LAPE for token distributions as

$\bm{p}_n'$8

and, for LLM reinforcement learning, focuses on “response entropy” $\bm{p}_n'$9, the average entropy over the top 20% highest-entropy tokens in the generated response (Kang et al., 9 Oct 2025). ERA regulates $$\mathrm{LAPE}(n) \;=\; H\!\big(\bm{p}_{n}'\big) \;=\; -\sum_{\ell=1}^{L} p_{n}'(\ell) \, \log \big(p_{n}'(\ell)\big).$$0 to stay within $$\mathrm{LAPE}(n) \;=\; H\!\big(\bm{p}_{n}'\big) \;=\; -\sum_{\ell=1}^{L} p_{n}'(\ell) \, \log \big(p_{n}'(\ell)\big).$$1 during training. For Qwen2.5-Math-7B, the paper reports a two-stage schedule, and on AIME’25 reports $$\mathrm{LAPE}(n) \;=\; H\!\big(\bm{p}_{n}'\big) \;=\; -\sum_{\ell=1}^{L} p_{n}'(\ell) \, \log \big(p_{n}'(\ell)\big).$$2 for ERA versus $$\mathrm{LAPE}(n) \;=\; H\!\big(\bm{p}_{n}'\big) \;=\; -\sum_{\ell=1}^{L} p_{n}'(\ell) \, \log \big(p_{n}'(\ell)\big).$$3 for GRPO, a $$\mathrm{LAPE}(n) \;=\; H\!\big(\bm{p}_{n}'\big) \;=\; -\sum_{\ell=1}^{L} p_{n}'(\ell) \, \log \big(p_{n}'(\ell)\big).$$4 relative gain (Kang et al., 9 Oct 2025).

The broader entropy picture is complemented by “Entropy in LLMs,” which treats LLM output as an information source and estimates entropy rates from generated text (Scharringhausen, 23 Feb 2026). That paper reports lower entropy for LLM-generated text than for OANC written and spoken language, with reported values of $$\mathrm{LAPE}(n) \;=\; H\!\big(\bm{p}_{n}'\big) \;=\; -\sum_{\ell=1}^{L} p_{n}'(\ell) \, \log \big(p_{n}'(\ell)\big).$$5 for OANC written, $$\mathrm{LAPE}(n) \;=\; H\!\big(\bm{p}_{n}'\big) \;=\; -\sum_{\ell=1}^{L} p_{n}'(\ell) \, \log \big(p_{n}'(\ell)\big).$$6 for OANC spoken, $$\mathrm{LAPE}(n) \;=\; H\!\big(\bm{p}_{n}'\big) \;=\; -\sum_{\ell=1}^{L} p_{n}'(\ell) \, \log \big(p_{n}'(\ell)\big).$$7 for “LLM accumulated, $$\mathrm{LAPE}(n) \;=\; H\!\big(\bm{p}_{n}'\big) \;=\; -\sum_{\ell=1}^{L} p_{n}'(\ell) \, \log \big(p_{n}'(\ell)\big).$$8,” and $$\mathrm{LAPE}(n) \;=\; H\!\big(\bm{p}_{n}'\big) \;=\; -\sum_{\ell=1}^{L} p_{n}'(\ell) \, \log \big(p_{n}'(\ell)\big).$$9 for “LLM accumulated, $$p_l(h) = \mathbb{E}_{x \sim D_l} [ I(a_h(x) > 0) ], \qquad P_l(h) = p_l(h) / \sum_{l'} p_{l'}(h),$$0” (Scharringhausen, 23 Feb 2026). In this decoding-oriented usage, LAPE functions as a diagnostic of uncertainty, diversity, and collapse rather than as a cross-language selectivity measure.

6. Broader entropy frameworks and conceptual significance

Outside multilingual-neuron analysis and decoding, LAPE has been generalized to other activation distributions. In “Mathematically Modeling the Lexicon Entropy of Emergent Language,” LAPE is interpreted as the Shannon entropy of the activation probability distribution over messages or words produced by agents in an emergent communication system (Boldt et al., 2022). With a lexicon of size $$p_l(h) = \mathbb{E}_{x \sim D_l} [ I(a_h(x) > 0) ], \qquad P_l(h) = p_l(h) / \sum_{l'} p_{l'}(h),$$1 and message probabilities $$p_l(h) = \mathbb{E}_{x \sim D_l} [ I(a_h(x) > 0) ], \qquad P_l(h) = p_l(h) / \sum_{l'} p_{l'}(h),$$2,

$$p_l(h) = \mathbb{E}_{x \sim D_l} [ I(a_h(x) > 0) ], \qquad P_l(h) = p_l(h) / \sum_{l'} p_{l'}(h),$$3

The paper’s FiLex stochastic process predicts directional effects of training steps, lexicon size, learning rate, rollout buffer size, and Gumbel-Softmax temperature on entropy, and the reported outcome is that 20 out of 20 environment–hyperparameter combinations match the predicted sign (Boldt et al., 2022).

A language-level extension appears in “Entropy2Vec: Crosslingual Language Modeling Entropy as End-to-End Learnable Language Representations” (Irawan et al., 5 Sep 2025). That work does not define LAPE explicitly, but it constructs a cross-lingual entropy matrix $$p_l(h) = \mathbb{E}_{x \sim D_l} [ I(a_h(x) > 0) ], \qquad P_l(h) = p_l(h) / \sum_{l'} p_{l'}(h),$$4 from monolingual LLMs and defines a language vector

$$p_l(h) = \mathbb{E}_{x \sim D_l} [ I(a_h(x) > 0) ], \qquad P_l(h) = p_l(h) / \sum_{l'} p_{l'}(h),$$5

The paper states that LAPE follows naturally by mapping per-language entropies into activation probabilities and computing their entropy (Irawan et al., 5 Sep 2025). This suggests a shift from unit-level selectivity toward language-level uncertainty profiles.

A longer information-theoretic background is provided by “The word entropy of natural languages,” which estimates block and source entropy across parallel texts and across 1001 languages (Bentz et al., 2016). That paper is not a LAPE paper, but it supplies the language-entropy tradition on which later entropy-based multilingual analyses build. It reports approximately normally distributed block entropies with mean $$p_l(h) = \mathbb{E}_{x \sim D_l} [ I(a_h(x) > 0) ], \qquad P_l(h) = p_l(h) / \sum_{l'} p_{l'}(h),$$6 and SD $$p_l(h) = \mathbb{E}_{x \sim D_l} [ I(a_h(x) > 0) ], \qquad P_l(h) = p_l(h) / \sum_{l'} p_{l'}(h),$$7, source entropies with mean $$p_l(h) = \mathbb{E}_{x \sim D_l} [ I(a_h(x) > 0) ], \qquad P_l(h) = p_l(h) / \sum_{l'} p_{l'}(h),$$8 and SD $$p_l(h) = \mathbb{E}_{x \sim D_l} [ I(a_h(x) > 0) ], \qquad P_l(h) = p_l(h) / \sum_{l'} p_{l'}(h),$$9, and a strong correlation between block and source entropies with Pearson $LAPE(h) = -\sum_{l=1}^k P_l(h) \log P_l(h).$0 (Bentz et al., 2016).

Across these literatures, the significance of LAPE is not that it names one invariant metric with a single object of measurement. Rather, it designates a recurrent information-theoretic construction: define an activation probability distribution over a relevant support—languages, tokens, features, or messages—and use Shannon entropy to measure how concentrated or uncertain that distribution is. In multilingual mechanistic interpretability, this yields a tractable criterion for language-specific neurons and features (Tang et al., 2024, Andrylie et al., 15 Jul 2025). In decoding, it becomes a measure of local uncertainty and a control target for generation (Ahmed et al., 5 Jan 2026, Kang et al., 9 Oct 2025). In emergent communication and cross-lingual entropy modeling, it becomes a compact description of symbolic or language-level uncertainty (Boldt et al., 2022, Irawan et al., 5 Sep 2025). The resulting body of work shows that entropy can serve as a common currency for specialization, selectivity, and uncertainty, even when the underlying object of “activation probability” changes across domains.