Semantic Faithfulness Metric

• Semantic Faithfulness (SF) is defined by minimizing the KL divergence between modeled answer and query transitions, ensuring LLM outputs closely match the intended semantic intent.
• The metric utilizes convex optimization over stochastic matrices to capture topic-level transitions, providing a normalized score for evaluation.
• Paired with Semantic Entropy Production (SEP), SF offers insights into irreversibility and guides real-time LLM evaluation, prompt engineering, and governance.

Semantic Faithfulness (SF) metrics quantify the degree to which a LLM’s (LLM) outputs adhere to the semantic intent of given prompts and contexts, providing a principled, unsupervised approach to evaluating model faithfulness and detecting hallucinations. SF is closely linked to information-theoretic and thermodynamic perspectives, treating LLMs as information engines that realize context-to-answer transformations subject to entropic constraints. The SF metric, its companion entropy-production scores, and related methods represent a rigorous framework at the intersection of information theory, stochastic thermodynamics, and unsupervised evaluation of LLM behavior (Halperin, 4 Dec 2025).

1. Foundations and Information-Theoretic Modeling

The SF framework conceptualizes the LLM as a bipartite information engine, where the model’s hidden layers function analogously to Maxwell’s demon, orchestrating transformations from a given context ($C$) to an answer ($A$) in response to a query or prompt ($Q$). The workflow is formalized as a Question-Context-Answer (QCA) triplet, each element being represented as a probability distribution over $N$ latent semantic topics, often constructed via clustering of sentence embeddings (Halperin, 4 Dec 2025).

The semantic transitions in the QCA triplet are modeled by:

• $Q$-dynamics: $p^{(q)} = p^{(c)T} Q$ ("goal" transition from context to prompt intent)
• $A$-dynamics: $p^{(a)} = p^{(c)T} A$ (realized answer transition from context to answer)

Here, $Q$ and $A$ are $N \times N$ row-stochastic matrices encoding transition probabilities between topic distributions.

2. Definition and Convex Optimization of the Semantic Faithfulness Metric

Semantic Faithfulness ($\mathcal{F}_S$) is defined on a single QCA triplet. The metric is computed via the Kullback-Leibler divergence between the realized answer transition matrix $A$ and the query goal matrix $Q$, minimized over admissible choices of $A$ and $Q$:

$$D(A\,\|\,Q) = \sum_{i,j} p_i^{(c)} A_{ij} \ln \frac{A_{ij}}{Q_{ij}}$$

This minimization is carried out under two key constraints:

1. Row-stochasticity: $\sum_j A_{ij} = 1$, $\sum_j Q_{ij} = 1$ for all $i$
2. Marginal-matching: $\sum_i p_i^{(c)} A_{ij} = p_j^{(a)}$, $\sum_i p_i^{(c)} Q_{ij} = p_j^{(q)}$

The joint convexity of $D(A\,\|\,Q)$ with respect to $(A, Q)$ permits efficient solution using variants of the Blahut–Arimoto or Csiszár–Tusnády alternating minimization algorithms.

The final SF score is normalized to $[0, 1]$ as:

$\mathcal{F}_S = \frac{1}{1 + D_{\min}}$

where $D_{\min} = \min_{A, Q} D(A\,\|\,Q)$. High SF values indicate that the actual topic transformation realized by the LLM closely matches the semantic intent articulated by the prompt, signifying high faithfulness.

3. Relationship to Entropy Production and Thermodynamics

The SF metric is intrinsically connected to a thermodynamic notion termed Semantic Entropy Production (SEP). In this analogy, the context-to-answer mapping via the LLM is interpreted as a non-equilibrium stochastic process; SEP quantifies the irreversibility—or semantic "noise"—incurred during this process (Halperin, 4 Dec 2025).

For topic distributions $p_i^{(c)}, p_j^{(a)}$, forward matrix $A_{ij}$, and time-reversed transition $A^R_{ji}$ satisfying marginal and stochasticity constraints, the total entropy production for one step is:

$$\Sigma_{\rm tot} = \sum_{i,j} p_i^{(c)} A_{ij} \ln \frac{A_{ij}}{A^R_{ji}} + H[p^{(a)}] - H[p^{(c)}]$$

SEP is then defined as the minimal KL divergence between $A$ and any admissible $A^R$:

$$\mathrm{SEP} = \min_{A^R} \sum_{i,j} p_i^{(c)} A_{ij} \ln \frac{A_{ij}}{A^R_{ji}} = D(A\,\|\,A^R_*)$$

where $A^R_*$ meets the specified stochastic constraints.

Empirically, SF and SEP are negatively correlated, with high faithfulness associated with low entropy production. However, they are not redundant; SEP uniquely captures facets of irreversibility and topic drift not reflected by SF alone.

4. Algorithmic Computation

The SF metric is computed in practice through a sequence of convex optimizations:

1. Inference of $A$ and $Q$ matrices, simultaneously minimizing $D(A\,\|\,Q)$ under the specified constraints.
2. Alternating updates for the Lagrange multipliers (denoted as $\xi_i$, $\nu_j$) implementable via block coordinate ascent, enabling rapid convergence.
3. SEP is similarly obtained by optimizing over the set of time-reversed dynamics.

A summary of computational steps is as follows:

Step	Operation	Purpose
1. Initialization	Set $\xi_i$, $\nu_j > 0$	Lagrange multipliers for optimization
2. $\xi$-step	Maximize $L$ in $\xi_i$	Update forward multiplier
3. $\nu$-step	Maximize $L$ in $\nu_j$	Update reverse multiplier
4. Construct $A^R$	$A^R_{ji} = A^*_{ij}/(\xi_i+\nu_j)$	Build time-reversed transition matrix
5. Compute SEP	$$\sum_{i,j} p_i^{(c)} A^*_{ij} \ln \frac{A^*_{ij}}{A^R_{ji}}$$	Evaluate total entropy production

5. Empirical Application and Interpretation

The SF and SEP metrics have been evaluated on LLM-based summarization tasks, such as extracting risk factors from SEC 10-K filings. In comparative experiments, higher SEP values were observed for broader prompts, indicating greater semantic expansion and "noisier" topic drifts, while targeted prompts produced lower SEP, reflecting more controlled, faithful topic transitions (Halperin, 4 Dec 2025).

The decomposition of SEP further revealed regimes where medium-dissipated heat was negative, suggesting that the LLM could "absorb" knowledge from its internal state to partially offset irreversibility, a distinctive thermodynamic signature.

6. Downstream Applications in LLM Evaluation and Control

SF and SEP are directly applicable to a range of LLM evaluation and governance settings:

• Candidate Answer Ranking: For $K$ alternative model completions, SEP is computed for each; the answer with lowest SEP (most "reversible") is selected to reduce hallucination risk.
• Real-Time Monitoring: Anomalous SEP spikes can be used to trigger human review or initiate fallback procedures.
• Reinforcement Learning Fine-Tuning: SEP can be incorporated as an auxiliary negative reward in RLHF, explicitly incentivizing entropy-minimizing (faithful) generations.
• Prompt Engineering: SEP heatmaps over paraphrase variations identify prompt formulations that minimize entropy production, guiding practitioners toward more controllable semantics.
• Dashboard and Governance: Both metrics support longitudinal and cross-domain auditing of LLM faithfulness in production environments.

7. Comparative and Theoretical Perspectives

The SF metric distinguishes itself from token-level measures (e.g., perplexity, predictive entropy) by fundamentally operating at the meaning (topic) level. It leverages structured topic transitions and their divergences to capture not only the lexical but also the thematic fidelity of model outputs to user intent. This semantic viewpoint affords a more granular, theoretical foundation for faithfulness analysis than string-level or surface-level metrics.

SEP and SF collectively formalize a duality between semantic alignment (faithfulness) and generative irreversibility (entropy production), establishing connections to both classical information theory and modern stochastic thermodynamics in the context of LLMs (Halperin, 4 Dec 2025). This synergy between theory and practical evaluation grounds the metrics in both interpretability and operational utility for LLM deployment and safety engineering.