Approximate Question-side Effect (AQE)

• Approximate Question-side Effect (AQE) is a diagnostic measure that quantifies the proportion of hallucination prediction performance attributable solely to question-side cues.
• It decomposes LLM hidden states into question-side (s_Q) and model-side (s_M) components using an approximate additive scheme and Shapley-style analysis.
• AQE is practically estimated using a secondary encoder like sBERT, revealing that significant predictive accuracy can arise from superficial question-aware patterns.

Approximate Question-side Effect (AQE) is a diagnostic quantity for hallucination prediction in LLMs that estimates how much predictive performance can be obtained from question-side information alone, rather than from the model’s own internal knowledge state. Introduced in work on “self-awareness of knowledge,” AQE addresses the possibility that high hallucination-prediction scores may reflect question-awareness—such as sensitivity to domain, question type, or other superficial regularities in prompts—rather than genuine model-side introspection. In this formulation, AQE is the question-side-only component of performance, and the complementary model-side component is approximated by subtracting AQE from the performance of a predictor that uses the full hidden state (Seo et al., 18 Sep 2025).

1. Conceptual basis: question-awareness versus self-awareness

The AQE framework begins by casting hallucination prediction as a binary classification problem over a model’s future answer correctness. Let $\theta$ be an LLM, $x$ a question, $y$ its ground-truth answer, and $\hat{y}=\theta(x)$ the model’s answer. A binary variable $k \in \{0,1\}$ indicates correctness, with $k=1$ when $\hat{y}$ matches $y$ and $k=0$ otherwise. During inference, one extracts an internal representation $s_{\theta,x}= \theta(x)$, denoted $x$0, typically a hidden state vector at the first answer token, and trains a classifier

$x$1

The central claim of the framework is that $x$2 mixes two qualitatively different sources of information:

$x$3

Here $x$4 denotes question-side information, including domain, question type, superficial patterns, and other cues that are about the question itself and are invariant across models. By contrast, $x$5 denotes model-side information: the model’s own state regarding the question, such as how much relevant knowledge it has or its internal confidence. The paper interprets self-awareness of knowledge as prediction based solely on the latter,

$x$6

and argues that prior hallucination-prediction results often conflate this with question-awareness (Seo et al., 18 Sep 2025).

This distinction matters because question-side regularities can be highly predictive without requiring any introspective access to the model’s epistemic state. A predictor may succeed by learning that certain domains, answer formats, or prompt structures are systematically easier or harder, even if it has no access to whether the model actually “knows” the answer. AQE was introduced to quantify precisely this confound.

2. Formal definition and decomposition

Let $x$7 denote an evaluation metric for hallucination prediction, specifically accuracy or AUROC. The framework considers the performance of a classifier on the full hidden state,

$x$8

and contrasts it with the quantities of conceptual interest,

$x$9

The paper assumes an approximate additive decomposition,

$y$0

which yields

$y$1

Within this scheme, AQE is defined as the question-side-only term,

$y$2

The remaining gap is interpreted as the self-awareness component.

The paper also motivates this decomposition through a Shapley-style marginal-contribution analysis. Using

$y$3

one identifies $y$4 with model-side information $y$5, $y$6 with question-side information $y$7, and $y$8 with a performance functional such as AUROC. In this view, AQE is $y$9, and the model-side contribution is the marginal gain obtained by adding $\hat{y}=\theta(x)$0 on top of $\hat{y}=\theta(x)$1 (Seo et al., 18 Sep 2025).

This formalization is intentionally approximate rather than exact. The paper does not claim a perfect disentanglement of question-side and model-side information; instead, AQE is presented as a practical quantity that is sufficiently informative to detect when benchmark performance is dominated by question-side shortcuts.

3. Practical estimation procedure

AQE is not computed by directly isolating $\hat{y}=\theta(x)$2 from the hidden states of the main LLM. Instead, the paper proposes an external approximation using a second, smaller model $\hat{y}=\theta(x)$3. Given a question $\hat{y}=\theta(x)$4, this auxiliary encoder produces an embedding

$\hat{y}=\theta(x)$5

which is taken to approximate question-side information:

$\hat{y}=\theta(x)$6

The motivating example uses Sentence-BERT as $\hat{y}=\theta(x)$7, with approximately 22.7M parameters. Because this encoder is trained for generic sentence embeddings and has very limited factual knowledge compared with the main LLM $\hat{y}=\theta(x)$8, the paper argues that a classifier $\hat{y}=\theta(x)$9 trained on $k \in \{0,1\}$0 should primarily exploit question-side patterns rather than answer knowledge. Under this assumption,

$k \in \{0,1\}$1

AQE is then instantiated as the accuracy or AUROC of $k \in \{0,1\}$2, and the self-awareness component is approximated by

$k \in \{0,1\}$3

The reported results use sBERT as $k \in \{0,1\}$4 and LLaMA-3-8B-Instruct as the main model $k \in \{0,1\}$5 (Seo et al., 18 Sep 2025).

A notable restriction is that AQE is defined for hidden-state-based predictors and related fusion methods that include hidden states. The paper states that when $k \in \{0,1\}$6 is a confidence score, AQE cannot be directly applied because confidence score is a one-dimensional value, which is too saturated to embed high-level information of the question. Consequently, pure confidence-based methods are treated as largely model-side, but their question-side contribution is not quantified through the same AQE procedure.

4. Dataset artifacts, shortcut structure, and empirical values

AQE is used to diagnose shortcut structure in several hallucination-prediction benchmarks. The studied datasets include short-form factoid QA benchmarks—ParaRel, Mintaka, HaluEval, HotpotQA, and SimpleQuestion—as well as Explain, a long-form factoid dataset introduced by the authors. The paper identifies three prominent classes of question-side shortcut. The first is broken questions or incomplete labeling, as in one-to-many relations with only one labeled answer, which can make valid alternatives appear hallucinated. The second is domain bias, where some domains are systematically easier or harder for the model. The third is question type bias, especially the elevated correctness rate of binary or multiple-choice items relative to open-ended questions (Seo et al., 18 Sep 2025).

The original-dataset AQE values reported for sBERT predicting the correctness of LLaMA-3-8B-Instruct are as follows:

Dataset	AQE$k \in \{0,1\}$7	AQE$k \in \{0,1\}$8
ParaRel	0.8261	0.7326
Mintaka	0.6667	0.6350
HaluEval	0.6837	0.6855
HotpotQA	0.7014	0.7250
SimpleQuestion	0.6813	0.8236
Explain	0.6940	0.6565

These values imply that a small external encoder that never sees the main model’s hidden state can often predict correctness with AUROC near $k \in \{0,1\}$9–$k=1$0. The paper therefore concludes that a model can easily achieve AUROC over $k=1$1 without any self-awareness, relying solely on question-aware skills such as domain classification. In this sense, high raw hallucination-prediction performance is not, by itself, evidence of introspective knowledge assessment.

The authors also evaluate refined benchmark variants intended to reduce shortcut availability. Examples include removing boolean or numerical questions from Mintaka, removing the “comparison” type from HotpotQA, and enforcing disjoint train/test domain splits for ParaRel and Explain. AQE$k=1$2 drops under these refinements—for example, Mintaka declines from $k=1$3 to $k=1$4, HotpotQA from $k=1$5 to $k=1$6, ParaRel from $k=1$7 to $k=1$8, and Explain from $k=1$9 to $\hat{y}$0—but does not disappear. The paper interprets this as evidence that known type and domain shortcuts explain a substantial part of performance, while residual question-side effects remain even after explicit controls (Seo et al., 18 Sep 2025).

5. Relationship to SCAO and model-side probing

AQE is closely linked to SCAO, or Semantic Compression by Answering in One word. SCAO modifies the prompt from a standard question-answer template,

$\hat{y}$1

to a constrained one-word template,

$\hat{y}$2

The purpose is not to improve answer quality per se, but to change the information content of the first answer token so that confidence and hidden-state signals more directly reflect retrieval of relevant knowledge (Seo et al., 18 Sep 2025).

The paper motivates SCAO through an analogy between a causal LLM and a dense retriever. In that analogy, the last hidden state at the pre-answer position acts like a query embedding, the vocabulary embedding matrix acts like a document index, and logits or softmax probabilities act like similarity scores. Under ordinary prompting, however, the first token is often dominated by grammatical artifacts or simple copying of question content. By forcing a one-word answer, SCAO reduces the need to generate grammatical scaffolding and makes the first token more like an entity-retrieval decision.

The empirical observations on Explain are explicit. Under normal prompts, $\hat{y}$3 of answers start by repeating the subject entity and $\hat{y}$4 begin with “the.” Under one-word prompts, only $\hat{y}$5 repeat the subject and only $\hat{y}$6 begin with “the.” The paper interprets this as evidence that SCAO suppresses grammatical and echoing shortcuts and pushes the model toward knowledge-driven retrieval patterns. In refined, lower-AQE settings, SCAO variants improve both total AUROC and the estimated self-awareness component. For example, on refined Mintaka (“+type+domain”), Conf+Probe(SCAO) reaches AUROC $\hat{y}$7 with $\hat{y}$8, compared with AUROC $\hat{y}$9 and $y$0 for Conf+Probe without SCAO; on refined HotpotQA (“+type”), Conf+Probe(SCAO) reaches AUROC $y$1 with $y$2, exceeding the corresponding non-SCAO variants (Seo et al., 18 Sep 2025).

This suggests that AQE and SCAO play complementary roles. AQE diagnoses how much apparent hallucination-prediction skill can be explained by question-side cues alone, whereas SCAO is designed to amplify model-side signals, especially in confidence-based methods that are less able to encode high-level question patterns.

6. Interpretation, limitations, and acronym ambiguity

AQE reframes hallucination prediction as a confounded proxy for self-awareness. Without such a decomposition, an AUROC above $y$3 could be interpreted as strong introspective ability, even though a substantial share of that performance may be obtainable from question-side shortcuts alone. Under the AQE interpretation, benchmark evaluation becomes a two-part question: how much performance comes from question-aware classification, and how much remains after that contribution is removed approximately (Seo et al., 18 Sep 2025).

The framework also has explicit assumptions and limitations. It assumes approximate linear separability and additivity between $y$4 and $y$5; in practice, their interaction may be more complex. It assumes that the auxiliary encoder $y$6 captures question-side information similar to that present in the main model’s hidden states, despite being in a different representational space. It further assumes that $y$7 has negligible overlap with the main model’s answer knowledge. Finally, AQE applies only to hidden-state-based or fusion predictors, not to pure confidence-based methods, and the reported study concentrates on short-form factoid QA plus the long-form factoid Explain dataset rather than reasoning-heavy or multi-step tasks (Seo et al., 18 Sep 2025).

A related source of confusion is acronym ambiguity. In the LLM literature discussed here, AQE denotes Approximate Question-side Effect (Seo et al., 18 Sep 2025). The same acronym also names “Aggressive Q-Learning with Ensembles” in deep reinforcement learning (Wu et al., 2021) and “Asymptotic-Quantized Estimator” in quantized neural networks (Chen et al., 2020). In approximate query processing, the ELECTRA work frames an “Approximate Question-side Effect” in a different sense, namely how query complexity, such as many predicates and complex WHERE clauses, affects approximation error and how predicate-aware modeling can control that degradation (Sheoran et al., 2022). This overlap in nomenclature does not indicate conceptual equivalence; it reflects independent acronym reuse across subfields.

A plausible implication is that AQE is best understood not as a standalone performance metric but as a benchmark-diagnostic instrument. In that role, it functions as a health check for hallucination datasets, a caution against over-interpreting raw AUROC, and a guide for designing evaluations that minimize question-side shortcuts while preserving access to genuine model-side signals.