---
title: Question Answerability Classification (AC)
url: https://www.emergentmind.com/topics/question-answerability-classification-ac
type: topic
---

# Question Answerability Classification (AC)

Question Answerability Classification (AC) is a task that formalizes the prediction of whether a given question, within the context of a particular system or evidence set, is likely to be (fully) answerable. Variants arise across community Q&A, conversational QA, information-seeking dialogue, document analysis, open-domain, and multimodal video settings. While problem statements, datasets, and forms of context differ, the core is a binary (or multi-class) supervised classification: given question q and evidence e (which may be a passage, corpus, video, or even just the question’s linguistic content), predict $y \in \{0,1\}$ (answerable/unanswerable). AC underpins moderation, question routing, data quality assessment, evaluation of knowledge bases, and hallucination avoidance in language models.

## 1. Formal Definitions and Problem Formulations

The mathematical formulation of AC is context-dependent:

- **Community Q&A**: Given $q \in Q$ and an observation window $t$, predict $y \in \{\text{Answered}, \text{Open}\}$, where $y=1$ if $q$ receives at least one answer within $t$ (Quora) [1703.04001].
- **Conversational QA**: Given $(q, c)$ (question and context sentence), predict $y=1$ iff $q$ is answerable from $c$ (CoQA/QNLI sentence-level), extended hierarchically to passage and conversation histories [2210.12979].
- **Information-Seeking Dialogue**: Classifier $f(q,s)$ predicts at sentence level if a candidate snippet $s$ contains (part of) the answer. Aggregations produce passage-level and ranking-level $y$ via max/mean [2401.11452].
- **Large Language Models**: For LLMs, AC reduces to computing whether the context $c$ supports an answer for $q$, approximately via activation-level scoring [2509.22449]: with hidden state $h^*=\mathbf{h}_{\ell^*,p^*}$, define 
  $$s(c,q) = \langle d, h^* \rangle$$
  and predict $y=1$ iff $s(c,q) > \tau$ for threshold $\tau$ learned on dev data.

A table summarizing representative AC task settings:

| Domain              | Input $(q, \cdot)$     | Context Used    | Label $y$                | Notable Formulation                      |
|---------------------|------------------------|-----------------|--------------------------|------------------------------------------|
| Community Q&A       | $q$                    | linguistic, meta| Answered / Open          | SVM on Quora, [1703.04001]               |
| Info-Seeking Search | $(q, P)$               | text corpus     | Answerable / Unanswerable| BERT, passage-sentence aggregation [2401.11452] |
| LLMs (Extractive QA)| $(q, c)$               | passage         | Answerable / Unanswerable| Directional activation scoring [2509.22449]     |
| Video QA            | $(q, V)$               | video, script   | Multiclass (5-way)       | Llama2/SeViLA, timestamp anchoring [2401.17343] |
| QA-based SCR        | $(q, d)$               | document        | Answerable / Unanswerable| LLM-prompted pipeline [2509.08304]       |

## 2. Methodologies: Modeling, Features, Architectures

Approaches to AC span classical classifiers, deep neural networks, LLM prompts, and activation-space analysis:

- **Feature-Based Supervised Learning**: Early work, e.g. on Quora [1703.04001], utilizes SVMs and engineered feature vectors. Features include:
    - Surface (length, OOV rate, $n$-gram presence)
    - Syntax (POS tag diversity: $\mathrm{POSDiv}(q)=-\sum_j p_j\log p_j$)
    - Topic modeling (LDA topic membership & diversity: $\mathrm{TopicDiv}(q)=-\sum_k p_k\log p_k$)
    - Edit/readability (ROUGE-LCS recall from original to edited question)
    - Psycholinguistics (LIWC category fractions)

- **Similarity/Nearest-Neighbor Models**: For detection of unclear or unanswerable questions, retrieval-based features from similar historical questions are used. Key features include BM25 similarity sums, code block presence, length, and cosine similarity between clarification keyphrases [1901.06168].

- **Neural Architectures**:
    - **Contextual Transformers**: Sentence-level and passage-level AC via fine-tuned BERT/ALBERT encoders, using joint inputs $[CLS]q[SEP]c[SEP]$ and a classification head [2210.12979, 2401.11452].
    - **Multimodal Fusion**: Video QA employs vision encoders (ViT-based or SeViLA) fused with transcript encoders; LLMs are augmented for 16k context tokens via rotary embeddings and FlashAttention [2401.17343].
    - **LLM Black-Box Evaluation**: Zero-shot or prompt-based answerability verdicts (e.g., does the model extract an answer, or does a “JUDGE” LLM assert semantic equivalence) [2309.12546, 2509.08304].

- **Activation-Space Methods (LLMs)**: Identification of a linear direction $d$ where
    $$v_{\ell,p} = \text{mean}(h^\mathrm{un}_{\ell,p}) - \text{mean}(h^\mathrm{ans}_{\ell,p})$$
    and ranking direction candidates by their ability to steer model output toward abstention (measured via the log-odds of an “unanswerable” token) [2509.22449].

## 3. Datasets and Labeling Strategies

Datasets underpinning AC research are characterized by their granularity, label rigor, and connection to downstream tasks:

### Community Q&A

- **Quora**: 822,040 questions, 1.8M answers, labels based on time-to-first-answer (1 or 3 months) [1703.04001].
- **Stack Exchange**: Multiple sites (SO, SuperUser, AskUbuntu, etc.) with clear/unclear labels inferred by comment and edit heuristics [1901.06168].

### Information-Seeking and Conversational Search

- **CAsT-Answerability**: Sentence, passage, and ranking-level annotations derived from TREC CAsT, based on nugget overlap, passage inclusion, and combinatorial ranking [2401.11452].
- **CoQA/QNLI-based**: AC modules trained on sentence-level labeled Q–A pairs, incorporating both single-turn and multi-turn conversational data [2210.12979].

### LLM-Based and Document-Level

- **Synthetic SQuAD Variants**: For SCR, answerability is controlled by surgical information removal across paraphrased documents, yielding $(q,d)$ pairs with guaranteed label correctness [2509.08304].
- **RepLiQA, NQ, MuSiQue, SQuAD2.0**: Balanced sets of answerable/unanswerable $(q,c)$ pairs for LLM activation-analysis studies [2509.22449].

### Multimodal/Video QA

- **YTCommentQA**: 2,332 user-generated questions across 2,004 YouTube videos, labeled by human annotators after timestamp verification, script/visual examination, and consensus [2401.17343].

## 4. Evaluation Metrics and Results

Performance is measured using standard classification metrics; implementation choices impact interpretability and generalization:

- **Community Q&A Models** [1703.04001]:
    - SVM (1 month): Accuracy = 76.26%, Precision = 0.763, ROC AUC ≈ 0.762.
    - Linguistic features alone: Accuracy = 74.18%.
- **Unclear/Answerability Entry Points** [1901.06168, 2401.11452]:
    - F1 (unclear/unanswerable class): CNN/BOW-LogReg ≈ 0.79.
    - BERT (sentence, passage, ranking): 0.752, 0.634 (max agg), 0.891 (ranking mean agg).
- **LLM Directions and Baselines** [2509.22449]:
    - Unanswerable recall (same dataset): classifier ≈ 87%, direction ≈ 83–86%.
    - Cross-dataset F1: direction drops only ~7.4%, classifier ~30.2%.
- **Synthetic QA Evaluation** [2309.12546, 2509.08304]:
    - PMAN (automatic metric): Accuracy for manually vetted not-yes/no: 0.94, F1 (answer/unanswerable) ≈ 0.91–0.94.
    - SCR AC datasets ensure 100% label fidelity by construction.
- **Video QA** [2401.17343]:
    - Binary segment-level F1: Llama-2 (13B): 55.49, SeViLA: 46.55.
    - Video-level multiclass accuracy: Llama-2 (13B): 37.70%, SeViLA: 35.27%; basic heuristics perform substantially worse.

A table of select results:

| Setting                    | Method/Baseline         | Top Metric(s)          | Value        |
|----------------------------|------------------------|------------------------|-------------|
| Quora (1 mo)               | SVM (full features)    | Accuracy               | 76.26%      |
| Quora (1 mo)               | Linguistic only        | Accuracy               | 74.18%      |
| StackEx (unclear det.)     | BoW LR / CNN           | F1 (unclear)           | ≈0.79       |
| CAsT (ranking AC)          | BERT (max/mean agg.)   | Accuracy               | 0.891       |
| LLM Direction (cross-ds)   | Direction (calibrated) | Macro-F1 / Recall      | up to 11.9% over classifier |
| YTCommentQA (binary seg)   | Llama-2 (13B)          | F1                     | 55.49       |

## 5. Feature Importance and Model Analysis

Comprehensive feature ablation and coefficient analyses across multiple works reveal:

- **Language Use**: LIWC psycholinguistic features, POS tag diversity ($\mathrm{POSDiv}$), and ROUGE-LCS all strongly correlate with answerability [1703.04001].
- **Structural/Behavioral Factors**: Edit counts, promotion behavior, and topic assignment changes are informative [1703.04001].
- **Similarity-derived Cues**: The need for clarifying information among similar questions (BM25+keyphrase statistics) robustly predicts unclear/unanswerable status [1901.06168].
- **Neural/Activation**: Linear directions in LLM activation space allow identification of an “unanswerable” subspace without retraining, capturing abstention cues generalizable across datasets [2509.22449].

Common failure modes include domain drift, low generalization under prompt-only or classifier-overfit approaches, surface question-type biases (attenuated in video QA), and challenges in modeling “partially” vs. “fully” answerable queries.

## 6. Applications and System Integration

AC modules are critical in applied and research settings:

- **Q&A Platforms**: Early screening for moderation, expert routing, and real-time improvement suggestions [1703.04001].
- **Question Generation Evaluation**: PMAN metric operationalizes answerability assessment for generated questions, correlating perfectly with human rank judgments across multiple models [2309.12546].
- **Conversational Agents**: Synthetic AC modules ensure that generated question–answer pairs in synthetic CQA data accurately reflect answerability, including triaging for “unknown” [2210.12979].
- **Semantic Document Analysis**: AC serves as the measurement instrument for semantic coverage relations (equivalence, inclusion, overlap) between document pairs [2509.08304].
- **LLM Hallucination Control**: Activation-direction approaches provide knobs to steer abstention, mitigating overconfident but unsupported generations [2509.22449].
- **Video QA**: Automated AC is foundational for full-pipeline response generation under significant multi-modality and long-context constraints [2401.17343].

## 7. Open Challenges and Future Directions

Research on AC consistently notes unresolved issues:

- **Partial/Graded Answerability**: Most work frames AC as binary even though degrees of answerability (full, partial, none) are prevalent [2401.11452].
- **Multimodal Reasoning**: Accurate integration of vision and script (including alignment, summarization, and cross-modal entailment) remains insufficiently addressed [2401.17343].
- **Robustness and Generalization**: The direction-based approach [2509.22449] generalizes better than discriminative classifiers, but further improvements (e.g., multi-layer, multi-position fusion) are open.
- **Efficient, Faithful Evaluation**: Black-box LLM metrics like PMAN [2309.12546] and zero-shot QA pipelines [2509.08304] are effective but expensive and sensitive to model/version drift.
- **Explainability and Feedback**: Most AC models predict $y$ but do not surface the minimal sufficient context or recommend clarifications; UI integration remains underexplored [1901.06168].
- **Dataset Limitations**: Many gold or synthetic datasets are bounded by annotation or LLM accuracy ceilings—for document-level SCR, only 8.2% of contexts support 100% reliable AC [2509.08304]. 

A plausible implication is that further integration of external knowledge bases, memory-augmented or hierarchical architectures, and richer supervision (multi-label, evidence-citing, uncertainty-aware) will be required for robust, general AC in dynamic or open-ended settings. Cross-domain and cross-modal transfer, as well as explainable, interactive AC, remain active research directions.

Source: https://www.emergentmind.com/topics/question-answerability-classification-ac