---
title: 'UnsolvableQA: Unanswerable Question Framework'
url: https://www.emergentmind.com/topics/unsolvableqa
type: topic
---

# UnsolvableQA: Unanswerable Question Framework

UnsolvableQA concerns the rigorous identification, synthesis, and evaluation of problems deliberately constructed to be unsolvable by a reasoning or question-answering (QA) system, as well as the design of training and evaluation methodologies that incentivize reliable detection and appropriate abstention. Central to the field is the alignment of large language models (LLMs) and other automated systems not only towards solving feasible tasks but also toward reliably recognizing intrinsic contradictions, missing information, or exceeding capacity—thereby mitigating hallucination and overconfidence on intractable instances [2512.01661]. The UnsolvableQA paradigm now underpins research across reading comprehension, multi-modal reasoning, knowledge base question answering, and foundational AGI evaluation.

## 1. Formal Definition and Taxonomy of Unsolvability

A problem is classified as unsolvable if, under the assumptions of the underlying domain (e.g., mathematics, logic, KB coverage, or factual completeness), no coherent solution can be constructed, or the solution is inherently unknowable. Unsolvability arises in several forms:

- **Objective contradiction**: The question or problem statement encodes mutually exclusive or axiom-violating constraints (e.g., "find an integer simultaneously greater and less than zero") [2512.01661].
- **Schema or data incompleteness**: In knowledge base QA, unsolvability results from type, relation, or entity omission causing either (i) no valid logical form to exist or (ii) logical form execution yielding an empty answer [2212.10189].
- **Information absence or retrieval miss**: In information-seeking settings, unanswerability arises when the context or evidence lacks the necessary information, or a question is posed with false or ambiguous premises [2010.11915].
- **Ambiguity or ill-posedness**: Some questions, especially in open-domain or philosophical settings, are unanswerable due to multiple interpretations or lack of a definitive ground truth [2309.04635].
- **Beyond capacity**: A distinction is made between objective unsolvability and problems that are solvable in principle but exceed model capability; reliable systems must prudently abstain in both cases [2512.01661].

These categories are operationalized via concrete annotations in benchmarks such as SQuAD 2.0 (adversarially crowdwritten unanswerables) [1806.03822], GrailQAbility (taxonomy of KB incompleteness) [2212.10189], and multi-modal sets like VisionTrap (contradictory logic or physically impossible image-question pairs) [2507.17262].

## 2. Construction of UnsolvableQA Benchmarks

UnsolvableQA datasets are developed using multiple methodologies tailored to domain characteristics:

- **Programmatic generation for logic and combinatorics**: For domains with tractable decision procedures, unsolvable instances are algorithmically verified by exhaustive search, SAT/CSP solvers, or explicit bottleneck injection. E.g., unsolvable Game24 and Hamiltonian Cycle problems in UnsolvableQA are generated and confirmed by exhaustive rational-arithmetic or SAT solving [2512.01661].
- **Reverse Construction for mathematics**: A solvable problem's solution chain (CoT) is tampered by injecting a logical or axiom contradiction, and the problem statement is reverse-engineered so the corrupted CoT would "solve" it, followed by verification that a strong model cannot produce a solution [2512.01661].
- **Synthetic context re-matching and augmentation**: Automated pipelines (AGent) create unanswerables by pairing questions with contexts that lack appropriate answers, using TF-IDF similarity and adversarial/plausibility filtering to maximize difficulty [2309.05103].
- **Lexico-semantic perturbation**: Antonym-swaps and entity-swaps generate diverse, high-fluency unanswerables that retain surface similarity to answerable questions, challenging models relying on overlapping tokens [2310.19403].
- **Curated open-domain harvesting**: Community-sourced unsolved problems ("UQ") are filtered by rules (e.g., engagement thresholds), LLM judges, and human experts to ensure both difficulty and real-world value, used in asynchronous evaluation platforms [2508.17580].

Empirical datasets often enforce a close pairing of solvable and unsolvable instances to facilitate robust decision boundary learning, with controlled test splits and systematic coverage across domains (e.g., six domain splits in [2512.01661], fine-grained unsolvability types in [2212.10189], and image-based logical impossibilities in [2507.17262]).

## 3. Model Architectures and Training Objectives

UnsolvableQA work has motivated model and training innovations for robust three-way decision making (solve, reject-unsolvable, abstain-overcapacity):

- **Composite reward frameworks**: The UnsolvableRL framework adopts Group-Relative Policy Optimization with a three-component reward, penalizing hallucinated answers on unsolvable problems ($R_{\text{detect}}$), rewarding abstentions when appropriate ($R_{\text{cal}}$), and directly optimizing answer accuracy ($R_{\text{acc}}$) [2512.01661].
- **Unified span and abstention architectures**: MRC models integrate span prediction and answerability classification—see null-span/BCE approaches in SQuAD 2.0 [1806.03822], U-Net’s universal node for fused Q/P encoding [1810.06638], as well as explicit "no logical form" targets in KBQA [2212.10189].
- **Iterative feedback with verifiers**: FUn-FuSIC for KBQA employs multi-pass LLM reasoning, weak verifiers (e.g., semantic back-translation, type-checking), and customized self-consistency for unanswerability to disambiguate empty-answer as genuine unsolvability [2406.14313].
- **Calibration and capability heads**: Adaptive incentive schedules (target accuracy $\tau$ increasing over training) regularize refusal behavior, avoiding degenerate "always answer" or "always abstain" policies [2512.01661].

Novelty in these formulations lies in explicit optimization against overconfidence, gradient interference (capability collapse), and careful calibration of negative signals, as established by both theoretical and empirical ablations [2512.01661].

## 4. Evaluation Metrics and Empirical Results

UnsolvableQA research relies on both standard and custom metrics to examine detection and calibration:

| Metric                | Purpose                      | Example Tasks                   |
|-----------------------|-----------------------------|---------------------------------|
| EM/F1 (all/split)     | Exact or token match         | SQuAD 2.0, GrailQAbility        |
| Unanswerable EM/F1    | Correct abstentions/rejects  | Logic, KBQA, VQA                |
| Pass rate / Verified rate | Fraction passing validator, fraction verified correct | UQ Platform [2508.17580]  |
| Calibration measures  | Accuracy at fixed threshold, abstention rate | VisionTrap, Impossible Test     |
| Regression/statistical analysis | Uncertainty as function of problem difficulty or category | Impossible Test [2411.14486]   |

Key empirical findings:
- Fine-tuned large models on UnsolvableQA exhibit substantial improvement: unsolvable detection rises from 36.3% to 90.9% under UnsolvableRL, and Game24 rejection accuracy improves from 17% to 99% [2512.01661].
- Without explicit exposure to unsolvable items, smaller models and LLMs exhibit "capability collapse," with unsolvability rejection accuracy approaching zero under continued training [2512.01661].
- In VQA, failure to abstain or over-abstention is common; e.g., LLaVA abstains 97% of the time when options are not presented, while Gemini Flash only abstains at 29% on impossible questions [2507.17262].
- On community-sourced unsolved problems (UQ), top models pass validation on only 15% of questions; human verification shows only a handful of genuine breakthroughs [2508.17580].
- On epistemic humility ("Impossible Test"), even the best (Gemini/Claude) models admit ignorance on ~69% of unsolvable items; performances are significantly lower in invention and NP-hardness categories [2411.14486].

## 5. Failure Modes and Theoretical Underpinnings

Failures in UnsolvableQA systems consistently manifest as overconfident hallucinations (proposing plausible but incorrect answers), or as degenerate abstention (universal rejection):

- **Gradient interference**: Training only on solvable problems suppresses refusal logits on unsolvable instances, destroying the boundary required to distinguish contradictions [2512.01661].
- **Verifier/generation mismatch**: In iterative KBQA, naive execution-guided feedback treats all empty answers as fixable, failing to recognize schema-level unsolvability; addition of weak verifiers and back-translation feedback is essential [2406.14313].
- **Heuristic bias**: VLMs in VisionTrap exhibit format sensitivity, e.g., guessing when prompted with options, abstaining when forced, reflecting underdeveloped uncertainty heads [2507.17262].
- **Data-augmentation limitations**: Entity and antonym swaps can still produce answerable cases (≈20% noise), reducing selectivity; purely answerable-skewed training produces models highly vulnerable to spurious context-answer overlap [2310.19403].

Gradient and statistical analyses expose the necessity of negative examples—unsolvable or out-of-capacity cases—to defend against both types of failure [2512.01661][2310.19403].

## 6. Implications for Alignment, AGI Evaluation, and Future Research

UnsolvableQA techniques now inform core recommendations for model reliability, alignment, and AGI evaluation:

- **Objective unsolvability exposure**: Only by synthetic/intrinsic contradiction exposure (not just "out of scope") can models avoid hallucination and evade capability collapse [2512.01661][2310.19403].
- **Calibration-by-design**: Incentivizing abstention proportionally to batch accuracy and penalizing false rejections, as in adaptive refusal schedules, prevents both reckless answering and universal declination [2512.01661].
- **Epistemic humility metrics**: AGI tests now incorporate "accuracy of uncertainty acknowledgment" (e.g., proportion of "I don't know" on Impossible Test), providing a quantifiable complement to standard accuracy [2411.14486].
- **Verifier-driven pipelines**: As in the UQ platform, community-driven, validator-assisted, and human-verified workflows are leveraged to track the genuine advancement of frontier models on unsolved problems [2508.17580].

Future research targets include scaling unsolvability detection to open-ended language, multi-hop reasoning, more diverse domains, and strategic negative data synthesis for global model calibration. Deeper theoretical work is warranted on gradient interference in underparameterized regimes, uncertainty estimation, and generalization of UnsolvableQA paradigms to multi-agent and embodied AI settings [2512.01661][2406.14313][2411.14486].

Source: https://www.emergentmind.com/topics/unsolvableqa