---
title: Automatic Difficulty Filtering
url: https://www.emergentmind.com/topics/automatic-difficulty-filtering
type: topic
---

# Automatic Difficulty Filtering

Automatic difficulty filtering denotes computational procedures that estimate the difficulty of items and then use those estimates to retain, discard, rank, weight, or route those items according to a target difficulty profile. In recent research, “difficulty” is not a single invariant quantity: it is operationalized as a psychometric parameter such as the Rasch item difficulty \(b_i\), a normalized benchmark rating derived from IRT or Glicko-2, an online pass rate under a current policy, a human-labeled scalar, an LLM-internal probe value, or a task-specific plausibility score for distractors [2606.08988] [2409.18433] [2504.03380] [2510.05969] [2511.01526] [2508.10175]. The common objective is to make difficulty actionable in assessment, curriculum design, benchmark curation, adaptive testing, reinforcement learning, and controlled content generation.

## 1. Difficulty as an operational variable

The literature treats difficulty as a measurable but context-dependent property of an item. In educational multiple-choice assessment, Automatic Question Difficulty Estimation is formulated as supervised regression to a continuous Rasch difficulty parameter \(b_i\), where higher \(b_i\) means a harder item, and prediction is evaluated with MSE and \(R^2\) [2606.08988]. In benchmark construction, Easy2Hard-Bench assigns item difficulties with IRT and Glicko-2 and then normalizes them to \([0,1]\), adding `rating_quantile` to support quantile-based filtering across six datasets spanning mathematics, programming, chess, science QA, and commonsense reasoning [2409.18433].

Other settings make difficulty explicitly model-relative. In reasoning-oriented reinforcement learning, a prompt’s difficulty is the current policy’s success probability,
\[
p(x)=\mathbb{E}_{y\sim\pi_{\text{init}}(\cdot|x)}[r_{\text{acc}}(x,y)],
\]
so easy problems have \(p(x)\approx 1\), hard problems have \(p(x)\approx 0\), and the most useful training problems lie in an intermediate band [2504.03380]. In MT difficulty estimation, difficulty is defined as expected translation quality for a source segment, translator, and target language, with lower human quality implying higher difficulty [2508.10175]. In LLM probing, difficulty is a scalar human label on DeepMath that can be decoded from the final-token representation by a linear probe [2510.05969].

The same plurality appears in generation tasks. Difficulty-controllable reading-comprehension question generation introduces a setting in which a model receives a sentence, an answer fragment, and a specified difficulty label, and must generate a question whose difficulty matches the label [1807.03586]. Difficulty-controllable cloze distractor generation reduces difficulty to a binary easy–hard distinction and labels distractors automatically using a QA ensemble, with hard distractors taken from the top third of scores and easy distractors from the bottom third [2511.01526]. In automatic fact checking, claim difficulty is presented as a multidimensional property governed by ambiguity, retrieval quality, limited access, unreliable evidence, and difficult inference rather than by a single scalar [2109.09689]. This suggests that automatic difficulty filtering is best understood as a family of operationalizations tied to the downstream task, the solver population, and the evidence environment.

## 2. Estimation architectures and filtering signals

Difficulty filters rely on heterogeneous estimators, ranging from psychometric models and structured neural regressors to graph algorithms and transformer probes.

| Setting | Difficulty signal | Filtering use |
|---|---|---|
| MCQ assessment | Predicted Rasch \(b_i\) from stem, key, and distractors | Select items in target \(\hat b\) bands [2606.08988] |
| Benchmark profiling | IRT or Glicko-2 rating plus quantile | Split into easy, medium, and hard bins [2409.18433] |
| Reasoning RL | Online pass rate \(\hat p(x)\) | Keep prompts with intermediate accuracy [2504.03380] |
| Cloze distractors | QA-ensemble score after Box–Cox normalization | Keep top third as hard, bottom third as easy [2511.01526] |
| MT evaluation | Expected translation quality or token difficulty weight | Select hard subsets or reweight hard tokens [2508.10175] [2107.14402] |

In structured assessment, architecture matters. The MCQ model in [2606.08988] treats the stem, key, and each distractor as distinct inputs, encodes them with BiLSTMs, and aggregates distractors either by order-aware concatenation or order-invariant summation. This design isolates the contribution of distractor content and order, and it directly supports difficulty filtering because the output is a continuous estimate \(\hat b_i\). The same paper argues that distractor structure is not incidental text but a determinant of item difficulty, since distractor plausibility and competition with the key drive whether an item is easy or hard.

Other work uses representations that are not item-structure-centric. “Probing the Difficulty Perception Mechanism of Large Language Models” fits a linear regressor
\[
\hat y=\mathbf{w}^{\top}\mathbf{h}+b
\]
on the last-token embedding \(\mathbf{h}\), and interprets \(\mathbf{w}\) as a “difficulty direction” in representation space [2510.05969]. “DiffQue” instead maps question difficulty comparison to edge directionality prediction in a question graph, combining network structure, metadata, temporal signals, and text-derived features to decide which of two CQA questions is harder [1906.00145]. For learner-facing sentence filtering, Arabic difficulty classification combines sentence embeddings, CEFR-graded lexical features, and syntactic indicators, while logic formalization difficulty estimation uses rule-based features such as number of quantifiers, number of implications, and word-order matching between sentence and formula [2103.04386] [2204.12197].

Taken together, these estimators show that automatic difficulty filtering is not tied to a single modeling family. It can be psychometric, structural, behavioral, graph-theoretic, or representational, provided the resulting score can be used to partition or rank items.

## 3. Filtering workflows: static, calibrated, online, and generative

A central distinction in the literature is between static filtering, online filtering, and generation under difficulty control.

Static predict–calibrate–select pipelines are exemplified by MicroCoder. Its Automatic Difficulty Filtering module prompts GPT‑4o for five 1–5 difficulty scores—Problem Comprehension Difficulty, Knowledge Breadth Requirements, Algorithmic Thinking Complexity, Implementation Difficulty, and Other Difficulty—and combines them with fixed weights \(0.05, 0.05, 0.45, 0.35,\) and \(0.10\) respectively. Calibration against Qwen-3-4B-thinking on LiveCodeBench v6 yields thresholds at 2.5 and 2.75, and filtering all problems with predicted difficulty below 2.5 removes about 30% of the dataset while removing over 65% of easy problems and increasing the proportion of hard problems to more than 50% [2603.07779].

Online filtering is more dynamic. In reasoning-oriented RL, filtering is performed at every update by estimating
\[
\hat p(x)=\frac{1}{G}\sum_{i=1}^{G} r_{\text{acc}}(x,y_i)
\]
from the current policy’s rollouts and retaining only prompts with \(T_{\text{Low}}<\hat p(x)<T_{\text{High}}\), typically with thresholds symmetric around 0.5 such as \((0.2,0.8)\) or \((0.3,0.7)\) [2504.03380]. The theoretical justification is that the lower bound
\[
D_{\mathrm{KL}}(\pi_{\text{init}}\Vert \pi^* \mid x)\ge \frac{1}{2\beta^2}p(x)(1-p(x))
\]
is maximized at \(p(x)=0.5\), so mid-difficulty prompts maximize a lower bound on useful policy change [2504.03380]. Game-level adaptation uses a related logic: the Intelligent Trial-and-Error framework defines level difficulty via agent win rate \(w\), uses a performance function \(p(w)\) that peaks at \(w=0.6\), and stops when \(p(w)\ge 0.75\), roughly \(w\in[0.5,0.7]\), thereby filtering toward “just-right” difficulty after only a few trials [2005.07677].

A third workflow is generative difficulty control. Difficulty-controllable reading-comprehension question generation explicitly conditions seq2seq generation on specified difficulty labels and reports that generated questions both improve BLEU and comply with the specified labels [1807.03586]. Difficulty-controllable cloze distractor generation goes further by first constructing a difficulty-annotated dataset through two-way distractor generation, grammatical and semantic filtering, and QA-ensemble labeling, then training a multitask generator with DCDG, ASDE, and DDDE tasks so that requested easy or hard distractors can be produced directly [2511.01526]. In such systems, filtering no longer acts only after item creation; it becomes a control signal during generation itself.

## 4. Filtering, weighting, and routing in practice

In assessment and item-bank construction, difficulty filtering is typically used to match items to a target band or to maintain a desired test-form distribution. The MCQ framework in [2606.08988] makes this explicit: once \(\hat b_i\) is available, items can be filtered directly by predicted Rasch difficulty, and the order-invariant distractor encoder offers nearly the same accuracy as the best order-aware model with about half as many parameters. Easy2Hard-Bench similarly supports filtering by `rating` or `rating_quantile`, enabling easy/medium/hard partitions, cross-domain curricula, and stress tests focused on the hardest slices of mathematics, coding, chess, or reasoning datasets [2409.18433].

In machine translation, filtering can be hard or soft. “Estimating Machine Translation Difficulty” uses source-only models such as Sentinel-src-24 and Sentinel-src-25 to scan large corpora and select the hardest source segments. On WMT24, choosing the hardest 25% of items with Sentinel-src-24 lowers average human quality from about 84.4 to 79.1 and reduces the proportion of perfect translations from about 20.7% to 12.1%, making the benchmark more discriminative [2508.10175]. “Difficulty-Aware Machine Translation Evaluation” instead performs soft filtering by reweighting reference tokens according to
\[
d(t)=1-\frac{1}{K}\sum_{k=1}^{K}\max_{h\in h_k}\text{sim}(t,h),
\]
so tokens that most systems fail to translate receive larger weight in DA-BERTScore [2107.14402]. The underlying idea is the same: easy content should contribute less to evaluation or selection than hard content.

Difficulty filtering also functions as routing. In fact checking, the proposal is to “triage claim difficulty” and “filter out or tag difficult claims” when retrieval is poor, evidence is inaccessible or unreliable, or inference is complex [2109.09689]. In community question answering, relative difficulty estimation can route hard questions to high-expertise users and easy questions to newcomers, and can support reward mechanisms that grant more credit for answering harder questions [1906.00145]. In language learning, Arabic sentence-difficulty models support filtering of learner materials by CEFR level or by a binary simple–complex distinction, which is directly useful for automatic text simplification and reading-material selection [2103.04386]. These applications show that filtering need not mean deletion; it often means assignment to the right consumer, solver, or stage of a pipeline.

## 5. Empirical validation

The empirical record shows that automatic difficulty filtering is useful but unevenly reliable across domains. In MCQ assessment, the best distractor-aware AQDE model achieves \(R^2=0.83\) for Natural Sciences and \(R^2=0.71\) for Social Sciences, improving on a stem-and-key baseline and providing a practical basis for filtering by predicted Rasch \(b\) [2606.08988]. In reasoning-oriented RL, balanced online difficulty filtering yields an additional 10% in AIME and 4% improvements in average over plain GRPO, and exceeds the maximum reward of plain GRPO within about 60% training time and training-set volume [2504.03380].

Generation studies show that explicit difficulty control can be learned. Difficulty-controllable reading-comprehension question generation reports that its framework produces questions with better BLEU and better compliance with specified difficulty labels [1807.03586]. Difficulty-controllable cloze distractor generation reports that its full multitask model causes generated easy distractors to be judged “Easiest” 64.23% of the time and generated hard distractors to be judged “Hardest” 73.25% of the time by GPT‑4o, with invalid rates of 0.2% for easy distractors and 5.1% for hard distractors; it also substantially outperforms GPT‑4o in aligning distractor difficulty with human perception [2511.01526].

Filtering models for language and question difficulty likewise show strong but task-specific performance. Arabic-BERT reaches F-1 0.80 on 3-way CEFR sentence classification, F-1 0.94 on binary simple–complex classification, and F-1 0.98 on sentence-pair semantic similarity classification for simplification pairs [2103.04386]. DiffQue improves relative-difficulty prediction in CQA by 28.77% higher F1 score and 28.72% higher AUC than the best baseline on one Stack Overflow dataset, with similarly large gains elsewhere [1906.00145]. By contrast, logic formalization difficulty estimation achieves only moderate alignment with student performance, with \(r=-0.42\) against First Attempt Correct and \(r=0.33\) against Average Attempts on all 274 exercises, though correlations improve to about \(-0.58\) and \(0.51\) in more homogeneous subsets [2204.12197]. These results indicate that filtering quality is highest when difficulty labels are behaviorally grounded and the input representation matches the structure of the domain.

## 6. Limitations and unresolved issues

A persistent limitation is that difficulty is relative rather than absolute. Easy2Hard-Bench notes that difficulty estimates depend on the solver population, whether humans in contests or ensembles of LLMs on a leaderboard [2409.18433]. Online RL filtering defines difficulty through the current policy’s pass rate, so the same prompt changes category as the policy improves [2504.03380]. MT difficulty is explicitly translator- and language-relative, and source-only estimators ignore some of that dependence for practical reasons [2508.10175]. Difficulty probes in LLMs are model-dependent as well: Qwen2.5-7B-Instruct exhibits clear “difficulty heads”, whereas Llama3.1-8B-Instruct does not [2510.05969].

Another limitation is incompleteness of the feature space. AQDE in MCQs is validated on two Chilean domains with a homogeneous 4-option format and does not model discrimination or DIF [2606.08988]. Claim-difficulty assessment has no standard difficulty annotations and must cope with ambiguity, poor ranking, inaccessible evidence, unreliable sources, and difficult inference simultaneously [2109.09689]. Logic-formalization difficulty estimation misses predicate complexity, pragmatic factors, and typicality of exercises, even though these materially affect student performance [2204.12197]. MicroCoder calibrates static thresholds 2.5 and 2.75 on one benchmark and one model family, which may not transfer unchanged to different models or task distributions [2603.07779].

The practical consequence is that automatic difficulty filtering is strongest when it is calibrated, domain-aware, and continuously validated against real outcomes. This suggests future systems will likely combine psychometric grounding, structure-aware modeling, solver-relative calibration, and explicit uncertainty handling rather than relying on a single universal difficulty score.

Source: https://www.emergentmind.com/topics/automatic-difficulty-filtering