Automatic Difficulty Filtering

Updated 4 July 2026

Automatic difficulty filtering is a method that quantifies item difficulty using metrics like Rasch parameters, IRT ratings, pass rates, and human or model probes.
It employs diverse estimation architectures—from psychometric models and neural regressor designs to graph algorithms—to filter, rank, and route content in assessments and adaptive systems.
Empirical studies show that calibrated, context-aware difficulty filtering enhances test efficacy, adaptive curriculum design, and machine learning performance, despite challenges in relative calibration.

Automatic difficulty filtering denotes computational procedures that estimate the difficulty of items and then use those estimates to retain, discard, rank, weight, or route those items according to a target difficulty profile. In recent research, “difficulty” is not a single invariant quantity: it is operationalized as a psychometric parameter such as the Rasch item difficulty $b_i$ , a normalized benchmark rating derived from IRT or Glicko-2, an online pass rate under a current policy, a human-labeled scalar, an LLM-internal probe value, or a task-specific plausibility score for distractors (Ortega et al., 8 Jun 2026, Ding et al., 2024, Bae et al., 4 Apr 2025, Lee et al., 7 Oct 2025, Kang et al., 3 Nov 2025, Proietti et al., 13 Aug 2025). The common objective is to make difficulty actionable in assessment, curriculum design, benchmark curation, adaptive testing, reinforcement learning, and controlled content generation.

1. Difficulty as an operational variable

The literature treats difficulty as a measurable but context-dependent property of an item. In educational multiple-choice assessment, Automatic Question Difficulty Estimation is formulated as supervised regression to a continuous Rasch difficulty parameter $b_i$ , where higher $b_i$ means a harder item, and prediction is evaluated with MSE and $R^2$ (Ortega et al., 8 Jun 2026). In benchmark construction, Easy2Hard-Bench assigns item difficulties with IRT and Glicko-2 and then normalizes them to $[0,1]$ , adding rating_quantile to support quantile-based filtering across six datasets spanning mathematics, programming, chess, science QA, and commonsense reasoning (Ding et al., 2024).

Other settings make difficulty explicitly model-relative. In reasoning-oriented reinforcement learning, a prompt’s difficulty is the current policy’s success probability,

$p(x)=\mathbb{E}_{y\sim\pi_{\text{init}}(\cdot|x)}[r_{\text{acc}}(x,y)],$

so easy problems have $p(x)\approx 1$ , hard problems have $p(x)\approx 0$ , and the most useful training problems lie in an intermediate band (Bae et al., 4 Apr 2025). In MT difficulty estimation, difficulty is defined as expected translation quality for a source segment, translator, and target language, with lower human quality implying higher difficulty (Proietti et al., 13 Aug 2025). In LLM probing, difficulty is a scalar human label on DeepMath that can be decoded from the final-token representation by a linear probe (Lee et al., 7 Oct 2025).

The same plurality appears in generation tasks. Difficulty-controllable reading-comprehension question generation introduces a setting in which a model receives a sentence, an answer fragment, and a specified difficulty label, and must generate a question whose difficulty matches the label (Gao et al., 2018). Difficulty-controllable cloze distractor generation reduces difficulty to a binary easy–hard distinction and labels distractors automatically using a QA ensemble, with hard distractors taken from the top third of scores and easy distractors from the bottom third (Kang et al., 3 Nov 2025). In automatic fact checking, claim difficulty is presented as a multidimensional property governed by ambiguity, retrieval quality, limited access, unreliable evidence, and difficult inference rather than by a single scalar (Singh et al., 2021). This suggests that automatic difficulty filtering is best understood as a family of operationalizations tied to the downstream task, the solver population, and the evidence environment.

2. Estimation architectures and filtering signals

Difficulty filters rely on heterogeneous estimators, ranging from psychometric models and structured neural regressors to graph algorithms and transformer probes.

Setting	Difficulty signal	Filtering use
MCQ assessment	Predicted Rasch $b_i$ from stem, key, and distractors	Select items in target $\hat b$ bands (Ortega et al., 8 Jun 2026)
Benchmark profiling	IRT or Glicko-2 rating plus quantile	Split into easy, medium, and hard bins (Ding et al., 2024)
Reasoning RL	Online pass rate $b_i$ 0	Keep prompts with intermediate accuracy (Bae et al., 4 Apr 2025)
Cloze distractors	QA-ensemble score after Box–Cox normalization	Keep top third as hard, bottom third as easy (Kang et al., 3 Nov 2025)
MT evaluation	Expected translation quality or token difficulty weight	Select hard subsets or reweight hard tokens (Proietti et al., 13 Aug 2025, Zhan et al., 2021)

In structured assessment, architecture matters. The MCQ model in (Ortega et al., 8 Jun 2026) treats the stem, key, and each distractor as distinct inputs, encodes them with BiLSTMs, and aggregates distractors either by order-aware concatenation or order-invariant summation. This design isolates the contribution of distractor content and order, and it directly supports difficulty filtering because the output is a continuous estimate $b_i$ 1. The same paper argues that distractor structure is not incidental text but a determinant of item difficulty, since distractor plausibility and competition with the key drive whether an item is easy or hard.

Other work uses representations that are not item-structure-centric. “Probing the Difficulty Perception Mechanism of LLMs” fits a linear regressor

$b_i$ 2

on the last-token embedding $b_i$ 3, and interprets $b_i$ 4 as a “difficulty direction” in representation space (Lee et al., 7 Oct 2025). “DiffQue” instead maps question difficulty comparison to edge directionality prediction in a question graph, combining network structure, metadata, temporal signals, and text-derived features to decide which of two CQA questions is harder (Thukral et al., 2019). For learner-facing sentence filtering, Arabic difficulty classification combines sentence embeddings, CEFR-graded lexical features, and syntactic indicators, while logic formalization difficulty estimation uses rule-based features such as number of quantifiers, number of implications, and word-order matching between sentence and formula (Khallaf et al., 2021, Mayn et al., 2022).

Taken together, these estimators show that automatic difficulty filtering is not tied to a single modeling family. It can be psychometric, structural, behavioral, graph-theoretic, or representational, provided the resulting score can be used to partition or rank items.

3. Filtering workflows: static, calibrated, online, and generative

A central distinction in the literature is between static filtering, online filtering, and generation under difficulty control.

Static predict–calibrate–select pipelines are exemplified by MicroCoder. Its Automatic Difficulty Filtering module prompts GPT‑4o for five 1–5 difficulty scores—Problem Comprehension Difficulty, Knowledge Breadth Requirements, Algorithmic Thinking Complexity, Implementation Difficulty, and Other Difficulty—and combines them with fixed weights $b_i$ 5 and $b_i$ 6 respectively. Calibration against Qwen-3-4B-thinking on LiveCodeBench v6 yields thresholds at 2.5 and 2.75, and filtering all problems with predicted difficulty below 2.5 removes about 30% of the dataset while removing over 65% of easy problems and increasing the proportion of hard problems to more than 50% (Li et al., 8 Mar 2026).

Online filtering is more dynamic. In reasoning-oriented RL, filtering is performed at every update by estimating

$b_i$ 7

from the current policy’s rollouts and retaining only prompts with $b_i$ 8, typically with thresholds symmetric around 0.5 such as $b_i$ 9 or $b_i$ 0 (Bae et al., 4 Apr 2025). The theoretical justification is that the lower bound

$b_i$ 1

is maximized at $b_i$ 2, so mid-difficulty prompts maximize a lower bound on useful policy change (Bae et al., 4 Apr 2025). Game-level adaptation uses a related logic: the Intelligent Trial-and-Error framework defines level difficulty via agent win rate $b_i$ 3, uses a performance function $b_i$ 4 that peaks at $b_i$ 5, and stops when $b_i$ 6, roughly $b_i$ 7, thereby filtering toward “just-right” difficulty after only a few trials (González-Duque et al., 2020).

A third workflow is generative difficulty control. Difficulty-controllable reading-comprehension question generation explicitly conditions seq2seq generation on specified difficulty labels and reports that generated questions both improve BLEU and comply with the specified labels (Gao et al., 2018). Difficulty-controllable cloze distractor generation goes further by first constructing a difficulty-annotated dataset through two-way distractor generation, grammatical and semantic filtering, and QA-ensemble labeling, then training a multitask generator with DCDG, ASDE, and DDDE tasks so that requested easy or hard distractors can be produced directly (Kang et al., 3 Nov 2025). In such systems, filtering no longer acts only after item creation; it becomes a control signal during generation itself.

4. Filtering, weighting, and routing in practice

In assessment and item-bank construction, difficulty filtering is typically used to match items to a target band or to maintain a desired test-form distribution. The MCQ framework in (Ortega et al., 8 Jun 2026) makes this explicit: once $b_i$ 8 is available, items can be filtered directly by predicted Rasch difficulty, and the order-invariant distractor encoder offers nearly the same accuracy as the best order-aware model with about half as many parameters. Easy2Hard-Bench similarly supports filtering by rating or rating_quantile, enabling easy/medium/hard partitions, cross-domain curricula, and stress tests focused on the hardest slices of mathematics, coding, chess, or reasoning datasets (Ding et al., 2024).

In machine translation, filtering can be hard or soft. “Estimating Machine Translation Difficulty” uses source-only models such as Sentinel-src-24 and Sentinel-src-25 to scan large corpora and select the hardest source segments. On WMT24, choosing the hardest 25% of items with Sentinel-src-24 lowers average human quality from about 84.4 to 79.1 and reduces the proportion of perfect translations from about 20.7% to 12.1%, making the benchmark more discriminative (Proietti et al., 13 Aug 2025). “Difficulty-Aware Machine Translation Evaluation” instead performs soft filtering by reweighting reference tokens according to

$b_i$ 9

so tokens that most systems fail to translate receive larger weight in DA-BERTScore (Zhan et al., 2021). The underlying idea is the same: easy content should contribute less to evaluation or selection than hard content.

Difficulty filtering also functions as routing. In fact checking, the proposal is to “triage claim difficulty” and “filter out or tag difficult claims” when retrieval is poor, evidence is inaccessible or unreliable, or inference is complex (Singh et al., 2021). In community question answering, relative difficulty estimation can route hard questions to high-expertise users and easy questions to newcomers, and can support reward mechanisms that grant more credit for answering harder questions (Thukral et al., 2019). In language learning, Arabic sentence-difficulty models support filtering of learner materials by CEFR level or by a binary simple–complex distinction, which is directly useful for automatic text simplification and reading-material selection (Khallaf et al., 2021). These applications show that filtering need not mean deletion; it often means assignment to the right consumer, solver, or stage of a pipeline.

5. Empirical validation

The empirical record shows that automatic difficulty filtering is useful but unevenly reliable across domains. In MCQ assessment, the best distractor-aware AQDE model achieves $R^2$ 0 for Natural Sciences and $R^2$ 1 for Social Sciences, improving on a stem-and-key baseline and providing a practical basis for filtering by predicted Rasch $R^2$ 2 (Ortega et al., 8 Jun 2026). In reasoning-oriented RL, balanced online difficulty filtering yields an additional 10% in AIME and 4% improvements in average over plain GRPO, and exceeds the maximum reward of plain GRPO within about 60% training time and training-set volume (Bae et al., 4 Apr 2025).

Generation studies show that explicit difficulty control can be learned. Difficulty-controllable reading-comprehension question generation reports that its framework produces questions with better BLEU and better compliance with specified difficulty labels (Gao et al., 2018). Difficulty-controllable cloze distractor generation reports that its full multitask model causes generated easy distractors to be judged “Easiest” 64.23% of the time and generated hard distractors to be judged “Hardest” 73.25% of the time by GPT‑4o, with invalid rates of 0.2% for easy distractors and 5.1% for hard distractors; it also substantially outperforms GPT‑4o in aligning distractor difficulty with human perception (Kang et al., 3 Nov 2025).

Filtering models for language and question difficulty likewise show strong but task-specific performance. Arabic-BERT reaches F-1 0.80 on 3-way CEFR sentence classification, F-1 0.94 on binary simple–complex classification, and F-1 0.98 on sentence-pair semantic similarity classification for simplification pairs (Khallaf et al., 2021). DiffQue improves relative-difficulty prediction in CQA by 28.77% higher F1 score and 28.72% higher AUC than the best baseline on one Stack Overflow dataset, with similarly large gains elsewhere (Thukral et al., 2019). By contrast, logic formalization difficulty estimation achieves only moderate alignment with student performance, with $R^2$ 3 against First Attempt Correct and $R^2$ 4 against Average Attempts on all 274 exercises, though correlations improve to about $R^2$ 5 and $R^2$ 6 in more homogeneous subsets (Mayn et al., 2022). These results indicate that filtering quality is highest when difficulty labels are behaviorally grounded and the input representation matches the structure of the domain.

6. Limitations and unresolved issues

A persistent limitation is that difficulty is relative rather than absolute. Easy2Hard-Bench notes that difficulty estimates depend on the solver population, whether humans in contests or ensembles of LLMs on a leaderboard (Ding et al., 2024). Online RL filtering defines difficulty through the current policy’s pass rate, so the same prompt changes category as the policy improves (Bae et al., 4 Apr 2025). MT difficulty is explicitly translator- and language-relative, and source-only estimators ignore some of that dependence for practical reasons (Proietti et al., 13 Aug 2025). Difficulty probes in LLMs are model-dependent as well: Qwen2.5-7B-Instruct exhibits clear “difficulty heads”, whereas Llama3.1-8B-Instruct does not (Lee et al., 7 Oct 2025).

Another limitation is incompleteness of the feature space. AQDE in MCQs is validated on two Chilean domains with a homogeneous 4-option format and does not model discrimination or DIF (Ortega et al., 8 Jun 2026). Claim-difficulty assessment has no standard difficulty annotations and must cope with ambiguity, poor ranking, inaccessible evidence, unreliable sources, and difficult inference simultaneously (Singh et al., 2021). Logic-formalization difficulty estimation misses predicate complexity, pragmatic factors, and typicality of exercises, even though these materially affect student performance (Mayn et al., 2022). MicroCoder calibrates static thresholds 2.5 and 2.75 on one benchmark and one model family, which may not transfer unchanged to different models or task distributions (Li et al., 8 Mar 2026).

The practical consequence is that automatic difficulty filtering is strongest when it is calibrated, domain-aware, and continuously validated against real outcomes. This suggests future systems will likely combine psychometric grounding, structure-aware modeling, solver-relative calibration, and explicit uncertainty handling rather than relying on a single universal difficulty score.