Task Difficulty Assessment

• Task Difficulty Assessment (TDA) is the estimation and ranking of how hard a task is for different solvers, using diverse metrics like psychometric parameters and algorithmic search cost.
• It integrates both task-agnostic and task-dependent signals, including cognitive complexity, solver performance, and physiological workload, to provide robust difficulty estimations.
• Recent approaches combine formal rating systems and adaptive difficulty methods to improve educational assessments, programming benchmarks, and AI evaluation frameworks.

Task Difficulty Assessment (TDA) is the estimation, ranking, or formal characterization of how hard a task, item, instance, or benchmark case is for a target solver, whether a human learner, a machine learning model, or a general-purpose agent. Across recent work, difficulty is not treated as a single universal scalar with a fixed meaning. Instead, it is operationalized through psychometric item parameters, cognitive taxonomies, solver performance, task-dependent uncertainty, physiological workload, source–target transfer discrepancy, or algorithmic search cost. This plurality is not accidental: the literature shows that “difficulty” depends on the object being assessed, the solver population, the interaction protocol, and the evaluation objective (Hernandez-Orallo, 2015, Sun et al., 19 May 2025, Toborek et al., 4 Jan 2026).

1. Conceptual scope and definitions

TDA spans at least four distinct but overlapping interpretations. In educational settings, difficulty is often an item property linked to learner success rates, solution times, cognitive complexity, or expert labels. One line of work explicitly distinguishes objective (true) task difficulty, defined from population-level performance statistics, from subjective task difficulty, meaning a learner’s internal judgment of hardness; it further studies the separate question of whether revealing the objective difficulty changes performance or motivation (Spielberg et al., 2022). In reading comprehension, item difficulty is grounded in Rasch-style psychometrics, where the probability of a correct response depends on learner ability and item difficulty (Raina et al., 2024). In adaptive question modeling, difficulty can also be tied to cognitive taxonomies such as Bloom’s levels, which encode the complexity of the reasoning required (V et al., 2022).

A second interpretation is model-centric. In curriculum learning, the key distinction is not simply between easy and hard examples, but between task-agnostic and task-dependent signals, and between human and model sources of difficulty. This yields four quadrants: task-agnostic human, task-agnostic model, task-dependent human, and task-dependent model difficulty. The central claim is that many common heuristics, such as length or readability, are poor proxies for what a neural model actually finds difficult to learn (Toborek et al., 4 Jan 2026).

A third interpretation is benchmark-relative. AGI-Elo defines difficulty as the rating of an individual test case on a scale shared with model or human competence, so that a case with rating $R_t$ is one that an agent with $R_a = R_t$ would solve with probability $0.5$ (Sun et al., 19 May 2025). Code-generation benchmarking work likewise treats difficulty as an empirical property of a programming task for LLMs, inferred from success across many prompt formulations rather than from a single benchmark prompt (Tambon et al., 2024).

A fourth interpretation is formal and computational. Universal psychometrics defines task difficulty as the logarithm of the computational effort needed to obtain and verify an acceptable policy for a stochastic interactive task. For a task $\mu$, tolerance $\epsilon$, and acceptable-policy set

$$A_{[\epsilon,\,\rightarrow v]}(\mu) = \{\pi : R^{[\,\rightarrow v\,]}(\pi,\mu) \ge 1 - \epsilon\},$$

difficulty is expressed through a Levin-style quantity

$$Kt_{[\epsilon,\,\rightarrow v]}(\mu) = \min_{\pi \in A_{[\epsilon,\,\rightarrow v]}(\mu)} \bigl(L(\pi) + \log S^{[\,\rightarrow v\,]}(\pi,\mu)\bigr),$$

where $L(\pi)$ is program length and $S$ is execution cost (Hernandez-Orallo, 2015).

2. Formal models of difficulty

Recent TDA work clusters around three formal traditions: psychometric models, competence–difficulty rating systems, and algorithmic-complexity models.

Psychometric formulations dominate educational ranking tasks. In multiple-choice reading comprehension, ground-truth item difficulty is given by Rasch-model estimates derived from human response distributions, with the underlying response model

$$P(\text{correct} \mid \theta, b) = \frac{1}{1 + e^{-(\theta - b)}}.$$

This makes difficulty cohort-independent and supports evaluation by rank correlation between predicted scores and Rasch difficulty (Raina et al., 2024). In adaptive educational question modeling, relatedness between cognitive complexity and difficulty is quantified by Cramér’s $R_a = R_t$0,

$R_a = R_t$1

with $R_a = R_t$2 reported for the association between Bloom levels and difficulty in QC-Science, indicating a strong association (V et al., 2022).

Competence–difficulty rating systems treat tasks and solvers on a common latent scale. AGI-Elo models both agents and test cases as “players,” with Gaussian rating beliefs and an Elo-style logistic interaction: $R_a = R_t$3 Difficulty is then the case rating $R_a = R_t$4, and mastery can be defined relative to oracle thresholds such as 50%, 90%, or 99% success on all test cases (Sun et al., 19 May 2025). This framework yields interpretable notions such as “easy for agent $R_a = R_t$5 at 90% success” for cases with $R_a = R_t$6.

Algorithmic formulations emphasize search effort rather than observed responses. Universal psychometrics generalizes beyond MDPs by defining tasks as asynchronous-time stochastic interactive processes implemented on a probabilistic Turing machine with explicit time, randomness, and sleep operations. Difficulty is the minimal logarithmic search-and-verification cost for obtaining an acceptable policy, not merely the complexity of executing a fixed policy once (Hernandez-Orallo, 2015). This perspective is especially relevant when task difficulty is meant to be agent-independent rather than fitted to a specific benchmark population.

These traditions are complementary rather than mutually exclusive. A plausible implication is that practical TDA systems increasingly combine them: psychometric grounding when human response data exist, benchmark-relative rating when many model–task interactions are available, and structural or algorithmic notions when extrapolation beyond observed responses is required.

3. Signals and estimation methods

The literature uses heterogeneous signals, but the clearest divide is between task-agnostic proxies and task-dependent evidence. On SNLI, task-agnostic human features such as sentence length, word rarity, syntactic complexity, readability, age of acquisition, concreteness, prevalence, and SLE show mostly low or moderate correlations among themselves, and perplexity from pretrained LLMs shows no meaningful correlation with these human-defined features. More importantly, regressions from task-agnostic features to task-dependent human or model difficulty are weak, with $R_a = R_t$7 for annotation entropy and $R_a = R_t$8 for task-dependent model metrics (Toborek et al., 4 Jan 2026). This is one of the strongest arguments against treating readability-style heuristics as general-purpose difficulty estimators.

Task-dependent signals are more effective. In educational question difficulty prediction, QDiff jointly predicts Bloom’s Taxonomy and difficulty. It uses a shared BERT encoder, a Bloom prediction head, and an interactive attention mechanism in which the predicted Bloom representation conditions attention over input tokens before difficulty prediction. The joint loss is

$R_a = R_t$9

On QC-Science, this architecture improves difficulty weighted F1 from 0.539 for BERT (base) to 0.560, and on QA-data from 0.661 to 0.705; random Bloom labels reduce QA-data performance to macro F1 0.656 and weighted F1 0.674, showing that meaningful auxiliary labels matter (V et al., 2022).

Physiological TDA offers another task-dependent route. A blink-based framework derives a baseline-corrected eye-aspect-ratio signal, converts it into a time–frequency spectrogram using bandpass filtering and the Lomb–Scargle periodogram, and feeds the result to a 2D LSTM. Hand-engineered blink rate and blink duration were not significantly sensitive to task difficulty, whereas blink entropy and the full spectrogram improved sensitivity. The 2D LSTM achieved 70.4% accuracy for event-based difficulty, 72.2% for binary subjective difficulty, and 77.8% for binary objective difficulty under leave-one-subject-out evaluation (Cho, 2021).

In multi-source domain adaptation, task difficulty can refer to the transfer gap between a source and a target domain. HEDN defines a per-source Task Difficulty Assessment score

$0.5$0

with a logistic schedule for $0.5$1, and routes the hardest source to a DANN-style Hard Network and the easiest source to a prototype-based Easy Network. This difficulty-aware routing improves cross-subject EEG emotion recognition to 93.58% on SEED and 79.82% on SEED-IV; replacing TDA with random source assignment reduces performance to 92.20% and 78.71% (Wang et al., 10 Nov 2025).

4. Major application areas

Educational assessment is the oldest and most diversified TDA domain in this corpus. Difficulty can be predicted from content alone, jointly with cognitive complexity, as in QDiff for K–12 science questions (V et al., 2022). It can also be ranked when continuous labels are scarce. For Cambridge multiple-choice reading comprehension, level-classification transfer reaches Spearman’s $0.5$2, reading-comprehension transfer reaches $0.5$3, zero-shot absolute assessment reaches $0.5$4, zero-shot comparative assessment reaches $0.5$5, and combining level classification with comparative LLM judgments reaches $0.5$6 (Raina et al., 2024). In AI-aided education, another line of work examines not only whether difficulty can be predicted, but whether revealing it should be part of the intervention. In matchstick riddles, revealing objective difficulty increases time spent on hard riddles without changing success rate, and the paper argues for separate AI components: one to predict difficulty and another to decide when to reveal it (Spielberg et al., 2022).

Programming and code generation form a second major application area. One strand treats difficulty as a property of benchmark tasks for LLMs. HardEval computes task difficulty by aggregating functional correctness and syntactic similarity across 18 prompts per task, 5 code samples per prompt, and 5 code models, with task difficulty defined as

$0.5$7

where $0.5$8 summarize performance under three levels of prompt context (Tambon et al., 2024). Another strand studies whether LLMs can act as difficulty judges. On 1,825 LeetCode problems, a LightGBM ensemble using TF–IDF and numeric features reaches 86.0% accuracy and macro F1 83.7%, whereas GPT-4o, used as a text-only judge with numeric metadata removed, reaches 37.75% accuracy and macro F1 35.6%. On the 385 real Hard problems, GPT-4o labels 321 as Easy, 43 as Medium, and only 21 as Hard (Tabib et al., 23 Nov 2025).

Benchmark-level TDA for generalist systems is now extending beyond classification or code generation. AGI-Elo rates both agents and individual test cases across ImageNet, COCO, MMLU, LiveCodeBench, Waymo, and NAVSIM, supporting cross-domain difficulty distributions and explicit “competency gap” estimates to defined mastery thresholds (Sun et al., 19 May 2025). For tool-using agents, TASTE constructs $0.5$9-Bench by sampling valid tool sequences with an Adaptive Contrastive $\mu$0-gram model, clustering them, instantiating them into tasks, and then applying difficulty evolution. The resulting benchmark more than doubles the number of unique tool combinations and causes models nearly saturating $\mu$1-Bench to drop sharply; Gemini-3-Flash falls from 0.82–0.94 to 0.28–0.61 depending on domain (Keren et al., 27 May 2026).

Other applications highlight the breadth of TDA. In translation studies, translationese is modeled as a response to translation task difficulty, with source-text syntactic complexity and translation-solution entropy emerging as the strongest predictors of translatedness across language pairs and modes (Kunilovskaya, 12 Mar 2026). In human–agent interaction, task difficulty is manipulated structurally in a $\mu$2 grid task and shown to increase one category of empathy—personal distress—relative to low difficulty, even though overall empathy declines from pre-task to post-task (Tsumura et al., 2022).

5. Empirical findings, controversies, and common misconceptions

A persistent misconception is that difficulty is captured adequately by simple surface heuristics. The strongest contrary evidence comes from curriculum learning on SNLI: task-agnostic human and model signals are largely independent, and only task-dependent human and model signals align meaningfully (Toborek et al., 4 Jan 2026). This suggests that improvements obtained by curricula based on length or readability cannot be assumed to arise from correctly ordering samples by model difficulty.

A second misconception is that powerful LLMs can reliably judge difficulty from text alone. Competitive programming provides a clear counterexample. GPT-4o, used as a pure natural-language difficulty assessor, performs far below an interpretable LightGBM model because numeric constraints and acceptance rates dominate Hard-vs-non-Hard separation. The paper also shows an internal inconsistency: GPT-4o labels 384 of 385 synthetic “Hard” problems it generated itself as Medium when re-judged under the same text-only protocol (Tabib et al., 23 Nov 2025). This undermines naïve deployment of LLM judges in educational or benchmark pipelines.

A third controversy concerns whether adaptive difficulty adjustment is uniformly beneficial. In an open-ended MIS learning task with 86 participants, predefined progression, self-determined choice, and knowledge-tracing-based DDA produce no significant differences in normalized learning gain, flow, or perceived difficulty. By contrast, post-hoc clustering of practice behavior reveals large differences in learning: “thoughtful searchers,” characterized by balanced set/unset actions, moderate time between clicks, and few resets, achieve significantly higher normalized learning gain than “few-shot rushers,” “strugglers,” and “board-clearers” (Schütt et al., 2023). A plausible implication is that in open-ended tasks, TDA based only on correctness and item difficulty is incomplete unless it also models exploration behavior.

A fourth misconception is that task difficulty is merely descriptive and pedagogically inert. The matchstick-riddle experiment on difficulty revelation shows that revealing objective difficulty changes behavior on hard tasks, specifically by increasing solution time and persistence, even when success rate is unchanged (Spielberg et al., 2022). This indicates that the communication of difficulty can itself become an intervention variable.

6. Benchmark construction, system design, and future directions

Current work increasingly treats TDA not only as post-hoc measurement but as a design primitive for building evaluation suites and adaptive systems. TASTE is a clear example. It begins from tool sequences rather than natural-language scenarios, uses LLM-judged plausibility to train an Adaptive Contrastive trigram model, clusters sampled sequences with semantically weighted edit distance, and then applies graduated difficulty evolution through adversarial user behavior, policy-pressure patterns, and database-grounded decoys. The result is a benchmark that is simultaneously more difficult and more structurally diverse than its predecessor (Keren et al., 27 May 2026).

AGI-Elo suggests a complementary route: difficulty-aware evaluation should be benchmark-intrinsic, with every test case receiving a rating on a shared competence–difficulty scale. Because the framework reports long-tail difficulty distributions, predictive performance curves, and oracle gaps, it supports a notion of “distance to mastery” that ordinary aggregate accuracy conceals. Across all AI agents and tasks studied there, gaps to 99%-confidence oracles remain large, around 1031–1185 rating points, while even 50%-confidence oracle gaps remain around 233–387 points for AI agents (Sun et al., 19 May 2025).

For system design, several concrete principles recur across the literature. Auxiliary tasks should be chosen when they are strongly associated with difficulty and predictable from the input, as Bloom’s labels are for educational science questions (V et al., 2022). Difficulty estimators for open-ended tasks should incorporate behavior traces, not just outcomes, because practice behavior can be more diagnostic of future learning than the adjustment policy itself (Schütt et al., 2023). Difficulty judges in programming and similar structured domains should be grounded in explicit numeric and operational features rather than text-only impressions (Tabib et al., 23 Nov 2025). In curriculum learning, lightweight task-dependent estimators are preferable to task-agnostic heuristics whenever feasible (Toborek et al., 4 Jan 2026).

Future directions in the corpus are consistent. They include richer IRT-like or rating-based models with more nuanced parameters, integration of textual difficulty with learner-response or item-response data, cross-domain extension to embodied and multimodal agents, behavior-aware adaptive curricula, and automated generation of benchmarks whose difficulty and coverage co-evolve with model capability (Sun et al., 19 May 2025, Keren et al., 27 May 2026). Taken together, these works suggest that TDA is moving from a peripheral annotation problem toward a central infrastructure problem for evaluation, personalization, and scientifically interpretable benchmarking.