Four-Quadrant Difficulty Signal Framework

• Four-Quadrant Categorisation of Difficulty Signals defines a framework that partitions proxies based on human vs. model sources and task-agnostic vs. task-dependent characteristics.
• The framework rigorously evaluates instance difficulty across various fields, including curriculum learning, psycholinguistics, physical reasoning, and language model analysis.
• Empirical studies show that combining distinct signal types can enhance adaptive training strategies and improve model performance.

A four-quadrant categorisation of difficulty signals offers a systematic framework for analyzing, designing, and evaluating proxies used to estimate instance difficulty in diverse domains such as curriculum learning, psycholinguistics, physical reasoning, and deep LLM analysis. This framework partitions the space of difficulty signals along two independent binary axes—source and scope (or, alternatively, stability)—enabling rigorous interrogation of which signals actually reflect underlying task challenge, for whom, and under what conditions. The approach generalizes across fields, from natural language understanding to physical manipulation and LLM diagnostics, providing a principled means to compare, combine, and critique the signals that drive adaptive training and evaluation.

1. Axes of Categorisation and Quadrant Formulation

The primary instantiations of the four-quadrant framework define two orthogonal axes:

1. Source of Signal:
   • Human-derived (H): Based on human intuition, annotation, psycholinguistic norms, or behavioral data.
   • Model-derived (M): Elicited from computational models, such as neural network predictions, training dynamics, or automated metrics.
2. Scope, Task Sensitivity, or Stability:
   • Task-agnostic (TA): Signals independent of the target task or label, typically shallow features or generic measures of variation.
   • Task-dependent (TD): Signals specific to the target task or label, requiring human annotation or model access to labels.
   • Alternatively, for model learning contexts, "Stability over training": Whether the signal aligns with ground-truth challenge or model accuracy as the model itself improves.

This yields four distinct quadrants:

Signal Type	Human-derived	Model-derived
Task-agnostic	TA–H	TA–M
Task-dependent	TD–H	TD–M

Alternate variants, as in system-level LLM diagnosis, re-cast the second axis as Stability/Alignment under RL training, resulting in:

	Stable/Improving	Degrading/Misaligned
Human-derived	Q1	Q2
Model-derived	Q3	Q4

Each quadrant is populated by domain-specific proxies, concretely defined and empirically evaluated.

2. Formal Definitions of Difficulty Proxies

The proxies populating each quadrant are formally specified using explicit mathematical notation and domain-informed criteria:

Task-Agnostic Human (TA–H)

Instance-level text heuristics, requiring no labels or model access.

• Sentence length: $d_{\text{length}}(x) = N$ for N tokens.
• Word rarity: $$d_{\text{rarity}}(x) = \frac{1}{N} \sum_{i=1}^N {-\log F(w_i)}$$, with $F(w)$ corpus frequency.
• Flesch Reading Ease: $$d_{\text{FRE}}(x) = 206.835 - 1.015 \cdot (\#\text{words}/\#\text{sentences}) - 84.6 \cdot (\#\text{syllables}/\#\text{words})$$
• Syntactic complexity: $$d_{\text{complexity}}(x) = \frac{1}{N} \sum_{i=1}^N \text{depth}(w_i)$$
• POS diversity: $$d_{\text{diversity}}(x) = |\{ \text{POS}(w_i) | i=1..N \}|$$
• Psycholinguistic norms (AoA, concreteness, prevalence): $$d_{\text{AoA}}(x) = \frac{1}{N} \sum_i \mathrm{AoA}(w_i)$$, analogously for other norms.

Task-Agnostic Model (TA–M)

Instance-level scores from pretrained, task-agnostic models.

• Masked-token perplexity (e.g., via BERT): $$d_{\text{ppl}}(x) = \exp\Big(-\tfrac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} L(w_i | x_{\setminus i})\Big)$$, for masked set $\mathcal{M}$.

Task-Dependent Human (TD–H)

Human annotation-derived measures conditioned on task.

• Label entropy: Given K labels with empirical probabilities $p_k = n_k / \sum_j n_j$, $d_{\text{entropy}}(x) = -\sum_{k=1}^K p_k \log p_k$.

Task-Dependent Model (TD–M)

Model-internal statistics over training on a specific task.

• Correct-label confidence: $c_e(x) = P_{\theta_e}(y | x)$ at epoch $e$.
• Correctness indicator: $$\mathrm{corr}_e(x) = \mathbb{I}\big[\arg\max P_{\theta_e}(\cdot | x) = y\big]$$
• Variability of confidence: $$d_{\text{var}}(x) = \sqrt{ \tfrac{1}{E} \sum_e (c_e(x) - \bar{c}(x))^2 }$$, with $\bar{c}(x) = \tfrac{1}{E} \sum_e c_e(x)$
• Training loss mean and std: $$\ell(x) = \tfrac{1}{E} \sum_e \mathcal{L}(\theta_e; x, y)$$

Variants in physical reasoning (block construction) define axes as:

• Physical Effort (E): Total kinetic work,

$$E = \sum_{k=1}^{K} \sum_{b} \tfrac{1}{2} m_b \|v_{b,k}\|^2$$

• Physical Risk (R): Empirical collapse probability after Gaussian perturbation, $R = 1 - \tfrac{1}{N} \sum_{j=1}^N S_j$, with $S_j$ indicating stability in MC simulation.

LLM analysis proxies include:

• Human-labeled difficulty ($y^H$): e.g., Item Response Theory on human performance.
• LLM-derived difficulty ($y^L$): e.g., LLM aggregate solution rates from benchmarks.

3. Quadrant Interpretation Across Domains

Curriculum Learning and NLU

• TA–H: Inspired by readability and psycholinguistics, these signals reflect assumptions about inherent linguistic complexity.
• TA–M: Language-model perplexity reflects distributional "typicality" under unsupervised pretraining.
• TD–H: Annotation entropy quantifies human disagreement, signaling items perceived as ambiguous or ill-defined.
• TD–M: Model learning dynamics directly capture "how hard" specific instances are for neural models to master during finetuning (Toborek et al., 4 Jan 2026).

Physical Manipulation (Block Building)

• Low Effort & Low Risk: Short moves, stable structures—rated easiest by humans.
• Low Effort & High Risk: Short moves, but fragile assemblies—demanding precision.
• High Effort & Low Risk: Much work, robust structures—challenging in time but not in care.
• High Effort & High Risk: Both physically demanding and attentionally taxing—maximal difficulty (Yildirim et al., 2019).

LLM Difficulty Encoding and Generalization

• Q1 (Human, Stable): Human-anchored difficulty is robust, stable across model scaling and under RL finetuning, and best aligns with accuracy improvements ($\hat{\rho} \approx 0.88$, scaling exponent $\alpha \sim 0.045$).
• Q4 (Model, Degrading): LLM-derived difficulty labels degrade or invert under post-training ($\hat{\rho} \downarrow$, $\beta = -0.63$), becoming less reflective of generalization (Lugoloobi et al., 20 Oct 2025).

A summary table appears below:

Quadrant	Example Signal	Empirical Utility
TA–H / Human,Stable	Length, AoA, IRT	Poor for model difficulty
TA–M / Model,Stable	LM Perplexity	Orthogonal to model error
TD–H / Human,Stable	Annotation entropy	Aligns with model loss
TD–M / Model,Degrading	LLM GSM8K difficulty	Misaligns after RL

4. Empirical Correlation Structure and Quantitative Results

Comprehensive empirical studies quantitatively benchmark the relationships among quadrants' proxies:

• Within TA–H: Most pairwise Pearson $r < 0.5$, often $r < 0.2$, except length/complexity ($r \approx 0.71$), AoA/FRE ($r \approx -0.58$); demonstrates orthogonality of surface-level features.
• TA–H ↔ TA–M: Virtually uncorrelated ($|r| < 0.1$); LM perplexity is distributionally rather than heuristically grounded.
• TA–H ↔ TD–H/TD–M: No meaningful alignment ($|r| \lesssim 0.1$); R² for multivariate prediction $< 0.1$.
• TD–H ↔ TD–M: Moderate positive association ($r \approx 0.4-0.6$); models and humans agree on ambiguity.
• Physical effort/risk model: Full model outperforms single-factor baselines in all quadrants ($r = 0.84-0.92$); only joint modeling recovers human trade-offs (Yildirim et al., 2019).
• LLM activation-probe structure: Probes for human-labeled difficulty remain predictive or improve as models scale or undergo RL; probes for model-derived difficulty degrade markedly when models improve ($\hat{\rho}$ drops by up to 50%) (Lugoloobi et al., 20 Oct 2025).

5. Implications, Limitations, and Recommendations

Empirical findings decisively show that task-agnostic difficulty signals—whether human or model-based—do not reliably predict which instances are difficult for neural learners. Their utility in curriculum learning appears to derive from auxiliary effects (e.g., pacing, regularization) rather than accurate difficulty estimation. Conversely, task-dependent human or model signals, particularly those that remain stable or improve under training, offer direct alignment with both human perception and model generalization (Toborek et al., 4 Jan 2026, Lugoloobi et al., 20 Oct 2025).

However, task-dependent human signals (e.g., annotation entropy, IRT) are costly to collect at scale, and model-derived proxies based on superficial statistics may become misaligned or even misleading as models improve.

Therefore, there is a need for:

• Lightweight, scalable, and task-aware proxies: e.g., probe classifiers on raw inputs, unsupervised task-space estimators, or training-dynamics metrics that maintain alignment as models evolve.
• Validation under model learning: System designers must empirically verify whether new or existing proxies are stable (Q1/Q3) or degrade (Q2/Q4) with model improvement.
• Combined/ensemble signals: Leveraging human-anchored signals for reliability and selected model-derived signals for adaptivity is recommended for robust curriculum and diagnostic pipelines (Lugoloobi et al., 20 Oct 2025).

6. Generalization to Other Domains and Framework Evolution

The four-quadrant approach generalizes across problem domains where difficulty is multi-factorial and observer-dependent. In physical reasoning, axes parameterize effort (objective work) and risk (precision/collapse), capturing the dual facets of human challenge (Yildirim et al., 2019). In language, axes span from shallow surface cues to deep training-dynamics statistics, anchored in both annotation-derived ambiguity and model uncertainty.

Potential extensions include recasting axes as measure provenance (e.g., external/expert vs. agent-internal), adaptivity under feedback, or domain-specific physical/cognitive dimensions. For LLM analysis, dynamic stability of signals under training emerges as a critical discriminator for trustworthy proxies. Ongoing work explores which signals, new or composite, can serve as lightweight, scalable surrogates for ground-truth challenge, supporting principled adaptive evaluation and curriculum design.

The four-quadrant categorisation thus provides a unifying, rigorous framework for dissecting, selecting, and improving difficulty signals—clarifying which signals matter, for whom, and under what modeling or empirical regimes (Toborek et al., 4 Jan 2026, Yildirim et al., 2019, Lugoloobi et al., 20 Oct 2025).