Cheatability Scores in Networks & Assessments

• Cheatability scores are quantitative measures defining cheating propensity by integrating network structure, statistical indices, and dynamic penalties.
• They are applied in social networks, educational assessments, online games, and automated grading to gauge and manage systemic vulnerabilities.
• Effective cheatability metrics inform preventive strategies and robust design choices to mitigate cheating via algorithmic and social interventions.

Cheatability scores are quantitative measures that capture the potential or propensity for individuals, systems, or configurations to enable, tolerate, or propagate cheating within a given domain. These scores arise from empirical and theoretical research across social networks, educational assessments, online games, automated grading, ranking systems, and LLM benchmarks. Cheatability scores are not monolithic metrics but reflect context-specific risk factors, statistical deviations, network structures, dynamic penalties, and susceptibility to adversarial exploitation.

1. Social Mechanisms and Embedding: Steam Community Example

In large-scale online gaming platforms such as the Steam Community, cheatability is fundamentally shaped by the structure of the social network and dynamic interactions among users (Blackburn et al., 2011). Cheaters in Steam are not marginal figures; their degree distribution within the declared friendship network closely mirrors that of fair players, both following a power law distribution with an exponent of approximately –0.92. However, detailed analysis uncovers significant homophily: about 70% of fair players have zero cheater friends, yet 70% of cheaters have at least 10% of their friends also flagged for cheating, and 15% of cheaters have more than half their network as cheaters.

Temporal propagation studies demonstrate that players with a higher absolute number and fraction of cheater friends are statistically more likely to become cheaters within subsequent intervals. Thus, cheatability scores in such contexts are emergent from both local network structure (e.g., fraction of cheater neighbors) and network position.

Cheatability is also influenced by social penalties: after being flagged, about 10% of cheaters restrict their privacy settings (versus 3% of non-cheaters), and 43% lose friends. These penalties feedback into the network, potentially reducing further propagation. Relationship closeness is quantifiable: for two users $u$ and $v$,

$$\text{Overlap}_{uv} = \frac{m_{uv}}{(k_u - 1) + (k_v - 1) - m_{uv}}$$

where $m_{uv}$ is the number of shared friends, $k_u$ and $k_v$ are degrees. Higher overlap is prevalent within homogeneous groups, informing cheatability risk models.

A cheatability score can be framed as a function of: (a) number/fraction of cheater friends; (b) centrality measures; (c) social penalties (privacy shifts and friend loss). Classifiers using such features achieve 65–74% accuracy in distinguishing cheaters from non-cheaters, indicating that network and social effect integration yields robust cheatability quantification.

2. Statistical and Algorithmic Indices in Educational Assessment

In education, cheatability scores manifest through statistically rigorous indices that measure answer-copying, with control for false positives and power (Romero et al., 2014, Richmond et al., 2015, Yamanaka et al., 2014).

The optimal answer-copying index (UMP test) frames cheat detection as:

$M_{cs} = \sum_{i=1}^N I_{csi}$

where $I_{csi}=1$ if students $c$ and $s$ give the same answer for item $i$. Under the null hypothesis of no cheating, $M_{cs}$ follows a Poisson binomial with item-specific probabilities $\pi_i$. The likelihood ratio

$$\lambda^A(x) = \frac{\hat{f}_N(x;\pi_1,\ldots,\pi_N,A)}{f_N(x;\pi_1,\ldots,\pi_N)}$$

increases with $x$; rejection of the null hypothesis occurs when $M_{cs} > k^*$, with $k^*$ controlling the type I error (false positive) rate.

Empirical studies compare $\omega$ and $\gamma$ indices. The conditional nominal response model with normal approximation ($\omega_2^s$) combines high power (low type II error) and calibration, providing practical cheatability measurement. Multiple hypothesis testing is addressed through a Bonferroni-style correction (Benjamini-Hochberg procedure), controlling the false discovery rate in massive cheating contexts.

Algorithmic methods further extend Item Response Theory by inferring sparse, pairwise cheating coefficients $w_{ik}$ within a Boltzmann machine framework (Yamanaka et al., 2014). A greedy decimation algorithm, leveraging pseudo likelihood maximization and iterative parameter zeroing, outperforms $L_1$-regularization for sparse cheating detection. ROC curves and error metrics demonstrate that cheatability, in this context, becomes a vector of inferred w's, each quantifying the degree of likely collusion between examinee pairs.

3. Quantitative Metrics in Online Gaming and Benchmark Manipulation

In games such as Wordle, cheatability scores emerge from probabilistic baselines juxtaposed with empirical user behavior (Dilger, 2023). The probability of solving a 2,315-target-word puzzle on the first try is $P_\text{random} = 0.043\%$; observed first-try rates of 0.2–0.5%—sometimes up to 1.0–5.2%—imply that thousands daily exploit external knowledge. A cheatability score may be rendered as a ratio:

$$C = \frac{\text{Observed Share \%}}{P_\text{random}}$$

where $C \gg 1$ signals significant cheatability, quantifiable at the population level.

Player loyalty to specific starting words and abrupt changes in collective guessing behavior (influenced by external cues) also inform the structural risk profile for cheating in such online settings.

4. Vulnerability of Automated Systems and Benchmarks

In automated settings (LLM benchmarks, automated grading), cheatability arises from both adversarial manipulation and systemic biases (Filighera et al., 2022, Zheng et al., 9 Oct 2024, Liang et al., 25 Jul 2025). For automated short answer grading, simple adversarial insertions of adjectives and adverbs into otherwise incorrect responses result in a 10–22 percentage point drop in grading accuracy across state-of-the-art transformers. This exposes the models' susceptibility to superficial statistical patterns, which are not easily detected by human graders—a cheatability gap between model and human assessment.

In LLM benchmarking, cheatability is reified by structured null-model outputs that can achieve top ranking scores regardless of input. On AlpacaEval 2.0, a null model achieves up to 86.5% LC win rate simply by outputting adversarially structured responses, exploiting biases in automated annotation. Adversaries can optimize prefixes using random search techniques to further boost this effect, even when using only paraphrased templates or public instruction sets. Attempts to filter adversarial outputs by perplexity are shown to be insufficient, emphasizing the challenge in designing robust automatic evaluations (Zheng et al., 9 Oct 2024).

5. Mitigating Contamination and Measuring Overestimation

A central cause of overestimated model performance—and consequent cheatability—on public benchmarks is contamination or biased overtraining on those test cases. The ArxivRoll framework addresses this by periodically generating new private benchmarks from recent arXiv articles using SCP (Sequencing, Cloze, and Prediction), and by directly quantifying overestimation via Rugged Scores (RS) (Liang et al., 25 Jul 2025). Two key metrics are defined:

• $\mathbf{RS_I}$: measures the relative shortfall between public and private benchmark scores for a model, exposing contamination.
• $\mathbf{RS_{II}}$: measures the variance of performance across private benchmarks, identifying domain-biased overtraining.

High RS values signal high cheatability: impressive public performance is due in part to contamination or unrepresentative overtraining rather than true generalization. This dynamic “one-time pad” approach ensures each benchmark is used only once, minimizing the opportunity for cheating and rendering the cheatability score a function of the discrepancy between public and private evaluations.

6. Preventive Strategies and Dynamic Penalty Effects

Effective countermeasures against cheatability involve both detection and design interventions. In MOOCs, the CAMEO strategy leverages dual accounts and fast timing; detection algorithms employ Bayesian methods, time thresholds, IP matching, and exclusion of public IP groupings (Northcutt et al., 2015). Preventive strategies such as delaying solution reveals and algorithmic assessment item generation reduce cheatability, confirmed by an order of magnitude drop in CAMEO prevalence in STEM courses that implement these interventions.

In automated benchmarks or grading, recommendations include adversarial training, low-confidence detection, human-in-the-loop review of suspicious cases, and careful management of dataset and template exposure (e.g., only paraphrased templates for annotators).

Social penalty dynamics—friendship loss, privacy restriction—feedback to reduce future cheatability by isolating detected cheaters, thereby diminishing propagation potential within networks (Blackburn et al., 2011).

7. Conceptual Synthesis and Cross-Domain Patterns

Across domains, cheatability scores integrate statistical, network, temporal, and behavioral signals:

• In social networks and online games, scores combine local structure, homophily, propagation risk, and isolation signals.
• In educational assessment, scores emerge from error-controlled statistical detection indices, interaction models, and correction for massive testing.
• In automated systems, scores measure output vulnerability, adversarial manipulation potential, and evaluation contamination.

The convergence of these methodologies highlights that cheatability is a multi-layered property: it measures not only immediate risk but also the integrity of evaluation, the interplay of social and technical defenses, and the adaptability of cheaters to evolving countermeasures. Quantitative cheatability scores thus serve both as a tool for risk management and as a guiding metric for designing more robust, fair, and generalizable systems across social, educational, and computational domains.