TinyChange Benchmark for LLM Auditing

• TinyChange Benchmark is a systematic evaluation suite that quantifies minimal model changes in large language models with fine-grained modifications.
• It employs log-probability tracking to detect subtle modifications such as single fine-tuning steps, parameter noise, and unstructured pruning with high accuracy.
• It provides a robust, cost-efficient tool for continuous LLM monitoring, ensuring reproducibility and reliable API deployment.

The TinyChange benchmark is a systematic evaluation suite designed to measure the sensitivity of audit methods to small, incremental changes in LLMs. It provides a parameterized collection of minimally modified model variants, enabling precise quantification of “minimum detectable change” for LLM monitoring methodologies—down to the level of a single fine-tuning step or fractional parameter alteration. The benchmark addresses a central challenge in LLM API deployment: ensuring model consistency in the face of unannounced, minute updates, which critically impact reproducibility and downstream application reliability. TinyChange is thus both a tool for empirical audit sensitivity analysis and a specification for robust, cost-efficient LLM model change detection (Chauvin et al., 3 Dec 2025).

1. Motivation and Design Principles

Central to the TinyChange concept is the “reproducibility gap” in LLM APIs: users expect version-pinned models to remain unchanged, while in practice, providers can—and routinely do—apply minor modifications such as fine-tuning on additional samples, unstructured weight pruning, small-scale quantization, system-prompt edits, or even backdoors. Most existing benchmarks only detect large version increments and incur substantial evaluation cost, rendering continuous post-deployment auditing infeasible.

TinyChange is designed to supply a systematic suite of LLM variants along multiple, fine-grained modification axes, serving as a ground truth scaffold for benchmarking detection power of audit mechanisms. The focus is on realistic, small model changes, in contrast to conventional benchmarks which are insensitive to such minimal updates.

2. Construction and Parameterization of Benchmark Suites

The benchmark suite comprises five open-weight LLMs, ranging from 0.5B to 8B parameters: Qwen-2.5 0.5B, Gemma-3 1B, Phi-3 Mini 4k, Llama-3.1 8B, and OLMo 2-7B. For each, variants are systematically constructed along four quantitative axes:

• Regular fine-tuning: 1 epoch, with {1, 2, 4, 8, …, 512} single-sample gradient steps
• LoRA fine-tuning (rank=8, α=8): same step-counts as above
• Unstructured pruning: removing a fraction ∈ {2⁻¹⁰, 2⁻⁹, …, 1} via either largest-magnitude or random selection
• Parameter noising: adding i.i.d. Gaussian noise, σ ∈ {2⁻¹⁵, …, 1}

Each finetuned variant uses in-distribution data: single-turn conversational data from LMSYS-Chat-1M (primarily drawn from GPT-4 output). The resulting matrix, 58 modifications × 5 models, totals 290 LLM variants (Chauvin et al., 3 Dec 2025).

Prompts for evaluation are drawn both as short stubs (even a single character “x”) and longer realistic chat queries. Empirical ablation demonstrates negligible (<1% AUC difference) detection power variation attributable to prompt length, indicating high robustness to prompt selection.

Sample generation protocol mandates S=10,000 outputs per prompt, mixing 16 specific test prompts with 48 random chat queries per batch, to emulate realistic noise conditions. For each output, the top-k=20 token log-probabilities of the first generated token are recorded.

3. Detection Methodology and Statistical Foundation

TinyChange evaluation centers on Log-Probability Tracking (LT), in which the output distribution of first-token log-probabilities forms the basis of a two-sample statistical test. For models $API^{(1)}$ and $API^{(2)}$ under test, and prompts x, the protocol is:

• Query each API with N prompts, collect top-k log-probs per output
• Construct log-prob matrices $T^{(1)}$, $T^{(2)}$ ∈ ℝ^{N×n_{tok}} over the observed token vocabulary $\mathcal V$
• Impute missing entries with sample-wise minimum log-prob
• Calculate empirical average log-prob per token for both APIs
• Compute test statistic $$S = \frac{1}{n_{\mathrm{tok}}} \sum_{i=1}^{n_{\mathrm{tok}}} |\bar a_i^{(1)} - \bar a_i^{(2)}|$$
• Estimate significance via permutation test: recompute S under B random partitions of the pooled data, yielding permutation p-value
• Reject “no change” hypothesis at significance α if $\hat p<\alpha$.

This test is optimized for sensitivity to minute model variations, and is robust to kernel noise introduced via prompt mixing.

4. Empirical Performance and Comparative Results

Performance evaluation examines both detection sensitivity (power) and efficiency (token cost):

• Detection Power:
  • LT achieves ROC-AUC > 0.9 for regular fine-tuning after only a single gradient step; alternative methods such as MMLU-ALG require ~4 steps, MET >16 steps for comparable detection
  • For pruning, LT detects down to 1/1024 weight removal (fraction = 2⁻¹⁰) with AUC > 0.9; MET only sensitive above 2⁻¹, MMLU up to 2⁻⁴
  • Across all 290 variants × 5 models: LT AUC = 0.915 (95% CI 0.864–0.958); MMLU-ALG = 0.878 (0.802–0.944); MET = 0.670 (0.612–0.731)
• Cost and Efficiency:
  • Tokens per test (input, output): LT (28, 20); MET (2.9×10⁴, 2.0×10⁴); MMLU-ALG (2.1×10⁵, 9.9×10³)
  • Projected annual cost (hourly GPT-4.1 testing): LT $0.14/year, MET$146/year, MMLU-ALG $332/year
  • LT thus represents a ≈10³× efficiency gain

Detection Method	Overall AUC (95% CI)	Token Cost (input, output)	Annual Cost (GPT-4.1, hourly)
Logprob Tracking	0.915 (0.864–0.958)	(28, 20)	$0.14
MMLU-ALG	0.878 (0.802–0.944)	(2.1×10⁵, 9.9×10³)	$332
MET	0.670 (0.612–0.731)	(2.9×10⁴, 2.0×10⁴)	$146

5. Limitations, Practical Considerations, and Future Directions

• APIs with logprob Access: Only 23% of OpenRouter endpoints return logprobs, constraining applicability of the LT procedure at present.
• Provider Evasion: Output-length constraints and ability for providers to identify and modify detection queries may restrict the approach (e.g., OpenAI requires outputs ≥16 tokens).
• Single-token Limitation: The method evaluates only the first output token; some model changes (e.g., end-of-sequence bias) may manifest in later tokens and evade detection.
• Change Attribution: LT flags changes but does not attribute them to specific causes (fine-tuning vs. pruning vs. system update).
• Methodological Extensions: Sequential/online testing (e.g., CUSUM charts), advanced prompt selection, subspace-aware statistics, and calibration of change-magnitude quantification are proposed for future work.

Integration of TinyChange-driven testing within internal or third-party infrastructure is recommended as a first-line LLM monitoring measure. When a statistically significant drift is flagged, more expensive or detailed audits can be triggered (Chauvin et al., 3 Dec 2025).

6. Context within Miniaturized Benchmarks

The term “TinyChange” also appears in the context of miniaturized evaluation suites for LLMs, specifically in the “tinyBenchmarks” methodology (Polo et al., 2024). Here, “TinyChange benchmark” refers to the design of highly compressed sample sets (e.g., k≈100 out of >10,000 examples) which preserve accuracy within ≲2% of full-benchmark results for suite-type evaluations such as MMLU, HE*LM, or AlpacaEval 2.0. A combination of stratified sampling and item-response-theory (IRT) based corrections is used to construct informative subsets. This dramatically reduces evaluation cost while maintaining high fidelity in model ranking and performance measures—the absolute error with 100 examples is ≈1.9% (MMLU) and 1.8% (Open LLM leaderboard, per-scenario). This sense of “TinyChange” benchmark is thus relevant for high-efficiency, low-variance LLM evaluation rather than fine-grained change detection per se (Polo et al., 2024).

7. Broader Impact and Research Significance

TinyChange provides a foundation for reproducible, high-sensitivity auditing and LLM evaluation under the constraints of real-world serving dynamics. The ability to detect single-step or 0.1%-scale model changes at three orders of magnitude lower cost than legacy methods enables continuous, granular API testing, facilitating accountable LLM deployment. The benchmark’s parameterized structure and open-weight model backbone support transparent and extensible research—critical for both industry and academic assessment of LLM drift and infrastructure changes.

The broader landscape includes both change detection (via statistical comparison of model outputs or logprobs) and efficient suite evaluation (via small, high-value sample sets), each leveraging the “tiny yet robust” design philosophy that is increasingly imperative as LLM evaluation scales and commercialization accelerate (Chauvin et al., 3 Dec 2025, Polo et al., 2024).