Vygotsky Distance: NLP Task Similarity Metric

• Vygotsky distance is a model-centric metric that measures similarity between NLP tasks by comparing normalized inversion counts of model ranking permutations.
• It enables efficient benchmark compression by identifying redundant tasks and selecting optimal subsets for validation with minimal performance loss.
• The method leverages MST visualization and clustering to reveal underlying task structures, supporting robust model evaluation and reliable performance prediction.

Vygotsky distance is a model-centric metric for quantifying the similarity between benchmark tasks based on the relative performance of machine learning models. Designed for NLP evaluation, Vygotsky distance measures how consistently various models (“students”) rank across different tasks, thereby forming a metric space over the set of tasks. This approach enables efficient benchmark compression, redundancy analysis, and principled selection of task subsets for model validation (Surkov et al., 2024).

1. Formal Definition and Metric Structure

Given a collection of $T$ tasks and $N$ models, each task $A$ is represented by a performance vector

$$p^{(A)} = (p_1^{(A)}, p_2^{(A)}, \ldots, p_N^{(A)}),$$

where $p_i^{(A)}$ is the score of model $i$ on task $A$. The corresponding task rankings are encoded as permutations $\pi_A$ obtained by sorting $p^{(A)}$ in descending order.

The unnormalized Vygotsky distance $w(A,B)$ between tasks $N$0 and $N$1 is defined as the number of model pairs whose order is reversed between $N$2 and $N$3:

$N$4

The normalized Vygotsky distance is

$N$5

with $N$6.

The metric properties—non-negativity, symmetry, triangle inequality, and identity of indiscernibles—are all satisfied, enabling application of metric-space analytical tools.

2. Theoretical Motivation

Traditional task similarity measures in benchmarks are often “dataset-first,” focusing on dataset characteristics, annotation schemas, or input distributions. Vygotsky distance adopts a “learner-first” perspective, comparing tasks in terms of the induced relative model orderings. Two tasks are considered close if models achieving high scores on one also perform well on the other, and distant if their relative model rankings differ substantially.

This model-centric methodology directly addresses practitioners’ needs: generalization, redundancy quantification, and validation fidelity. Because Vygotsky distance is agnostic to dataset domain, format, or label choices, it robustly identifies practical similarities and differences among tasks, even when their surface features diverge (Surkov et al., 2024).

3. Algorithmic Computation and Complexity

The protocol for computing the Vygotsky distance matrix proceeds as follows:

1. Leaderboard Matrix Construction: Collect $N$7 such that $N$8. Normalize scores per task if metrics differ.
2. Permutation Induction: For each task $N$9, obtain $A$0 by sorting $A$1.
3. Distance Calculation: For each pair of tasks $A$2 with $A$3, count the number of inversions (model pairs where order is reversed) and normalize by $A$4.
4. Metric Matrix Formation: Populate the symmetric $A$5 distance matrix.

The overall time complexity is $A$6, which is practical for benchmarks of standard size (e.g., $A$7, $A$8).

This metric structure supports downstream analyses such as construction of minimum-weight spanning trees (MSTs), clustering, and benchmark redundancy evaluation.

4. Benchmark Compression and Predictive Modeling

Vygotsky distance enables principled benchmark compression via public/private splits of tasks. By selecting a subset (“public”) of tasks for direct evaluation and learning a predictor to estimate performance or rank on the remaining (“private”) tasks, it is possible to retain high model comparison and score estimation fidelity with substantially fewer evaluation tasks.

The procedure involves exhaustive enumeration of public/private partitions (for $A$9 tasks, $$p^{(A)} = (p_1^{(A)}, p_2^{(A)}, \ldots, p_N^{(A)}),$$0 splits). Predictors—binary classifiers for rank comparison or regressors for score estimation—are trained using the model performance vectors from the public tasks to predict average or relative outcomes on private tasks.

Compression rates and predictive performance are summarized below:

Benchmark	T (orig)	T' (public)	Compression	Class. Acc.	Regr. RMSE
GLUE	9	4	44%	0.80	0.18
SuperGLUE	12	5	42%	0.81	0.20
CLUE	10	4	40%	0.79	0.22
RussianSuperGLUE	9	4	44%	0.82	0.19

Average compression across 30 NLP benchmarks is approximately 40%, with classification accuracy around 0.80 and regression RMSE near 0.20 on held-out tasks at this level (Surkov et al., 2024).

5. Empirical Findings

Evaluation across GLUE, SuperGLUE, CLUE, RussianSuperGLUE, and 26 additional NLP benchmarks led to the following key results:

• An optimal compression rate is achieved at approximately 40% of the original tasks retained for direct evaluation.
• With SVM, GP, and MLP predictors, task ranking reconstruction on held-out tasks exceeds 80% accuracy; average score predictions achieve RMSE $$p^{(A)} = (p_1^{(A)}, p_2^{(A)}, \ldots, p_N^{(A)}),$$1.
• Both classification accuracy and regression RMSE exhibit U-shaped dependence on the compression rate: performance declines for both very small and very large public sets, with a stable optimum near 40%.

Aggregated predictor performance (≤40% compression):

Predictor	Acc	F1	Prec	Recall	AUC	MSE	RMSE	MAE	MaxErr	R²
SVM	0.74	0.70	0.64	0.83	0.71	0.33	0.20	0.26	0.61	0.41
GP	0.79	0.67	0.68	0.72	0.78	0.40	0.35	0.29	0.72	0.00
MLP	0.78	0.75	0.80	0.95	0.70	0.49	0.55	0.42	0.79	0.00

This demonstrates that a typical NLP benchmark can be reduced by approximately 60% with minimal loss in evaluation fidelity (Surkov et al., 2024).

6. Structural Insights and Visualization

Metric-space tools such as MST visualization reveal the structure of task sets in practical benchmarks. MSTs expose “convex” clusters of highly interchangeable tasks, for example, paraphrase recognition datasets, as well as structural outliers like RTE among NLI tasks. This facilitates targeted dataset development and points to redundant or underrepresented regions in the task space.

In dynamic evaluation settings, maintaining a small public set for frequent validation and forecasting private component scores offers substantial efficiency in continuous integration and deployment pipelines.

7. Limitations and Future Directions

The current methodology has been validated exclusively on NLP tasks; extension to vision, speech, and other domains remains untested, though readily applicable. The exhaustive split enumeration scales exponentially with $$p^{(A)} = (p_1^{(A)}, p_2^{(A)}, \ldots, p_N^{(A)}),$$2; for large-scale benchmarks, greedy heuristic subset selection becomes necessary. Accurate Vygotsky distance estimation requires a moderately sized pool of pre-evaluated models, and results degrade for new tasks with sparse baselines.

Research avenues include generalization to streaming and continual task arrival, incorporation of model-family priors in inversion weighting, approximation methods for scaling to large $$p^{(A)} = (p_1^{(A)}, p_2^{(A)}, \ldots, p_N^{(A)}),$$3, and leveraging metric-space frameworks (such as spectral clustering or persistent homology) to identify latent task communities (Surkov et al., 2024).