Data Kernel Perspective Space (DKPS)

• DKPS is a mathematical framework that embeds black-box generative model outputs into a Euclidean space, enabling robust statistical comparisons.
• It employs classical multidimensional scaling on mean sample embeddings to create low-dimensional representations for visualization and performance prediction.
• DKPS is applied in synthetic data benchmarking and model performance tracking, offering query-efficient evaluation with asymptotic consistency guarantees.

The Data Kernel Perspective Space (DKPS) is a mathematical and algorithmic framework for comparing black-box generative models, including LLMs and neural machine translation (NMT) systems, using only their observable outputs to a shared set of queries. DKPS provides a Euclidean geometry in which each model is represented by a point; pairwise distances in this space correspond to the statistical similarity of the models’ output distributions. DKPS enables rigorous empirical comparison of model behaviors, supports query-efficient performance prediction, and provides asymptotic performance guarantees for quality assessment of synthetic data generators. The term “kernel” in DKPS refers to the use of modern embedding techniques for quantifying model output differences, though classical Mercer kernel or RKHS structure is not always strictly enforced in all applications.

1. Formal Definition and Mathematical Construction

Let $\mathcal F$ be a family of black-box generative models $f : \mathcal Q \to \mathcal X$, where $\mathcal Q$ is the query/input space and $\mathcal X$ the space of responses. Fix an embedding $g: \mathcal X \to \mathbb R^p$ (e.g., a sentence or feature embedding) and a finite query set $Q = \{q_1, \dots, q_m\} \subset \mathcal Q$. For $n$ models $f_1, \dots, f_n \in \mathcal F$, let $r$ denote the number of output replicates per model-query pair.

• For each $i$ and $f : \mathcal Q \to \mathcal X$0, the (empirical) mean sample embedding:

$f : \mathcal Q \to \mathcal X$1

Stack to $f : \mathcal Q \to \mathcal X$2 for model $f : \mathcal Q \to \mathcal X$3.

• The pairwise distance matrix $f : \mathcal Q \to \mathcal X$4 is:

$f : \mathcal Q \to \mathcal X$5

(Frobenius norm on $f : \mathcal Q \to \mathcal X$6 matrices).

• Classical multidimensional scaling (MDS) is applied to $f : \mathcal Q \to \mathcal X$7 to find $f : \mathcal Q \to \mathcal X$8 points in $f : \mathcal Q \to \mathcal X$9 minimizing the raw-stress objective:

$\mathcal Q$0

• The result is the DKPS embedding $\mathcal Q$1, with $\mathcal Q$2 for each $\mathcal Q$3.

This construction is robust to model architecture and modality, requiring only black-box access to model outputs and a suitable embedding $\mathcal Q$4.

2. Statistical Guarantees and Theoretical Results

A primary theoretical result for DKPS is the convergence of the empirical MDS embedding $\mathcal Q$5 to its population counterpart $\mathcal Q$6 as the number of output replicates $\mathcal Q$7. Under bounded covariance conditions on $\mathcal Q$8, for fixed $\mathcal Q$9 models and $\mathcal X$0 queries, Acharyya et al. (2024) establish:

$\mathcal X$1

for all model pairs $\mathcal X$2 as $\mathcal X$3 (Acharyya et al., 2024).

Extensions cover:

• Growing $\mathcal X$4 (number of queries),
• Simultaneous growth of $\mathcal X$5 (number of models, queries, replicates), with analogous convergence, provided the average trace-covariance of $\mathcal X$6 decays appropriately.

DKPS-based $\mathcal X$7-NN regression for performance prediction is also shown to be minimax and query-efficient under Lipschitzness and sufficient reference coverage (Helm et al., 8 May 2026). No generalization bound of the form $\mathcal X$8 is given, and in many published applications, the “kernel” refers purely to the embedding-induced distance rather than to a Mercer kernel.

3. Practical Computation and Application Scenarios

The DKPS workflow consists of:

1. Selecting $\mathcal X$9 models, $g: \mathcal X \to \mathbb R^p$0 queries, an embedding $g: \mathcal X \to \mathbb R^p$1, and number of replicates $g: \mathcal X \to \mathbb R^p$2,
2. Computing mean response embeddings $g: \mathcal X \to \mathbb R^p$3,
3. Constructing the pairwise distance matrix $g: \mathcal X \to \mathbb R^p$4,
4. Applying classical MDS (solving via eigen-decomposition of the double-centered Gram matrix),
5. Using learned $g: \mathcal X \to \mathbb R^p$5 for downstream tasks.

This yields a $g: \mathcal X \to \mathbb R^p$6-dimensional geometry suitable for visualization, clustering, or regression.

Notable application domains include:

• Synthetic data benchmarking: Assessing when a synthetic-data model is “close” to human-annotated data (according to $g: \mathcal X \to \mathbb R^p$7).
• Query-efficient model evaluation: Predicting full-benchmark scores for new models, given a small query budget and cached responses (Helm et al., 8 May 2026).
• Performance tracking of generative models: Monitoring change in outputs after fine-tuning, prompt modifications, or other interventions.

A common pseudocode outline is as follows:

$n$7

Table 1 organizes the key stages:

Step	Input	Output
1. Embed	Model outputs, embedding $g: \mathcal X \to \mathbb R^p$8	$g: \mathcal X \to \mathbb R^p$9
2. Distances	$Q = \{q_1, \dots, q_m\} \subset \mathcal Q$0	$Q = \{q_1, \dots, q_m\} \subset \mathcal Q$1
3. MDS	$Q = \{q_1, \dots, q_m\} \subset \mathcal Q$2, dimension $Q = \{q_1, \dots, q_m\} \subset \mathcal Q$3	$Q = \{q_1, \dots, q_m\} \subset \mathcal Q$4

4. Empirical Examples and Performance Benchmarks

DKPS has been applied to:

• Neural Machine Translation (NMT):
  • English→Zulu back-translation: Sequential (top-1) and batch (top-$Q = \{q_1, \dots, q_m\} \subset \mathcal Q$5) synthetic data from a Sockeye transformer.
  • Embeddings computed using LASER3, reduced by PCA.
  • DKPS revealed that batch-generated synthetic data exhibits higher variance and bias, mapping to a broader region in DKPS space and farther from the human reference (Browder et al., 4 Feb 2026).
• Contrastive Preference Optimization (CPO) in LLM fine-tuning:
  • Preferred (sequential) and dispreferred (batch) outputs modeled as Gaussian in reduced embedding space.
  • DKPS with Mahalanobis distance captured the geometric “bleeding” of variance/bias between model output types post-CPO (Browder et al., 4 Feb 2026).
• Benchmark prediction with cached responses:
  • On HELM-Lite (Math, LegalBench, MedQA, WMT-14) with up to $Q = \{q_1, \dots, q_m\} \subset \mathcal Q$6 models and $Q = \{q_1, \dots, q_m\} \subset \mathcal Q$7 queries, DKPS-based regression halved or better the mean absolute error (MAE) compared to direct sample-mean estimators.
  • Offline query selection via DKPS-maximization further improved accuracy at low $Q = \{q_1, \dots, q_m\} \subset \mathcal Q$8 (Helm et al., 8 May 2026).

A key insight is that offline selection of query sets (maximizing $Q = \{q_1, \dots, q_m\} \subset \mathcal Q$9 fit of DKPS-regression on references) is critical for maximizing predictive value, especially when the query budget is low.

5. Relation to Kernel Methods and Limitations

Despite its nomenclature, in most DKPS literature, “kernel” refers to the embedding-induced dissimilarity rather than an explicit positive-definite kernel or feature map into a Reproducing Kernel Hilbert Space (RKHS). All actual DKPS computations use (empirical) Euclidean or Mahalanobis distances. No methodology for kernel selection, tuning, or formal construction of RKHS structure is developed in the major references (Browder et al., 4 Feb 2026, Helm et al., 8 May 2026, Acharyya et al., 2024).

In work on few-shot vision-language adaptation, true RKHS-based methods (e.g., ProKeR) do explicitly invoke a Mercer kernel $n$0 and solve regularized regression in $n$1 (Bendou et al., 19 Jan 2025), but this is distinct from the DKPS pipeline conceptualized for generative-model comparison.

Principal limitations include:

• Lack of generalization or finite-sample performance bounds beyond MDS-consistency as $n$2.
• Heuristic choice of $n$3, query set $n$4, and DKPS target dimension $n$5.
• Absence of out-of-the-box support for missing responses or stochastic evaluation metrics—though extensions are suggested (Helm et al., 8 May 2026).
• Single-task evaluation in major empirical studies.

6. Insights, Best Practices, and Future Directions

Empirical utility of DKPS has been demonstrated across language and vision tasks, with major successes in query-efficient benchmarking, geometry-driven synthetic data analysis, and regression-based performance prediction. The following best practices are emphasized in the literature:

• Use diverse, representative model reference sets and high-quality (task-relevant) embeddings $n$6 to maximize DKPS informativeness.
• Employ offline query selection to optimize DKPS regression fit when the query budget is constrained.
• Interpret DKPS geometry as an input to further empirical analysis (e.g., identification of systematic bias or variance inflation via CPO).

Future research directions include debiasing/correcting distortions identified in DKPS, integrating DKPS geometry as a regularizer into fine-tuning pipelines, and extending DKPS comparison to additional tasks such as summarization or retrieval (Browder et al., 4 Feb 2026). Alternative (nonlinear) embeddings and distance metrics, as well as theoretical extensions to partial-coverage and stochastic settings, are also active areas.

7. References and Comparative Summary

Reference	Focus	Key Results
(Browder et al., 4 Feb 2026)	DKPS for transformer-synthetic data evaluation	Consistency, NMT & CPO case studies
(Helm et al., 8 May 2026)	Query-efficient evaluation via cached responses	Theory & empirics for DKPS regression
(Acharyya et al., 2024)	DKPS consistency theory	General convergence guarantees
(Bendou et al., 19 Jan 2025)	ProKeR: RKHS perspective for vision-language	Global KRR outperforms local adapters

DKPS is established as a robust, model-agnostic framework for embedding and comparing generative models using black-box responses. Its main formal guarantee is the asymptotic consistency of MDS-based geometry with increasing data. The framework’s strength lies in algorithmic simplicity, empirical utility, and extensibility to diverse downstream tasks rather than in a tight connection to classical kernel methods or explicit rate guarantees.