DIEM: Dimension Insensitive Euclidean Metric

• DIEM is a framework that neutralizes dimensional bias in Euclidean metrics by bounding per-coordinate contributions and normalizing distance scales.
• It enhances robustness and interpretability in high-dimensional data analysis, supporting reliable k-NN classification and efficient metric compression.
• DIEM outperforms traditional measures by providing consistent similarity judgments, mitigating outlier effects, and improving storage and retrieval performance.

The Dimension Insensitive Euclidean Metric (DIEM) encompasses a class of methodologies and metrics designed to achieve robustness, interpretability, and optimality in the analysis and compression of high-dimensional data spaces where traditional distance measures, such as the standard Euclidean distance or cosine similarity, exhibit dimension-dependent pathologies. The term "DIEM" refers specifically to metric constructions that minimize or eliminate the dependence of dissimilarity estimates on the ambient dimension, thereby supporting fair and stable vector comparison, clustering, and information retrieval in large-scale, high-dimensional systems (Hassanat, 2014, &&&1&&&, Indyk et al., 2021).

1. Formal Definitions and Instantiations

Several independent proposals for DIEM have emerged, each targeting a different facet of the dimensionality problem in metric spaces. Two principal forms are:

1.1 Per-Coordinate DIEM (Hassanat, 2014)

Let $$A = (A_1, ..., A_m), B = (B_1, ..., B_m) \in \mathbb{R}^m$$. Define

$$f(a, b) = \begin{cases} 1 - \frac{1+\min(a, b)}{1+\max(a, b)} & \text{if } \min(a, b) \ge 0 \ 1 - \frac{1+\min(a, b) + |\min(a, b)|}{1+\max(a, b) + |\min(a, b)|} & \text{if } \min(a, b) < 0 \end{cases}$$

The DIEM is then

$D_{\mathrm{DIEM}}(A, B) = \sum_{i=1}^m f(A_i, B_i)$

The per-coordinate contribution $f(a, b) \in [0, 1)$ is upper-bounded, ensuring that no single outlier dominates. As $|a-b| \to \infty$, $f(a, b) \to 1$; thus, $D_{\mathrm{DIEM}}(A, B) < m$ (Hassanat, 2014).

1.2 Detrended and Variance-Normalized Euclidean DIEM (Tessari & Hogan, 2024)

For $a, b \in \mathbb{R}^n$ with $a_i, b_i \in [V_m, V_M]$, let

$$d_n(a, b) = \| a - b \|_2 = \sqrt{ \sum_{i=1}^n (a_i - b_i)^2 }$$

Let $E[d_n]$ denote the expected Euclidean distance for random pairs $(a, b)$ and $\sigma_n^2 = \mathrm{Var}[d_n]$. DIEM is defined as

$$\mathrm{DIEM}_n(a, b) = (V_M - V_m) \frac{d_n(a, b) - E[d_n]}{ \sigma_n^2 }$$

This construction removes the $\sqrt{n}$ growth of Euclidean distance, normalizes variability, and restores units (Tessari et al., 2024).

1.3 DIEM in Euclidean Metric Compression (Andoni et al., 2021)

"DIEM" also labels a dimension-insensitive hierarchical compression scheme for $\ell_2$ metrics. The approach combines tree-based hierarchical clustering, randomized grid rounding, and unbiased estimation to achieve near-optimal sketches of distance information—outperforming Johnson–Lindenstrauss approaches for large $n$ (Indyk et al., 2021).

2. Metric and Statistical Properties

The per-coordinate DIEM of (Hassanat, 2014) is a true metric:

• Non-negativity: $f(a, b) \ge 0$
• Identity: $f(a, b) = 0 \iff a = b$
• Symmetry: $f(a, b) = f(b, a)$
• Triangle inequality: $f(a, c) \le f(a, b) + f(b, c)$

Summing these properties yields a bona-fide metric $D_{\mathrm{DIEM}}$.

The detrended DIEM (Tessari et al., 2024) is a standardized dissimilarity score:

• Symmetric and translation-invariant.
• Not a metric: can take negative values and does not satisfy the triangle inequality.
• Centered mean ($0$) and constant variance $\sim$ for $n \gtrsim 7$.
• Bounded only via the data range $[V_m, V_M]$, with landmarks:
  • $\mathrm{DIEM}_{\min}$ (identical vectors), $\mathrm{DIEM}_{\mathrm{orth}}$ (orthogonal pairs), $\mathrm{DIEM}_{\max}$ (maximally dissimilar).
• Distributions of DIEM converge rapidly with increasing dimension $n$, in contrast to cosine similarity or normalized Euclidean.

3. Comparative Analysis with Standard Distance Measures

Traditional high-dimensional similarity scores—such as cosine similarity and Euclidean distance—exhibit strong dimension dependence:

Metric	Mean as $n \to \infty$	Variance as $n \to \infty$	Boundedness
Cosine similarity	$\sim$ constant or $0$	$\to 0$	Yes, $[0, 1]$
Normalized Euclidean	$\sim$ constant	$\to 0$	Yes
Raw Euclidean norm	$\propto \sqrt{n}$	$\sim$ constant	No
DIEM (detrended)	$0$	$\sim$ constant	No (interpretable)

Cosine similarity exhibits variance collapse, making discrimination poor at large $n$. Raw Euclidean grows with $\sqrt{n}$, distorting comparisons. The per-coordinate DIEM saturates each axis’s contribution, making the measure robust to outliers and insensitive to dominant coordinates. The detrended/normed DIEM centers similarity judgments at a random baseline and ensures statistical interpretability regardless of $n$ (Tessari et al., 2024).

4. Algorithmic Integration and Complexity

4.1 Per-Coordinate DIEM in 1-NN Classification

• No adjustment required to k-NN algorithms beyond replacing the distance kernel.
• Computational complexity $O(n m)$ per test sample, identical to ordinary Euclidean; the principal overhead is additional per-coordinate arithmetic (min, max, abs, division) (Hassanat, 2014).
• Outperforms Euclidean, Manhattan, and older “Wave–Hedges” measures on numerous real datasets (mean accuracy: DIEM $0.81$, ED $0.78$, MD $0.79$, Wave-Hedges $0.56$ on 19 benchmarks).

4.2 DIEM in Metric Compression

• Sketch size (in bits) for $n$ points, distortion $\epsilon$, aspect ratio $\Phi$:

$$b = \Theta(\epsilon^{-2} n \log n + n \log \log \Phi)$$

• Algorithm entails (i) constructing a hierarchical tree over data, (ii) randomized grid quantization for leaves, and (iii) unbiased, low-variance surrogates for pairwise distances (Indyk et al., 2021).
• Provably surpasses Johnson–Lindenstrauss reduction in storage.

5. Empirical and Simulated Evaluations

5.1 Nearest Neighbor Classification

For per-coordinate DIEM (Hassanat, 2014):

• Benchmarked on 19 UCI datasets spanning a wide variety of feature domains and class cardinalities.
• Classification with $k=1$, 10-fold randomized train/test splits; DIEM delivered highest mean accuracy, particularly excelling over metrics that fail with zero and negative values.

5.2 Monte Carlo Simulations for High-Dimensional Similarity

For detrended DIEM (Tessari et al., 2024):

• Random vector pairs sampled from domains (all real, positive, negative).
• Cosine, normalized Euclidean, and raw Euclidean all show strong dimension effects (variance collapse or mean drift).
• DIEM yields invariant null distributions for $n \gtrsim 7$ in all tested domains.
• No empirical real-world (e.g., LLM) application is documented; all results pertain to controlled simulation.

6. Applications, Limitations, and Open Problems

DIEM frameworks support various goals in high-dimensional data analysis:

• Robust similarity measures for k-NN, PCA, SVD, embedding-based information retrieval, neuromotor control synergies (Hassanat, 2014, Tessari et al., 2024).
• Consistent thresholding across models of different sizes due to dimension-invariant variance.
• Efficient metric compression for distance-preserving sketching and fast search (Indyk et al., 2021).

Known limitations include:

• Per-coordinate DIEM may under-emphasize moderate differences when many features differ.
• The detrended DIEM is not a true metric; interpretation of negative values depends on context; triangle inequality fails.
• Estimation of expected value and variance (for normalization) may lack closed forms or require sampling.
• Extension to non-Euclidean norms (e.g., $L_1$, $L_\infty$) and empirical validation on non-simulated (real-world) data or deep learning embeddings remain open directions (Tessari et al., 2024).

7. Concluding Remarks and Prospective Directions

DIEM methodologies fundamentally address the bias, interpretability, and compressibility challenges endemic to classical high-dimensional distance measures. By bounding, normalizing, or detrending per-coordinate or overall vector contributions, DIEM promises more consistent and fair vector comparisons, supports more efficient storage and retrieval, and unlocks new avenues for hypothesis testing in machine learning, neuroscience, and related technical fields. Further work is required to adapt DIEM approaches for generalized norms, empirical machine learning datasets—including LLM embeddings—and to identify strictly non-negative, triangle-inequality-satisfying variants suitable for broader applications (Hassanat, 2014, Tessari et al., 2024, Indyk et al., 2021).