Multidimensional Scaling (MDS)

• Multidimensional Scaling (MDS) is a set of techniques that embed high-dimensional dissimilarity data into low-dimensional Euclidean spaces while preserving pairwise distances.
• It encompasses classical, metric, non-metric, Sammon mapping, and Isomap variants, each using distinct optimization methods to faithfully reconstruct data geometry.
• MDS techniques scale to large datasets via landmark and interpolation methods and extend to multi-view, non-Euclidean, and infinite-dimensional frameworks.

Multidimensional Scaling (MDS) is a foundational family of techniques in geometry-aware data analysis and manifold learning. It seeks a configuration of points in a geometric space such that the pairwise distances among these points reflect a given matrix of input dissimilarities or distances. Applications encompass dimensionality reduction, visualization, feature learning, computational linguistics, shape matching, and network analysis. MDS encompasses classical (eigendecomposition-based) algorithms, iterative stress-minimization procedures, nonlinear and manifold variants such as Isomap, as well as scalability adaptations for large data and extensions to multiple, heterogeneous dissimilarity sources.

1. Mathematical Foundations and Classical MDS

Let $\{x_i\}_{i=1}^{n}$ denote a finite collection of objects, each typically represented as a point in $\mathbb{R}^d$ (but not necessarily with explicit coordinates), and let $D = (d_{ij})$ be an $n \times n$ matrix of pairwise dissimilarities $d_{ij}$. The central goal is to find $\{y_i\}_{i=1}^n \subset \mathbb{R}^p$ for $p \ll n$ such that their Euclidean distances $\|y_i - y_j\|$ closely approximate the given dissimilarities.

Classical MDS is based on the following sequence (Ghojogh et al., 2020, Adams et al., 2019):

• Compute squared distances $D^{(2)}=(d_{ij}^2)$.
• Center with $H = I_n - (1/n) \mathbf{1}_n \mathbf{1}_n^T$.
• Form the double-centered Gram matrix: $\mathbb{R}^d$0.
• Compute the eigendecomposition $\mathbb{R}^d$1.
• Select the top $\mathbb{R}^d$2 eigenpairs $\mathbb{R}^d$3 to form the embedding: The $\mathbb{R}^d$4th point $\mathbb{R}^d$5 has coordinates $\mathbb{R}^d$6.

This configuration uniquely minimizes the so-called strain (Frobenius norm between empirical and reconstructed Gram matrices), achieving exact recovery when the input matrix $\mathbb{R}^d$7 is Euclidean (Adams et al., 2019).

2. Variants: Metric, Non-metric, Sammon, and Isomap

• Metric MDS: Seeks $\mathbb{R}^d$8 minimizing

$\mathbb{R}^d$9

for fixed positive weights $D = (d_{ij})$0 (often $D = (d_{ij})$1). The objective is nonconvex and typically tackled by gradient-based or majorization algorithms such as SMACOF (Ghojogh et al., 2020, Boyarski et al., 2017).

• Non-metric MDS: Preserves only the rank order of dissimilarities. Introduces a monotonic, typically isotonic-transform $D = (d_{ij})$2 and minimizes

$D = (d_{ij})$3

alternating between updating $D = (d_{ij})$4 and fitting $D = (d_{ij})$5 (Ghojogh et al., 2020).

• Sammon Mapping: A weighted form where

$D = (d_{ij})$6

with normalization $D = (d_{ij})$7, emphasizing fidelity for small distances (Ghojogh et al., 2020).

• Isomap: Replaces $D = (d_{ij})$8 by an estimated geodesic distance, using shortest paths on a neighborhood graph. After transforming these distances to a centered Gram matrix, Isomap applies the classical spectral embedding, capturing nonlinear manifold structures (Ghojogh et al., 2020).

3. Computation and Scalability for Large Data

For $D = (d_{ij})$9, classical algorithms become impractical due to cubic complexity. Partition-based approaches decompose the global problem:

• Landmark MDS: Selects $n \times n$0 "landmarks", runs classical MDS on them, and triangulates new points via analytic formulas (Gower’s interpolation) (Delicado et al., 2020).
• Interpolation MDS, Reduced MDS, Pivot MDS, Divide-and-Conquer MDS, Fast MDS: Variants that partition, align, and stitch together local MDS results; see Table 1.

Algorithm	Complexity (Time)	Memory	Comment
Landmark MDS	$n \times n$1	$n \times n$2	Efficient, low-memory
Interp MDS	$n \times n$3	$n \times n$4	Fastest in benchmark
Pivot MDS	$n \times n$5	$n \times n$6	Direct inner-product
Div&Conq MDS	$n \times n$7	$n \times n$8	More robust, slightly biased
Fast MDS	$n \times n$9	$d_{ij}$0	Recursive, higher const

Empirical evaluations on million-point EMNIST images demonstrated that all these methods closely match the quality of classical MDS but differ in time/memory footprint (Delicado et al., 2020).

4. Extensions: Multiple Views, Non-Euclidean Embeddings, and Feature Learning

• Multi-view MDS (MVMDS): For $d_{ij}$1 heterogeneous distance matrices (views) on the same objects, minimizes a weighted sum of stresses with automatic weight learning:

$d_{ij}$2

with controller parameter $d_{ij}$3. Alternating minimization yields both the embedding $d_{ij}$4 and optimal view-weights $d_{ij}$5 (Bai et al., 2016).

• Feature Learning via MDS: MDS can serve as a feature learning framework when applied to high-level, semantics-sensitive distances (e.g., spatial pyramid matching) between objects rather than raw pixel or Euclidean distances (Wang et al., 2013). The process learns vector codes so that Euclidean distances between codes reflect semantic similarity.
• Non-Euclidean Targets: MDS can also be adapted to hyperbolic or spherical target spaces, with corresponding definitions of distance and gradient, enabling more faithful embeddings for inherently hierarchical or non-flat data (Cvetkovski et al., 2011, Liu et al., 2022).

5. Infinite, Continuous, and Operator-Theoretic Generalizations

MDS extends to infinite metric measure spaces $d_{ij}$6 by defining the "double-centered" kernel:

$d_{ij}$7

and embedding via the spectrum of the associated Hilbert-Schmidt operator $d_{ij}$8 on $d_{ij}$9. The canonical embedding $\{y_i\}_{i=1}^n \subset \mathbb{R}^p$0 (with $\{y_i\}_{i=1}^n \subset \mathbb{R}^p$1) optimally minimizes infinite-sample "strain" (Adams et al., 2019, Kassab, 2019).

Continuous MDS frameworks permit the analysis of limit behavior as $\{y_i\}_{i=1}^n \subset \mathbb{R}^p$2, establishing $\{y_i\}_{i=1}^n \subset \mathbb{R}^p$3-consistency of embeddings and proposing "Approximate Lipschitz Embedding" (ALE) to guarantee equicontinuity and uniform convergence (Trosset et al., 2024).

6. Optimization, Computational Complexity, and Algorithmic Guarantees

• Majorization and Iterative Schemes: The SMACOF algorithm and its variants leverage quadratic majorization of the stress surface, ensuring monotonicity (Boyarski et al., 2017, Rajawat et al., 2016).
• Complexity and Hardness: Global minimization of metric MDS stress is NP-hard, even in one dimension and for bounded-diameter graphs, ruling out efficient exact algorithms (Demaine et al., 2021). On the other hand, polynomial-time approximation schemes (PTAS) are available in special cases (e.g., small-diameter graphs), and quasi-polynomial approximations have been developed for general dissimilarity matrices with controlled aspect ratios (Bakshi et al., 2023).
• Stochastic and Streaming Algorithms: Stochastic SMACOF and related incremental algorithms provide scalable and provably convergent solutions for large, dynamically evolving networks (Rajawat et al., 2016).

7. Applications and Theoretical Impact

Applications span manifold learning, visualization, feature extraction for object recognition, cross-linguistic semantic mapping, shape analysis, and cooperative network localization (Ghojogh et al., 2020, Wang et al., 2013, Klis et al., 2020, Rajawat et al., 2016). MDS forms the foundation for related methods such as Isomap and Spectral Generalized MDS, enables kernel-based, out-of-sample extensions, and admits both supervised and statistically informed extensions (e.g., $\{y_i\}_{i=1}^n \subset \mathbb{R}^p$4-informed MDS for microbiome studies) (Kim et al., 2023).

Theoretical advances include:

• Operator-theoretic analysis and infinite-sample generalizations (Adams et al., 2019, Trosset et al., 2024).
• Consistency under random sampling and empirical-measure convergence (Kassab, 2019).
• Nontrivial error estimates for MDS under noise, regularization, or alternative divergence metrics (e.g., Sinkhorn divergence for shape spaces) (Yachimura et al., 2024, Peterfreund et al., 2018).
• New frameworks for embedding with guaranteed performance and statistical interpretability (Demaine et al., 2021, Kim et al., 2023).

These developments collectively underscore MDS as a central, extensible tool for distance-preserving geometric embedding, equipped with rigorous foundations, practical adaptations, and across-the-board impact in data science and computational geometry.