---
title: Multidimensional Scaling (MDS)
url: https://www.emergentmind.com/topics/multidimensional-scaling-mds
type: topic
---

# Multidimensional Scaling (MDS)

Multidimensional Scaling (MDS) is a foundational family of techniques in geometry-aware data analysis and manifold learning. It seeks a configuration of points in a geometric space such that the pairwise distances among these points reflect a given matrix of input dissimilarities or distances. Applications encompass dimensionality reduction, visualization, feature learning, computational linguistics, shape matching, and network analysis. MDS encompasses classical (eigendecomposition-based) algorithms, iterative stress-minimization procedures, nonlinear and manifold variants such as Isomap, as well as scalability adaptations for large data and extensions to multiple, heterogeneous dissimilarity sources.

## 1. Mathematical Foundations and Classical MDS

Let $\{x_i\}_{i=1}^{n}$ denote a finite collection of objects, each typically represented as a point in $\mathbb{R}^d$ (but not necessarily with explicit coordinates), and let $D = (d_{ij})$ be an $n \times n$ matrix of pairwise dissimilarities $d_{ij}$. The central goal is to find $\{y_i\}_{i=1}^n \subset \mathbb{R}^p$ for $p \ll n$ such that their Euclidean distances $\|y_i - y_j\|$ closely approximate the given dissimilarities.

**Classical MDS** is based on the following sequence ([2009.08136], [1907.01379]):
- Compute squared distances $D^{(2)}=(d_{ij}^2)$.
- Center with $H = I_n - (1/n) \mathbf{1}_n \mathbf{1}_n^T$.
- Form the double-centered Gram matrix: $B = -\tfrac12 H D^{(2)} H$.
- Compute the eigendecomposition $B = V \Lambda V^T$.
- Select the top $p$ eigenpairs $(v_k, \lambda_k > 0)$ to form the embedding: The $i$th point $y_i$ has coordinates $(\sqrt{\lambda_1} v_1(i), ..., \sqrt{\lambda_p} v_p(i))^T$.

This configuration uniquely minimizes the so-called _strain_ (Frobenius norm between empirical and reconstructed Gram matrices), achieving exact recovery when the input matrix $D$ is Euclidean ([1907.01379]).

## 2. Variants: Metric, Non-metric, Sammon, and Isomap

- **Metric MDS**: Seeks $Y = \{y_i\}$ minimizing 
  $$
  \mathrm{Stress}(Y) = \left( \sum_{i<j} w_{ij} ( \|y_i - y_j\| - d_{ij} )^2 \right)^{1/2}
  $$
  for fixed positive weights $w_{ij}$ (often $1$). The objective is nonconvex and typically tackled by gradient-based or majorization algorithms such as SMACOF ([2009.08136], [1709.03484]).

- **Non-metric MDS**: Preserves only the rank order of dissimilarities. Introduces a monotonic, typically isotonic-transform $f$ and minimizes
  $$
  \sum_{i<j} ( d_{ij} - f(\|y_i - y_j\|) )^2
  $$
  alternating between updating $Y$ and fitting $f$ ([2009.08136]).

- **Sammon Mapping**: A weighted form where
  $$
  w_{ij} = 1/d_{ij}, \quad \mathrm{Cost}(Y) = \frac{1}{a} \sum_{i<j} \frac{ ( \|y_i - y_j\| - d_{ij} )^2 }{ d_{ij} }
  $$
  with normalization $a = \sum_{i<j} d_{ij}$, emphasizing fidelity for small distances ([2009.08136]).

- **Isomap**: Replaces $d_{ij}$ by an estimated geodesic distance, using shortest paths on a neighborhood graph. After transforming these distances to a centered Gram matrix, Isomap applies the classical spectral embedding, capturing nonlinear manifold structures ([2009.08136]).

## 3. Computation and Scalability for Large Data

For $n \gg 10^4$, classical algorithms become impractical due to cubic complexity. Partition-based approaches decompose the global problem:
- **Landmark MDS**: Selects $\ell \ll n$ "landmarks", runs classical MDS on them, and triangulates new points via analytic formulas (Gower’s interpolation) ([2007.11919]).
- **Interpolation MDS, Reduced MDS, Pivot MDS, Divide-and-Conquer MDS, Fast MDS**: Variants that partition, align, and stitch together local MDS results; see Table 1.

| Algorithm      | Complexity (Time)  | Memory      | Comment                 |
|----------------|-------------------|-------------|-------------------------|
| Landmark MDS   | $O(n\ell)$        | $O(\ell^2)$ | Efficient, low-memory   |
| Interp MDS     | $O(n\ell)$        | $O(\ell^2)$ | Fastest in benchmark    |
| Pivot MDS      | $O(n\ell)$        | $O(n\ell)$  | Direct inner-product    |
| Div\&Conq MDS  | $O(n\ell^2)$      | $O(\ell^2)$ | More robust, slightly biased |
| Fast MDS       | $O(n\log n)$      | $O(\ell^2)$ | Recursive, higher const |

Empirical evaluations on million-point EMNIST images demonstrated that all these methods closely match the quality of classical MDS but differ in time/memory footprint ([2007.11919]).

## 4. Extensions: Multiple Views, Non-Euclidean Embeddings, and Feature Learning

- **Multi-view MDS (MVMDS)**: For $M$ heterogeneous distance matrices (views) on the same objects, minimizes a weighted sum of stresses with automatic weight learning:
  $$
  J(X, \alpha) = \sum_{v=1}^M \alpha^{(v)}{}^\gamma \sum_{i<j} w_{ij} ( d_{ij}^{(v)} - \|x_i - x_j\| )^2
  $$
  with controller parameter $\gamma$. Alternating minimization yields both the embedding $X$ and optimal view-weights $\alpha$ ([1605.00286]).

- **Feature Learning via MDS**: MDS can serve as a feature learning framework when applied to high-level, semantics-sensitive distances (e.g., spatial pyramid matching) between objects rather than raw pixel or Euclidean distances ([1306.3294]). The process learns vector codes so that Euclidean distances between codes reflect semantic similarity.

- **Non-Euclidean Targets**: MDS can also be adapted to hyperbolic or spherical target spaces, with corresponding definitions of distance and gradient, enabling more faithful embeddings for inherently hierarchical or non-flat data ([1105.5332], [2210.15081]).

## 5. Infinite, Continuous, and Operator-Theoretic Generalizations

MDS extends to infinite metric measure spaces $(X, d, \mu)$ by defining the "double-centered" kernel:
$$
K_B(x,s) = -\tfrac12 d(x,s)^2 - \int K_A(w,s) d\mu(w) - \int K_A(x,z) d\mu(z) + \iint K_A(w,z) d\mu(w) d\mu(z)
$$
and embedding via the spectrum of the associated Hilbert-Schmidt operator $T_{K_B}$ on $L^2(X,\mu)$. The canonical embedding $f(x) = (\sqrt{\hat\lambda_1} \phi_1(x), \ldots)$ (with $\hat\lambda_i = \max\{\lambda_i,0\}$) optimally minimizes infinite-sample "strain" ([1907.01379], [1904.07763]).

Continuous MDS frameworks permit the analysis of limit behavior as $n \to \infty$, establishing $L^p$-consistency of embeddings and proposing "Approximate Lipschitz Embedding" (ALE) to guarantee equicontinuity and uniform convergence ([2402.04436]).

## 6. Optimization, Computational Complexity, and Algorithmic Guarantees

- **Majorization and Iterative Schemes**: The SMACOF algorithm and its variants leverage quadratic majorization of the stress surface, ensuring monotonicity ([1709.03484], [1612.07089]).
- **Complexity and Hardness**: Global minimization of metric MDS stress is NP-hard, even in one dimension and for bounded-diameter graphs, ruling out efficient exact algorithms ([2109.11505]). On the other hand, polynomial-time approximation schemes (PTAS) are available in special cases (e.g., small-diameter graphs), and quasi-polynomial approximations have been developed for general dissimilarity matrices with controlled aspect ratios ([2311.17840]).
- **Stochastic and Streaming Algorithms**: Stochastic SMACOF and related incremental algorithms provide scalable and provably convergent solutions for large, dynamically evolving networks ([1612.07089]).

## 7. Applications and Theoretical Impact

Applications span manifold learning, visualization, feature extraction for object recognition, cross-linguistic semantic mapping, shape analysis, and cooperative network localization ([2009.08136], [1306.3294], [2012.04946], [1612.07089]). MDS forms the foundation for related methods such as Isomap and Spectral Generalized MDS, enables kernel-based, out-of-sample extensions, and admits both supervised and statistically informed extensions (e.g., $F$-informed MDS for microbiome studies) ([2308.00354]).

Theoretical advances include:
- Operator-theoretic analysis and infinite-sample generalizations ([1907.01379], [2402.04436]).
- Consistency under random sampling and empirical-measure convergence ([1904.07763]).
- Nontrivial error estimates for MDS under noise, regularization, or alternative divergence metrics (e.g., Sinkhorn divergence for shape spaces) ([2409.14687], [1801.10229]).
- New frameworks for embedding with guaranteed performance and statistical interpretability ([2109.11505], [2308.00354]).

These developments collectively underscore MDS as a central, extensible tool for distance-preserving geometric embedding, equipped with rigorous foundations, practical adaptations, and across-the-board impact in data science and computational geometry.

Source: https://www.emergentmind.com/topics/multidimensional-scaling-mds