---
title: Relative uLSIF (RuLSIF)
url: https://www.emergentmind.com/topics/relative-ulsif-rulsif
type: topic
---

# Relative uLSIF (RuLSIF)

Relative uLSIF (RuLSIF) is a non-parametric statistical method for direct estimation of α-relative density ratios and their associated divergences between two probability distributions. RuLSIF generalizes the unconstrained Least-Squares Importance Fitting (uLSIF) method by introducing an α-mixture in the denominator of the density ratio, conferring boundedness and improved numerical stability. It is particularly effective in applications such as change-point detection in time-series settings, where robust, analytic, and sample-efficient divergence estimation is required [1203.0453].

## 1. Formal Definition of α-Relative Density Ratio and Relative Pearson Divergence

Let $p(x)$ and $q(x)$ be two probability densities on $\mathbb{R}^d$. For a prescribed parameter $0 \leq \alpha < 1$, RuLSIF introduces the α-mixture density
$$
p_\alpha(x) = \alpha\,p(x) + (1-\alpha) q(x),
$$
and defines the α-relative density ratio as
$$
r_\alpha(x) = \frac{p(x)}{p_\alpha(x)} = \frac{p(x)}{\alpha\,p(x) + (1-\alpha) q(x)}.
$$

The associated α-relative Pearson (PE) divergence is then given as:
$$
D_\mathrm{PE}^\alpha(p\|q) = \frac{1}{2} \int p_\alpha(x) \left(r_\alpha(x) - 1\right)^2 dx,
$$
quantifying the discrepancy between $p(x)$ and $q(x)$ through their α-relative density ratio. For $\alpha=0$, this reduces to the conventional density-ratio formulation used in uLSIF.

## 2. Direct Density-Ratio Estimation via Squared-Loss with Regularization

RuLSIF models the unknown $r_\alpha(x)$ by a kernel expansion:
$$
\hat{r}(x; \theta) = \sum_{\ell=1}^b \theta_\ell K(x, x_\ell),
$$
where $\{ x_\ell \}_{\ell=1}^b$ are basis points—often sampled from $q$ or both $p$ and $q$—and $K(\cdot, \cdot)$ is a positive-definite kernel, typically Gaussian. The method determines $\theta \in \mathbb{R}^b$ by minimizing the regularized expected squared error:
$$
J(\theta) = \frac{1}{2} \int \left( \hat{r}(x; \theta) - r_\alpha(x) \right)^2 p_\alpha(x) dx + \frac{\lambda}{2} \| \theta \|^2,
$$
with regularization parameter $\lambda > 0$. Expanding this, the minimization can be expressed as:
$$
J(\theta) = \text{const} + \frac{1}{2} \theta^{\mathsf{T}} H_\alpha \theta - h^{\mathsf{T}} \theta + \frac{\lambda}{2} \| \theta \|^2,
$$
where
$$
\begin{aligned}
H_\alpha &= \alpha\, \mathbb{E}_{x \sim p} [K(x)K(x)^{\mathsf{T}}] + (1-\alpha)\, \mathbb{E}_{x \sim q} [K(x)K(x)^{\mathsf{T}}], \\
h &= \mathbb{E}_{x \sim p} [K(x)].
\end{aligned}
$$

Empirical estimation replaces expectations with averages over samples $\{ x_i \}_{i=1}^n \sim p$ and $\{ x_j' \}_{j=1}^m \sim q$:
$$
\begin{aligned}
\hat{H}_\alpha &= \alpha \frac{1}{n} \sum_{i=1}^n K(x_i) K(x_i)^{\mathsf{T}} + (1-\alpha) \frac{1}{m} \sum_{j=1}^m K(x_j') K(x_j')^{\mathsf{T}}, \\
\hat{h} &= \frac{1}{n} \sum_{i=1}^n K(x_i).
\end{aligned}
$$

## 3. Analytic Solution and Computational Considerations

Setting $\nabla_\theta \hat{J}(\theta)=0$, RuLSIF yields the closed-form solution:
$$
\theta^* = (\hat{H}_\alpha + \lambda I_b)^{-1} \hat{h},
$$
where $I_b$ is the $b \times b$ identity matrix. The density-ratio estimate at any $x$ is then
$$
\hat{r}(x) = \sum_{\ell=1}^b \theta^*_\ell K(x, x_\ell).
$$

Dominant computational costs are the formation of the $b \times b$ matrix $\hat{H}_\alpha$ ($O((n+m)b^2)$) and its inversion ($O(b^3)$). The number of basis points $b$ is often set such that $b \ll n + m$ using subsampling or reduced kernel centers. Unlike KLIEP, RuLSIF requires no iterative quadratic programming (QP), resulting in faster computation and greater numerical stability for large-scale problems.

## 4. Algorithmic Implementation Details

Key components in practical deployment are summarized as follows:

| Component           | Common Choices / Procedures | Notes                                              |
|---------------------|----------------------------|----------------------------------------------------|
| Kernel $K(x,y)$     | Gaussian ($K(x,y) = \exp(-\|x-y\|^2/(2\sigma^2))$) | Positive-definite; $\sigma$ typically cross-validated |
| Basis points $\{x_\ell\}$ | All $p$-samples or random subset | Subsampling manages computational cost              |
| $\alpha$            | Small value (e.g., $0.1$)  | Ensures $r_\alpha(x) \leq 1/\alpha$                |
| $\sigma$, $\lambda$ | K-fold cross-validation    | Based on minimizing hold-out $\hat{J}$             |

The computation of $\hat{H}_\alpha$ and $\hat{h}$ is $O((n+m)b^2)$, and solution of the linear system is $O(b^3)$. This approach offers significant advantages in runtime and scalability compared to iterative QP methods.

## 5. Application: Change-Point Detection in Time Series

RuLSIF is applied to change-point detection in multi-dimensional time series $\{ x_t \} \subset \mathbb{R}^d$ as follows:

1. Two consecutive retrospective windows of length $n$ are used:
   - $X = \{ x_{t-n+1}, \ldots, x_t \} \sim p$
   - $X' = \{ x_{t+1}, \ldots, x_{t+n} \} \sim q$
2. RuLSIF estimates $r_{\alpha, p\|q}(x)$ and $r_{\alpha, q\|p}(x)$.
3. The empirical α-relative PE divergences, using the dual form, are
   $$
   \widehat{\mathrm{PE}}_\alpha(p\|q) = -\frac{\alpha}{2n}\sum_{x \in X} \hat{r}(x)^2 - \frac{1-\alpha}{2n}\sum_{x \in X'} \hat{r}(x)^2 + \frac{1}{n}\sum_{x \in X} \hat{r}(x) - \frac{1}{2}
   $$
   with an analogous estimate for $\widehat{\mathrm{PE}}_\alpha(q\|p)$.
4. The symmetric change-point score is $S(t) = \widehat{\mathrm{PE}}_\alpha(p\|q) + \widehat{\mathrm{PE}}_\alpha(q\|p)$.
5. Sliding $t$ yields a sequence of scores; change-points correspond to peaks in $S(t)$.

This approach has been successfully validated on artificial and real-world datasets, including domains of human activity sensing, speech, and Twitter data [1203.0453].

## 6. Theoretical Properties and Empirical Performance

RuLSIF offers the following properties:

- **Boundedness**: $r_\alpha(x) \leq 1/\alpha$ for $\alpha > 0$, providing control over high ratio estimates, unlike $p/q$ which may be unbounded.
- **Convergence Properties**: RuLSIF attains improved non-parametric convergence rates compared to uLSIF (the special case $\alpha=0$); see [Yamada et al., NIPS 2013, referenced in 1203.0453].
- **Numerical Stability**: The analytic solution leads to lower condition numbers and avoids the difficulties of iterative QP solvers.
- **Robustness**: Experimental results demonstrate that RuLSIF yields higher AUCs in change-point detection tasks than both uLSIF- and KLIEP-based approaches on various types of data, indicating increased empirical robustness [1203.0453].

## 7. Summary and Extensions

RuLSIF extends uLSIF by introducing the α-mixture denominator in the density-ratio estimate, which bounds and regularizes the ratio, ensuring stability and improved estimation. It maintains the analytic, non-iterative property of uLSIF, leading to efficient computation. Empirical results on synthetic and real-world data show that RuLSIF enables more accurate and stable non-parametric change-point detection compared to uLSIF and KLIEP. The broader significance of RuLSIF lies in its generality for divergence-based applications where robust and scalable density-ratio estimation is essential [1203.0453].

Source: https://www.emergentmind.com/topics/relative-ulsif-rulsif