---
title: Segmented Confidence Sequences for Anomaly Detection
url: https://www.emergentmind.com/topics/segmented-confidence-sequences-scs
type: topic
---

# Segmented Confidence Sequences for Anomaly Detection

Segmented Confidence Sequences (SCS) is an online, unsupervised framework devised for robust anomaly detection in nonstationary time series. SCS employs statistically principled confidence sequences within locally stationary segments, where the segmentation is data-driven and adapts to evolving regimes. The construction yields locally adaptive thresholds maintaining time-uniform Type I error control, thereby improving reliability over fixed or globally adaptive thresholds in the presence of regime shifts, concept drift, or multi-scale distributional changes [2508.06638].

## 1. Definitions and Mathematical Framework

Let $\{x_1,x_2,\ldots\}$ denote a stream of real-valued anomaly-scores (such as reconstruction errors), indexed by $t=1,2,\ldots$. Each $x_t$ is assumed bounded: $a \leq x_t \leq b$. The timeline is partitioned into $K$ non-overlapping segments with breakpoints $0 = \tau_0 < \tau_1 < \tau_2 < \dots < \tau_K = \infty$, so segment $k$ covers $t\in(\tau_{k-1},\tau_k]$. Within each segment $k$, define the local sample mean $\bar{x}_t^{(k)} = S_t^{(k)}/n_t^{(k)}$ where $n_t^{(k)} = t-\tau_{k-1}$ and $S_t^{(k)} = \sum_{i=\tau_{k-1}+1}^t x_i$.

A confidence sequence (CS) for the (unknown) segment mean $\mu^{(k)}$ is a sequence of intervals $C_t^{(k)} = [L_t^{(k)}, U_t^{(k)}]$ constructed so that
$$
P\bigl(\exists\, t\in(\tau_{k-1},\tau_k]:\,\mu^{(k)}\notin C_t^{(k)}\bigr)\leq \alpha_k,
$$
where $\alpha_k$ is the error allocated to segment $k$, with $\sum_{k=1}^K\alpha_k\leq\alpha$. An anomaly at time $t$ is flagged if $x_t$ lies outside $C_t^{(k)}$ for its current segment and, optionally, also fails a global percentile filter.

## 2. Construction of Confidence Sequences and Segmentation

### 2.1. Confidence Sequence Formulas

For $x_t\in[a,b]$ and error $\alpha_k$ per segment, the nonparametric Hoeffding-style confidence sequence for $\mu^{(k)}$ at time $t$ is
$$
L_t^{(k)} = \bar{x}_t^{(k)} - (b-a)\sqrt{\frac{\ln(2/\alpha_k)}{2n_t^{(k)}}}, \qquad
U_t^{(k)} = \bar{x}_t^{(k)} + (b-a)\sqrt{\frac{\ln(2/\alpha_k)}{2n_t^{(k)}}}.
$$
Alternatively, use empirical standard deviation $\sigma^{(k)}$ and a scaling coefficient $c(\alpha_k)$:
$$
\text{bound\_width}_t^{(k)} = c(\alpha_k)\,\sigma^{(k)},
$$
with $c(\alpha_k)$ chosen by threshold-dependent rules. Then set $L_t^{(k)} = \bar{x}_t^{(k)} - \text{bound\_width}_t^{(k)}$, $U_t^{(k)} = \bar{x}_t^{(k)} + \text{bound\_width}_t^{(k)}$.

### 2.2. Segmentation Algorithms

Two segmentation strategies are provided:
- **APCA (Adaptive Piecewise Constant Approximation):**
  - For a window $[\ell, r]$, candidate splits at $s$ minimize $left\_error + right\_error$.
  - Splits accepted if $\min_{s}(left\_error + right\_error) < no\_split\_error\times \text{improvement threshold}$.
  - Segmentation halts at minimal segment length or if improvement is insufficient. For flat regions (coefficient of variation $< 0.1$), segments default to size $\max(200,\lfloor n/15 \rfloor)$.
- **K-means Clustering on Sliding-Window Features:**
  - Feature extraction: mean, std, median, and skewness per window.
  - Features normalized and clustered into $n_{segments}$ via K-means; failures result in single-segment treatment.

### 2.3. Composite Anomaly Detection Rule

Anomaly at $t$ in segment $k$ is triggered if $x_t < L_t^{(k)}$ or $x_t > U_t^{(k)}$. Optionally, a global percentile filter requires $x_t > \theta_p$ where $\theta_p$ is the $p$-th percentile (e.g., $p=99$) of training residuals.

## 3. Statistical Guarantees

SCS inherits its statistical accuracy from nonparametric confidence-sequence theory [Howard et al., 2021]. For independently sampled $x_t \in [a,b]$ in a segment, the constructed confidence sequence intervals $C_t^{(k)}$ satisfy
$$
P(\exists\, t\in(\tau_{k-1},\tau_k]:\,\mu^{(k)}\notin C_t^{(k)})\leq\alpha_k,
$$
and by a union bound or error allocation,
$$
P(\exists\, t:\text{non-anomalous point flagged}) \leq \sum_{k=1}^K \alpha_k \leq \alpha.
$$
Because segmentation depends only on past data, coverage properties are preserved via optional-stopping arguments, even if segmentation is adaptive.

## 4. Algorithmic Implementation

The SCS workflow, in high-level pseudocode, is as follows:

1. **Offline Segmentation:** Segment data via APCA or K-means, as selected.
2. **Initialization:** For each segment $k$, initialize $n^{(k)}\gets 0$, $S^{(k)}\gets 0$, $\sigma^{(k)}\gets$ empirical std of a training window.
3. **Online Update:** For each $x_t$:
    - Assign segment $k$ by $t\in(\tau_{k-1},\tau_k]$.
    - Update statistics ($n^{(k)}$, $S^{(k)}$, $\bar{x}^{(k)}$).
    - Compute bound width by Hoeffding or empirical std formula.
    - Flag anomaly if $x_t$ outside $[L_t^{(k)}, U_t^{(k)}]$ and, if enabled, $x_t >$ percentile threshold.

**Computational Complexity:**
- Segmentation: APCA is worst-case $O(N^2)$, typically $O(N\log N)$ with pruning; K-means is $O(Ndk)$ per iteration on windowed features.
- Online update: $O(1)$ per data point.
- Memory: Requires storing segment boundaries and per-segment aggregates.

## 5. Parameterization and Operational Considerations

- **Confidence Level ($\alpha$):** Common choices are 0.05 or 0.01. Lower $\alpha$ yields wider bounds, fewer false alarms, and lower sensitivity.
- **APCA Improvement Thresholds:** Typical values are 0.7 for high variance, 0.5 for moderate variance. Minimum segment length of $200$ or $\lfloor N/15 \rfloor$.
- **Boundedness:** Enforce or approximate $x_t \in [a,b]$ by truncation or winsorization.
- **Empirical $\sigma^{(k)}$ Updates:** Can employ Welford’s algorithm for online updates.
- **Global Percentile Filter:** Values like $p=99$ provide robustness against local fluctuations; can be disabled to maximize recall at the expense of precision.
- **Assumptions:** Within each segment, scores are approximately stationary and independent (or weakly dependent).

## 6. Empirical Benchmarks and Comparative Results

Evaluation was performed on 151 inline semiconductor sensor traces with approximately 10% defective wafers. The baseline used a fixed 99th-percentile residual threshold. Key SCS results are presented below (Δ relative to baseline):

| Method                | ΔPrecision | ΔRecall | ΔF1-Score |
|-----------------------|------------|---------|-----------|
| SCS APCA ($\alpha=0.99$)   | –0.3282    | +3.9952 | +1.9074   |
| SCS KMEANS ($\alpha=0.99$) | –0.3999    | +1.6643 | +0.9262   |
| SCS APCA ($\alpha=0.95$)   | –0.4290    | +6.1595 | +2.1289   |
| SCS KMEANS ($\alpha=0.95$) | –0.4656    | +3.3286 | +1.4148   |

Anomaly counts at $\alpha=0.99$:

| Method                | TP | TN   | FP  | FN   |
|-----------------------|----|------|-----|------|
| Baseline (99th pct)   |  6 | 1608 | 12  | 137  |
| SCS APCA ($\alpha=0.99$)   | 30 | 1516 | 104  | 113  |
| SCS KMEANS ($\alpha=0.99$) | 16 | 1556 | 64   | 127  |

Key findings include a fivefold increase in true positives with SCS APCA ($30$ vs $6$), raising recall from ~4% to ~30%, and an F1-score roughly doubling relative to the baseline for $\alpha=0.99$ and more than doubling for $\alpha=0.95$. The percentile filter improves precision at the expense of recall, while K-means segmentation produces slightly less aggressive segmentation than APCA and avoids short segments in smooth series [2508.06638].

## 7. Context, Related Work, and Applications

SCS provides statistically rigorous local adaptation for anomaly detection in nonstationary time series where global or fixed-threshold approaches are rendered inadequate by distributional drift or regime changes. The framework is unsupervised, suitable for settings with scarce labeled anomalies, and is designed for applications such as manufacturing process control, IT infrastructure monitoring, and sensor data streams.

SCS builds conceptually on the theory of confidence sequences for time-uniform, nonparametric inference [Howard et al., 2021], online segmentation methods such as APCA [Keogh et al., 2001], and builds upon work in sequential quantile estimation under concept drift [Wang et al., 2023]. Its guarantee of explicit, interpretable false alarm rates and empirically validated reliability makes it appropriate for high-stakes or automated monitoring scenarios [2508.06638].

Source: https://www.emergentmind.com/topics/segmented-confidence-sequences-scs