Embed-KCPD: Unsupervised Text Segmentation

• Embed-KCPD is a training-free, nonparametric method that applies kernel change-point detection on sentence embeddings for unsupervised text segmentation.
• It leverages a penalized optimal partitioning framework coupled with dynamic programming and the PELT algorithm for scalable segmentation.
• The method offers theoretical guarantees under short-range dependency, ensuring consistent boundary detection with proven localization accuracy.

Embed-KCPD is a training-free, nonparametric method for unsupervised text segmentation that leverages kernel change-point detection (KCPD) on sentence embeddings. The approach is grounded in a penalized optimal partitioning framework with theoretical guarantees under short-range ($m$-dependent) structure, tailored for applications where text boundaries are subjective and expensive to label. The method combines modern pretrained embeddings with a kernel-based dispersion cost and a scalable dynamic programming solver.

1. Problem Formulation and Mathematical Objective

Embed-KCPD operates on a sequence of $T$ atomic text units, $X_1,\dots,X_T$ (sentences, paragraphs, or dialogue turns). Each unit is transformed into a fixed embedding $Y_t=f(X_t)\in\mathbb{R}^d$ via a pretrained encoder (e.g., sBERT, MPNet, RoBERTa, OpenAI text-embedding-3-small). The segmentation task assumes there exist true boundaries $0=\tau_0<\tau_1<\cdots<\tau_K<\tau_{K+1}=T$ such that distributions of $Y_t$ are stationary within each block $\{\tau_{k-1}+1,\ldots,\tau_k\}$ and change at boundaries.

The penalized kernel cost for any candidate partition $\boldsymbol\tau'=(\tau'_0,\ldots,\tau'_{K'+1})$ is

$$L(\boldsymbol\tau') = \sum_{k=1}^{K'+1} \widehat{C}(\tau'_{k-1}+1,\tau'_k) + \beta_T K'$$

where for a segment $[s,e]$,

$$\widehat{C}(s,e) = \sum_{t=s}^e k(Y_t, Y_t) - \frac{1}{e-s+1} \sum_{i=s}^e\sum_{j=s}^e k(Y_i, Y_j)$$

with $k$ a positive-definite kernel (e.g., RBF, cosine). The optimal segmentation is %%%%10%%%%.

2. Optimization Strategy and Implementation

The minimization of $L(\boldsymbol\tau')$ can be performed exactly with dynamic programming in $O(T^2)$ time. In practice, Embed-KCPD utilizes the Pruned Exact Linear Time (PELT) algorithm, which leverages segment cost additivity and candidate pruning to achieve near-linear expected complexity under typical conditions. The penalty parameter $\beta_T$ controlling the number of segments is set as $\beta_T = C\sqrt{T\log T}$, with $C$ chosen via unsupervised heuristics, notably the “elbow” of change-point count versus $C$.

Pseudocode for Embed-KCPD is as follows:

Input: Text units X[1..T], sentence encoder f, kernel k, penalty C Output: Change-point indices τ_hat 1. for t=1..T do 2. Y[t] ← f(X[t]) 3. end for 4. Compute penalty β_T = C * sqrt(T log T) 5. (Optional) Precompute Gram matrix K[i,j] = k(Y[i],Y[j]) or prefix sums 6. Initialize cost F[0]=−β_T, R={0}, prev[0]=0 7. for t=1..T do 8. # PELT pruning 9. F[t] = ∞ 10. for s in R do 11. cost = F[s] + Cseg(s+1,t) + β_T 12. if cost < F[t] then 13. F[t] = cost; prev[t]=s 14. end if 15. end for 16. R = R ∪ {t} 17. Prune any s in R with F[s]+Cseg(s+1,t)+β_T > F[t] 18. end for 19. # backtrack 20. τ_hat = ∅; t=T 21. while t > 0 do 22. s = prev[t]; τ_hat = {s} ∪ τ_hat; t = s 23. end while 24. return τ_hat

Here, $Cseg(a,b)$ computes $\widehat{C}(a,b)$ using cumulative sums or, for cosine kernels, $O(d)$ vector operations.

3. Statistical Assumptions and Theoretical Guarantees

Embed-KCPD analysis is predicated on several stylized assumptions:

• $m$-dependence: $\{Y_t\}$ is $m$-dependent, i.e., $Y_t \perp Y_{t'}$ for %%%%24%%%%, with strict stationarity within segments.
• Characteristic kernel property: the kernel $k$ is positive-definite, bounded ($[0,M]$), and characteristic (so MMD is a metric).
• Detectability and margin: Minimum squared RKHS mean shift $\Delta_*^2 > 0$ between adjacent segment distributions.
• Segment separation and penalty: Minimum segment length $\ell_T \gg \sqrt{T\log T}$ and penalty $\beta_T \geq 16M\sqrt{2(8m+5)T\log T}+2M(1+6m)$.

Key results include:

• Consistency: Under the prior, probability that $\widehat{K}=K$ tends to 1 as $T\to\infty$.
• Localization: All true boundaries are recovered within $O(\sqrt{T\log T})$ of the ground truth, relative to segment size.

Formally,

$$\Pr\left( \max_{1\leq k\leq K} \min_{1\leq j\leq \widehat{K}} |\widehat{\tau}_j - \tau_k| / \ell_T \leq \varepsilon \right) \to 1$$

for any $\varepsilon$ as $T\to\infty$.

4. Embedding and Kernel Choices

Embed-KCPD requires sentence-level embedding vectors, with recommended encoders including:

Encoder	Normalization	Kernel Type
sBERT, MPNet, RoBERTa	$\ell_2$-norm	RBF, Cosine
text-embedding-3-small	$\ell_2$-norm	RBF, Cosine

RBF kernel (characteristic): $$k(y, y') = \exp\left(-\frac{||y - y'||^2}{2\sigma^2}\right)$$ Cosine kernel (not characteristic): $k(y, y') = y^\top y', \quad \|y\|_2 = \|y'\|_2 = 1$

Best practices use the median-heuristic for $\sigma$ in RBF, and $\ell_2$ normalization prior to cosine similarity computation.

5. Empirical Evaluation and Benchmarks

Embed-KCPD has been empirically validated on synthetic and natural datasets:

• Synthetic: Choi’s benchmarks (varying segment counts), with simulated $m$-dependent sequences via LLMs (GPT-4.1).
• Natural: Wiki-300, Wiki-50, Elements, and arXiv-abstract concatenations.
• Case Study: Segmentation of Taylor Swift’s tweets into semantically coherent phases corresponding to real-world events.

Performance is quantified via $P_k$ (“false‐same vs. false‐diff” error) and WindowDiff, with lower values indicating better segmentation. Representative results (Choi dataset):

Method	$P_k$\% ↓	WD ↓
Embed-KCPD (cosine kernel)	5.2	5.2
Coherence (2024)	4.0	4.4
GraphSeg (2016)	7.2	9.0
TextTiling	46	–

On Wikipedia (Wiki-300), Embed-KCPD matches or outperforms supervised baselines such as NTS.

6. Simulation Framework for Dependence Validation

An LLM-based generation procedure enables control of the $m$-dependence in synthetic documents for theory–practice validation. Each sentence is generated conditioning on the previous $m$ sentences and a topic prompt, producing finite-memory Markov text. Pieces from multiple topics are concatenated to produce documents with known true boundaries, facilitating empirical verification that segmentation errors decrease with $O(\sqrt{T\log T})$ scaling as $T$ increases.

7. Practical Recommendations, Limitations, and Open Challenges

Embed-KCPD is most effective in segmentation domains with moderate to long segments. Cosine kernel is preferred when lexical overlap signals boundaries; RBF kernel is robust to semantic shifts across heterogeneous documents. The penalty calibration via elbow heuristics works well empirically, but deriving fully data-driven penalty selection remains open.

The $m$-dependence model is stylized; natural language data may manifest longer-range structure, potentially affecting separation guarantees. Embed-KCPD is currently offline; extension to streaming or online KCPD with comparable theoretical robustness is a topic for future research.

---

References: (Jia et al., 26 Jan 2026, Diaz-Rodriguez et al., 3 Oct 2025)