Patch Features Compression Module (PFCM)

• PFCM is a generic module that reduces complexity by aggregating, clustering, or quantizing dense patch embeddings into higher-level semantic representations.
• It employs techniques like Density-Peaks Clustering, attention aggregation, and K-means quantization to merge redundant information while retaining critical features.
• PFCMs enable scalable applications in text-video retrieval, multi-vector document search, and time-series forecasting, balancing computational efficiency with accuracy.

The Patch Features Compression Module (PFCM) is a generic architectural component used in modern vision, vision-language, and multivariate sequence models to reduce computational and storage complexity by condensing a large set of patch-level embeddings into a compact representation. Designed for high-dimensional and multi-modal inputs (e.g., image patches, video frames, or sensor time-series segments), PFCMs aggregate, cluster, and/or quantize raw patch features into higher-level semantic entities, salient patch sets, or vector codes. These compressed representations enable efficient downstream processing, alignment with textual or sequential modalities, and scalable retrieval. PFCMs have been implemented in diverse tasks such as text-video retrieval, multi-vector document search, and multivariate time-series forecasting (Xie et al., 22 Jan 2026, Bach, 19 Jun 2025, Qin et al., 6 Jan 2025).

1. Conceptual Motivation and Rationale

The primary challenge addressed by PFCMs is the inefficiency or redundancy inherent in treating every patch as an independent entity during subsequent model stages. In computer vision and vision-LLMs, the spatial granularity of patch embeddings—output by modules such as CLIP or Vision Transformers—enables fine local processing but incurs heavy costs in both storage and computation. Moreover, many patches correspond to background or irrelevant content.

PFCMs are motivated by principles observed both in biological vision and large-scale statistical learning frameworks:

• Human micro-perception: Analogous to the selective focus of human vision, where attention is allocated only to semantically meaningful subregions once a coarse focus is set. For instance, in video retrieval, PFCM serves as the spatial analog of a temporal frame selection module, focusing attention on object-level units within a frame once high-level temporal redundancy is removed (Xie et al., 22 Jan 2026).
• Late-interaction systems' cost: Multi-vector retrieval architectures that score all patch-to-patch or segment-to-segment interactions see costs scale quadratically with patch count. Compressing patches via PFCM can significantly reduce the interaction complexity, storage, and latency without degrading task-specific accuracy (Bach, 19 Jun 2025, Qin et al., 6 Jan 2025).
• Causal and cross-modal structure: In time series, signals are recorded at high spatiotemporal frequencies. Compressing these into sensor-level summaries preserves dependencies while enabling tractable modeling of inter-variable and temporal relationships (Qin et al., 6 Jan 2025).

2. Core Methodologies and Algorithms

PFCM design is context-dependent. Three primary instantiations, each optimized for its application domain, have emerged in recent literature.

(a) Clustering + Attention Aggregation

In text-video retrieval (HVD), the PFCM aggregates patch embeddings into salient visual entities via Density-Peaks Clustering (DPC-KNN) followed by self-attention (Xie et al., 22 Jan 2026):

• DPC-KNN: Local patch densities $\rho_i$ and distance indicators $\delta_i$ are computed for each patch $p_i$. Cluster centers are selected as patches maximizing $\rho_i \cdot \delta_i$.
• Cluster-Attention Fusion: Each cluster center projects to a query; attention is calculated against all patches within the group, and the resulting fused "entity" embedding replaces the constituent patches.
• Iterative Compression: This process repeats $R$ times, recursively compressing the patch set (e.g., $150 \to 75 \to 38 \to 19$ patches per frame in three rounds).

(b) Quantization and Pruning

In multi-vector document retrieval (ColPali/HPC-ColPali), the PFCM applies:

• K-Means Quantization: Patch embeddings $x_i \in \mathbb{R}^D$ are assigned to a nearest centroid in a codebook $\{c_0, \dots, c_{K-1}\}$, reducing representation from $4D$ bytes per patch to $1$ byte (e.g., $128$-d features compressed $32\times$).
• Attention-Guided Pruning: Only the top-$p\%$ most salient patches, as determined by attention weights $a_i$, are retained; others are pruned dynamically at query time—accelerating sparse late-interaction.
• Optional Binary Encoding: Centroid indices are encoded as $b$-bit strings for Hamming distance-based search, enabling sublinear CPU performance (Bach, 19 Jun 2025).

(c) Global Compression via Attention

In multivariate time-series forecasting (Sensorformer), the PFCM compresses a $D \times N$ patch tensor (D variables, N patches) into $D$ "sensor" vectors:

• The last patch of each variable forms the query $Q$.
• All patches are flattened to keys/values.
• Multi-head attention aggregates the global patch sequence into one vector per variable, followed by sequential LayerNorm and MLP blocks.
• This compressed representation effectively summarizes temporal dynamics in a computationally efficient form (Qin et al., 6 Jan 2025).

3. Mathematical Formulations

Example: PFCM via DPC-KNN Clustering and Attention (Xie et al., 22 Jan 2026)

Let $X \in \mathbb{R}^{N \times D}$ be patch embeddings. At each round:

1. Cluster Center Selection:

$$\rho_i = \exp\left( - \frac{1}{k} \sum_{p_j \in \text{KNN}(p_i)} \|p_i - p_j\|_2 \right)$$

$$\delta_i = \begin{cases} \min_{j: \rho_j > \rho_i} \|p_i - p_j\|_2 & \text{if } \rho_i < \max_j \rho_j \ \max_j \|p_i - p_j\|_2 & \text{otherwise} \end{cases}$$

Select top-$M$ centers by largest $\rho_i \cdot \delta_i$.

2. Attention Aggregation:

$Q = X_c W_Q, \quad K = X W_K, \quad V = X W_V$

$$A = \text{softmax}(Q K^\top/\sqrt{d}), \quad Y = A V$$

$E = \text{LayerNorm}(Y + X_c W_{\text{res}})$

3. Iterative Compression: Replace $X$ with $E$ and repeat for $R$ rounds.

Example: K-Means Quantization (Bach, 19 Jun 2025)

Patch embedding $x_i$ is assigned index $q_i = \arg\min_k \|x_i - c_k\|^2$, stored as $1$ byte. Optionally, $q_i$ is encoded in $b = \lceil \log_2 K \rceil$ bits for binary Hamming search.

Example: Multi-Head Attention-Based Compression (Qin et al., 6 Jan 2025)

Let $E \in \mathbb{R}^{D \times N \times d}$ be the patched time-series embedding.

$Q = E_{:, N-1, :} \in \mathbb{R}^{D \times d}$

$K = V = \text{reshape}(E, [D N, d])$

$$\text{Attn}(Q, K, V) = \text{Softmax}\left(\frac{Q K^\top}{\sqrt{d}}\right) V$$

Apply two rounds of residual + LayerNorm + MLP to obtain $D$ compressed sensor vectors.

4. Practical Implementation Details

Key implementation aspects vary by domain and architecture, as illustrated in the following table:

System	Compression Method	Typical Hyperparameters	Notes
HVD (Xie et al., 22 Jan 2026)	DPC-KNN + Attention	$k=10$, $M=\lfloor\lambda N_p\rfloor$, $\lambda=0.5$, $R=3$	50% reduction per round, LayerNorm after residual, dropout $p=0.1$
HPC-ColPali (Bach, 19 Jun 2025)	K-Means + Pruning + Binary	$K=256$/$512$, $p=40$–$60$\%	Offline clustering, dynamic pruning, optional binary mode
Sensorformer (Qin et al., 6 Jan 2025)	Global Attention	$d_{\text{model}}=256$, multi-head	Attention Q: last patch, K/V: all patches, 2x LayerNorm+MLP

Further details:

• Layer normalization and residual connections are consistently used to stabilize the compressed representations.
• Dropout (typically $p=0.1$) is applied to regularize attention scores.
• PFCM can be iteratively composed to progressively reduce the patch set.
• In HPC-ColPali, only compact or binary codes are retained in memory or storage for scalable indexing and retrieval.

5. Empirical Impact and Trade-Offs

PFCMs yield substantial improvements in computational efficiency, scalability, and in some cases, retrieval or predictive accuracy. Key empirical findings include:

• Text-Video Retrieval (HVD): PFCM alone yields $+1.6$ R@1 absolute improvement over a no-compression baseline; combined with coarse temporal frame selection, gains total $+4.1$ (R@1 $44.7 \to 48.8$) (Xie et al., 22 Jan 2026).
• Multi-Vector Retrieval (HPC-ColPali): Storage reduced up to $32\times$ (float32 to $1$ byte/code); pruning $60$\% of patches yields query latency halved (e.g., $120\text{ms} \to 60\text{ms}$). With $K=256$, $p=60\%$, nDCG@10 drops by only $1.2\%$ relative to full-precision, demonstrating minimal loss (Bach, 19 Jun 2025).
• Time-Series Forecasting (Sensorformer): PFCM reduces the computational complexity of self-attention from $O(D^2 N^2 d)$ to $O(D^2 N d)$. Empirical results indicate a $30$–$40\%$ reduction in training time and up to $50\%$ lower peak memory usage; removing PFCM modestly degrades accuracy, while omitting the second stage (which leverages the compressed representation) degrades it catastrophically (Qin et al., 6 Jan 2025).
• Compression Ratios: Aggressive compression (low $\lambda$ or $p$) risks loss of critical detail (e.g., entity boundaries or time-series anomalies), while insufficient compression limits computational or storage gains.

6. Applications and Cross-Domain Variants

PFCMs are deployed in a variety of domains:

• Text-Video and Text-Image Retrieval: PFCMs bridge the gap between low-level visual patches (from CLIP or ViT) and high-level semantic alignment with textual tokens, enforcing coarse-to-fine focus at both temporal (frame) and spatial (patch/entity) granularities (Xie et al., 22 Jan 2026).
• Large-Scale Multi-Vector Document Search: For retrieval augmented generation, legal summarization, and multimodal search, PFCMs facilitate indexing and late-interaction via lightweight codes and selective patch scoring, integrating quantization and pruning for trade-off tuning (Bach, 19 Jun 2025).
• High-Dimensional Time-Series Analysis: Compression is essential for tractable attention over long, multivariate time-series, allowing the model to focus on variable-level summaries and capture inter-variable dependencies with feasible memory and time budgets (Qin et al., 6 Jan 2025).

A plausible implication is that PFCMs, through principled aggregation, can serve as a generic architectural component for any modality exhibiting excessive local redundancy or for tasks demanding late-interaction scalability.

7. Limitations, Tuning, and Future Directions

Although PFCMs achieve strong empirical and efficiency gains, several caveats and tuning considerations have been reported:

• Compression–Accuracy Trade-Off: Over-compression (e.g., too few centroids or excessive patch pruning) results in the loss of fine-grained entity information and degrades task performance, while under-compression limits efficiency benefits.
• Data- and Task-Dependence: Optimal hyperparameters ($K$, $p$, $\lambda$, $R$) are highly dataset- and application-specific, necessitating empirical calibration.
• Extension to Other Modalities: Current designs are closely coupled to spatial (image/video), multivariate, or multi-modal embeddings, but may generalize to audio or event streams via analogous patch/segmentization.

Continued research investigates dynamic, content-adaptive PFCMs, integration with hierarchical (multi-stage) pruning, and hybrid quantization-attention architectures for increasingly demanding real-world applications (Xie et al., 22 Jan 2026, Bach, 19 Jun 2025, Qin et al., 6 Jan 2025).