Cross-Variate Patch Embedding (CVPE)

• CVPE is a design paradigm that constructs joint embedding spaces by mapping localized patches across different modalities, enabling fine-grained cross-domain analysis.
• It employs modality-adaptive patch extraction and contrastive learning techniques, as demonstrated in Patch2CAD for 2D–3D retrieval and Time-LLM for multivariate forecasting.
• By integrating lightweight cross-variate modules at the embedding stage, CVPE delivers improved model robustness and efficiency in scenarios with partial or weak inter-variable correlations.

Cross-Variate Patch Embedding (CVPE) is a design paradigm for learning localized, cross-domain or cross-variable feature representations by operating at the patch (local region) level. CVPE targets the construction of joint embedding spaces that bridge modalities or variate sets, such as 2D image patches and 3D CAD geometry, or multiple time series channels. The approach emphasizes explicit local correspondence and flexible context integration, providing enhanced robustness in scenarios with partial information or variable correlations. Two distinct realizations—Patch2CAD for 2D–3D retrieval and Time-LLM for multivariate forecasting—illustrate CVPE’s core principles, architectures, and empirical impact in recent literature (Kuo et al., 2021, Shin et al., 19 May 2025).

1. Core Definition, Motivation, and Scope

Cross-Variate Patch Embedding is defined as a process for mapping sets of small, spatially or temporally localized patches—drawn from different variates (modalities, channels, or domains)—into a shared embedding space where semantically or statistically similar patches are close, and nonsimilar ones distant. The foundational goal is to enable fine-grained part-level or cross-channel reasoning without enforcing strict global alignment or dependence.

In the 2D–3D context (Kuo et al., 2021), CVPE aligns patches of object-centric RGB images with correspondingly sized surface patches of 3D CAD models, robustly bridging the photometric and geometric domains. In time series forecasting (Shin et al., 19 May 2025), CVPE injects cross-variate dependencies at the patch-embedding stage into otherwise channel-independent transformer architectures, thus balancing model efficiency with inter-series sensitivity.

The motivation for CVPE is twofold: (1) real-world data frequently lacks exact global matches (e.g., no CAD model exactly matching an observed object, or multimodal time series where inter-series dynamics crucially impact prediction), and (2) localized correspondences are often more robust to occlusion, truncation, or missing data, and better capture structural or inter-variable relationships.

2. Patch Extraction, Correspondence, and Embedding Construction

The patch extraction process in CVPE is modality-adaptive but unified in principle: each signal (image, CAD render, or univariate time series) is partitioned into fixed-size patches, which are then independently embedded.

• In Patch2CAD (Kuo et al., 2021), 2D patches are randomly sampled from object-mask-cropped RGB images with a standard size set to a third of the object bounding box. Corresponding 3D patches are sampled from rendered normal maps of CAD models across 16 viewpoints (identified via K-medoid clustering). Patch correspondences are established by computing the intersection-over-union (IoU) of self-similarity histograms over intra-patch surface normal differences, yielding soft positive and negative sets for contrastive learning.
• In time series CVPE (Shin et al., 19 May 2025), each channel is normalized and divided into partially overlapping or disjoint patches of length $L_P$, projected linearly into feature space. All channel-wise patch embeddings are stacked into a unified tensor, setting the stage for cross-channel context augmentation.

This process ensures that local variations, whether geometrically or temporally defined, are encoded at the patch level before cross-variability is introduced. Local patch-level focus is essential for robustness, especially in scenes with partial information, occlusions, or only partial overlap in the available variate sets.

3. Embedding Network Architectures and Cross-Variate Modules

In both exemplar settings, CVPE functions as a localized embedding followed by a lightweight, cross-variate context mechanism.

• Patch2CAD (Kuo et al., 2021):
  • Separate encoders $f_I$ and $f_C$ (parameter-untied, but structurally identical ResNet-18 FPNs) process 2D image and 3D CAD patches, respectively. After three 3×3 convolutional layers, global average pooling, and L$_2$ normalization, each patch resides as a 512-dimensional feature on the unit hypersphere.
  • Similarity in the joint embedding space employs a cosine metric, with temperature scaling to calibrate contrastive learning.
• Time-LLM with CVPE (Shin et al., 19 May 2025):
  • CVPE integrates two components post-linear-projection:
  • 1. Learnable Positional Encoding: Position table $W_P \in \mathbb{R}^{P \times d_m}$ adds spatial context shared across all channels.
  • 2. Router-Attention Block: A two-stage multi-head attention with $C$ learnable routers per channel ($R \in \mathbb{R}^{N \times C \times d_m}$). Stage one aggregates channel context into routers; stage two redistributes router-encoded context back to each channel-patch. No additional projections are required ($W^Q = W^K = W^V = I$); output passes through MLP and layer normalization for the final embedding.
  • The router-attention mechanism injects inter-variate awareness into patch embeddings while preserving channel independence in later model stages.

Both architectures are parameter-efficient and designed for linear time complexity with respect to the number of channels and patches. All cross-variate information is isolated to the embedding step, controlling overhead and avoiding model-wide parameter coupling.

4. Loss Functions, Optimization, and Theoretical Properties

• Patch2CAD (Kuo et al., 2021):

  • Employs a multi-positive InfoNCE contrastive loss:

    $$\mathcal{L}_e = -\sum_{a \in A} \log \left\{ D_p(a) / [ D_p(a) + C D_n(a) ] \right\}$$

  where $D_p(a)$ and $D_n(a)$ are softmax-normalized average positive and negative similarities (exponentiated temperature-scaled cosine), with $C$ rebalancing positive/negative ratios. - Optimization uses SGD with momentum and weight decay. Image and 3D branches are initialized independently; image encoder benefits from segmentation pretraining.

• Time-LLM with CVPE (Shin et al., 19 May 2025):

  • Retains the original forecasting head and loss, modifying only the patch embedding step.
  • Complexity is $O(NP d_m H)$ for the router attention (with $H$ attention heads), a negligible increase over the base embedding. Learnable parameters from positional and router tables are modest compared to transformer or LLM backbones.

A plausible implication is that localizing cross-variate context to the embedding stage endows models with strong inductive biases for transferability, sample efficiency, and robustness, without incurring parameter bloat or overfitting risk commonly observed in fully channel-dependent architectures.

5. Application Pipelines: 2D–3D Retrieval and Time Series Forecasting

Patch2CAD: RGB-to-CAD shape retrieval and pose estimation (Kuo et al., 2021)

• Input objects are detected and masked, patches sampled, and patch-level CVPE encodings derived.
• Each patch retrieves its top $K_r$ CAD candidates via nearest neighbor search; majority voting across patches yields an object-level retrieval.
• Simultaneous pose estimation leverages a separate pose head, with rotation as classification and regression over K-medoid bins and translation as 2D box offsets.
• Inference per image is approximately 74 ms.

Time-LLM with CVPE: Channel-independent time series forecasting (Shin et al., 19 May 2025)

• Per-channel RevIN normalization and patch embedding receive the CVPE module before embedding are passed to the cross-attention reprogramming and LLM backbone.
• Subsequent stages of Time-LLM remain channel-independent, preserving efficiency and compositionality.

These pipelines illustrate the modularity of the CVPE concept: localized, context-enriched embeddings act as drop-in enhancers without entailing system-level architectural changes.

6. Empirical Evaluation and Comparative Performance

Patch2CAD (Kuo et al., 2021)

Empirical benchmarks demonstrate clear advantages for patch-based embedding over global alternatives, especially in scenarios lacking exact global matches:

Dataset / Metric	Patch2CAD AP_mesh	Mask2CAD AP_mesh	Relative Gain
ScanNet (Pred Box)	10.3	8.4	+22%
ScanNet (GT Box)	12.9	10.5	+23%
Pix3D (S1)	30.9	33.2	-

• Shape F-score improves from 60.6 to 63.8.
• Top-k retrieval recall consistently outperforms Mask2CAD.
• Ablations show patch size (optimal at 0.33 of ROI), use of surface normals, and patch/retrieval count to be critical.

Time-LLM with CVPE (Shin et al., 19 May 2025)

In long-term multivariate forecasting across seven benchmarks, on datasets with strong inter-series correlations (Weather, Traffic), CVPE delivers reductions in MSE up to 6.7%. On datasets with weak inter-channel correlation (ETTh1, ECL), performance matches the baseline, corroborating CVPE’s selective impact.

Dataset	Time-LLM + CVPE (MSE)	Original (MSE)	% Δ MSE
Weather	0.228	0.239	-4.6%
Traffic	0.126	0.135	-6.7%
ETTh1	0.445	0.453	-1.8%

This suggests CVPE enables channel-independent models to capture relevant cross-variable signals efficiently, without sacrificing robustness on otherwise uncorrelated data.

7. Limitations and Prospective Directions

While CVPE provides robust, context-aware embeddings, some limitations remain:

• In Patch2CAD (Kuo et al., 2021), retrieval is limited to existing CAD database shapes; it cannot synthesize or deform unseen part assemblies. Full scene or layout generation remains unaddressed.
• For multivariate time series (Shin et al., 19 May 2025), CVPE’s cross-variate modeling is restricted to the embedding step; all subsequent modeling remains channel-independent, potentially missing higher-order interactions.
• Both implementations rely on fixed patch extraction heuristics; adaptive or content-aware patching is a prospective extension.

A plausible implication is that integrating CVPE with generative or scene-level architectures could extend its applicability, especially in domains requiring synthesis beyond candidate-based retrieval or forecasting with complex spatiotemporal interaction structure.

• Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image (Kuo et al., 2021)
• Enhancing Channel-Independent Time Series Forecasting via Cross-Variate Patch Embedding (Shin et al., 19 May 2025)