GaussMedAct: Gaussian Action Evaluation in Medicine

• GaussMedAct is a multivariate Gaussian encoding framework for fine-grained medical action evaluation that models rapid, noisy spatiotemporal patterns with adaptive Gaussian decomposition.
• It employs a hybrid spatial encoding with Cartesian and vector streams to capture both joint and bone-level features, enabling efficient and robust motion analysis.
• The method achieves state-of-the-art performance on CPREval-6k with reduced computational cost, using a streamlined CNN head and EM-based Gaussian tokenization.

GaussMedAct is a multivariate Gaussian encoding framework for fine-grained medical action evaluation in video, with a focus on the robust, efficient representation of rapid and noisy spatiotemporal motion patterns using adaptive Gaussian decomposition and hybrid skeletal encodings. The method is designed for real-time inference and achieves state-of-the-art accuracy on the CPREval-6k clinical benchmark, outperforming previously established graph convolutional approaches with only a fraction of the computational cost (Yang et al., 13 Nov 2025).

1. Temporally Scaled Multi-Dimensional Motion Encoding

In GaussMedAct, the core representation projects each skeletal joint's observed 2D trajectory over $T$ video frames into a three-dimensional "spatio-temporal" space, where the third coordinate is the temporally rescaled time index. For each joint $i \in \{1,\dots,M\}$, the trajectory is $\mathcal{X}_i = \{\mathbf{x}_{i,t}\}_{t=1}^T$ with $$\mathbf{x}_{i,t} = (x_{i,t}, y_{i,t}, t) \in \mathbb{R}^3$$. To balance temporal and spatial scales, a global parameter $\alpha>0$ rescales the time dimension:

$$\mathbf{x}_{i,t} \leftarrow (x_{i,t},\; y_{i,t},\; \alpha t)$$

This standardized embedding equalizes the impact of per-frame spatial variation and temporal progression, making static and dynamic temporal correlations commensurable within the subsequent statistical modeling.

2. Adaptive 3D Gaussian Tokenization of Action Primitives

GaussMedAct models each joint's projected trajectory $\mathcal{X}_i$ as a mixture of $K$ adaptive 3D Gaussian components:

$$p(\mathbf{x} \mid \boldsymbol{\theta}_i) = \sum_{k=1}^K \pi_{i,k} \;\mathcal{N}(\mathbf{x} \mid \boldsymbol{\mu}_{i,k},\,\boldsymbol{\Sigma}_{i,k})$$

where $\pi_{i,k}$ are mixture weights ($\sum_{k}\pi_{i,k}=1$), $\boldsymbol{\mu}_{i,k}$ are means in $\mathbb{R}^3$, and $\boldsymbol{\Sigma}_{i,k}$ are $3 \times 3$ positive-definite covariance matrices. Fitting is performed via the classical Expectation-Maximization (EM) algorithm on per-joint data samples.

Anisotropic motion features are captured by decomposing each covariance as $\Sigma_{i,k}=R_{i,k}\,S_{i,k}^2\,R_{i,k}^\top$ with $$S_{i,k} = \mathrm{diag}(s_{i,k}^x,\, s_{i,k}^y,\, s_{i,k}^t)$$ (scale parameters for the spatial and temporal axes) and $R_{i,k}$ a rotation matrix parameterized by a unit quaternion $q_{i,k} \in \mathbb{R}^4$. This parameterization makes each Gaussian primitive equivariant to orientation and allows flexible modeling of localized, axis-aligned or oblique motion clusters.

From each Gaussian component, a 10D "action token" is constructed by concatenating $\boldsymbol{\mu}_{i,k}$, $\mathbf{s}_{i,k}$, and $q_{i,k}$. These tokens collectively summarize the distributional shape, directionality, and dispersion of the motion.

3. Hybrid Spatial Encoding: Cartesian and Vector Streams

Medical skeletal motion contains informative spatial hierarchies at both the joint and bone (vector) levels. GaussMedAct adopts a dual-stream architecture:

• Cartesian (Joint-based) Stream: Encodes each joint as $\mathbf{J}_i = [x_i, y_i, t]$.
• Vector (Bone-based) Stream: For each skeleton edge $(i \to j)$ encodes relative displacement as $\Delta r_{ij} = \|\mathbf{J}_j - \mathbf{J}_i\|_2$ and orientation $\theta_{ij} = \arctan2(y_j - y_i, x_j - x_i)$, forming $$\mathbf{B}_{ij} = [\Delta r_{ij},\, \theta_{ij},\, t]$$.

Each stream is independently embedded via shallow neural encoders (e.g., 1D/2D CNNs) and later fused. Fusion strategies include cross-attention between the two stream-specific feature maps or interleaved concatenation at the token level. This architectural separation preserves the distinct geometric and relational semantics before their integration, enhancing motion discrimination.

4. Pipeline, Training Regimen, and Architectural Considerations

The full GaussMedAct inference pipeline consists of:

1. Skeleton Extraction: RTMpose produces joint coordinates from 32-frame clips ($\sim$4–5 GFLOPs).
2. Feature Construction: Compute $\mathbf{J}_i$ and $\mathbf{B}_{ij}$; run the EM algorithm to fit $K\approx6$ Gaussians per joint/bone, producing 10D tokens.
3. Feature Fusion and Classification: Tokens are fed to a spatio-temporal CNN head with global pooling and a final linear classification layer.

The loss function combines cross-entropy (with label smoothing) and MixUp-based augmentation, where synthetic samples $(\tilde{x}, \tilde{y})$ are created as convex combinations of randomly paired samples $(x_i,y_i)$, $(x_j,y_j)$ with $\lambda\sim\mathrm{Beta}(\alpha, \alpha)$, enhancing rare-class robustness.

Training enforces valid Gaussian parameters ($\pi_{i,k}\geq0$, $\sum_k \pi_{i,k}=1$, $s_{i,k}^u>0$, $\|q_{i,k}\|=1$). The computational cost is approximately 4.45 GFLOPs per skeleton-only sample, with ablations (removing the bone stream) reducing this to 2.23 GFLOPs.

5. Empirical Performance and Comparative Results

On the CPREval-6k dataset (6,372 videos, 22 clinical action classes):

Model	Top-1 Accuracy	Top-5 Accuracy	Class-Mean Accuracy	GFLOPs
GaussMedAct	92.12 %	98.36%	90.82 %	4.45
ST-GCN	86.18 %	—	—	43.76

This yields a +5.94% accuracy improvement over the ST-GCN baseline, with only $\approx$10% of the computational burden. In cross-benchmark experiments on CPR-Coach, GaussMedAct achieves 95.24% Top-1 accuracy (+2.78% over the prior best) and matches the best Top-3 performance, demonstrating both strong generalization and robustness under domain shift.

6. Technical Advantages and Model Robustness

The multivariate Gaussian representation provides several benefits:

• Action Primitive Compactness: Adaptive Gaussians parameterize temporally and spatially local motion segments, facilitating noise robustness and outlier suppression in fine-grained evaluation.
• Anisotropic Modeling: Full $3\times3$ covariance with rotation invariance captures the correct spread and alignment of rapid, articulated motion, outperforming axis-aligned or norm-based pooling schemes.
• Hybrid Representation: Separate modeling of joints and bones leverages both framewise and relational cues, boosting detection accuracy, especially for subtle or occluded actions.
• Efficiency: Tokenized Gaussian representations dramatically reduce FLOPs, enabling real-time inference (4.45 GFLOPs vs. 43.76 for ST-GCN).

A plausible implication is that the reduced-dimensional, yet information-rich, Gaussian action tokens could serve as a task-agnostic pre-processing or compression stage in broader medical action analysis pipelines.

7. Benchmarking and Open-Source Resources

GaussMedAct was developed and validated on the CPREval-6k dataset, a comprehensive multi-view, multi-label medical action benchmark comprised of 6,372 expert-annotated videos and 22 clinical labels. The experimental protocol includes real-time evaluation of both intra-set and cross-dataset transfer, providing a stringent test of robustness and generalization.

The method and supporting data are detailed in the publication "Multivariate Gaussian Representation Learning for Medical Action Evaluation" (Yang et al., 13 Nov 2025).