--- title: Skeleton-Cache for Zero-Shot Action Recognition url: https://www.emergentmind.com/topics/skeleton-cache type: topic --- # Skeleton-Cache for Zero-Shot Action Recognition Skeleton-Cache is a training-free test-time adaptation framework designed to improve the generalization of skeleton-based zero-shot action recognition (SZAR) models to unseen action classes during inference. It reframes the inference process as a non-parametric, retrieval-based classification over a dynamically constructed cache of structured skeleton descriptors, combining both global and fine-grained local representations. Skeleton-Cache uniquely integrates semantic priors from large language models (LLMs) to guide the fusion of descriptor-wise predictions, enabling dynamic adaptation to unseen action distributions without additional training or access to original training data [2512.11458]. ## 1. Motivation and Problem Setting Skeleton-based zero-shot action recognition seeks to classify actions represented as skeleton sequences, where test-time classes are disjoint from those available during training. Conventional test-time adaptation methods either require gradient-based updates to the model—which introduces risk of overfitting and inference latency—or rely solely on a holistic skeleton embedding, neglecting spatial and temporal details critical for action discrimination. Skeleton-Cache addresses these challenges through two core innovations: (a) a lightweight non-parametric cache that stores structured descriptors dynamically collected from confident test exemplars, and (b) per-class, per-descriptor fusion weights derived from LLMs, which ensure semantically meaningful adaptation to new action categories. ## 2. Cache Architecture and Descriptor Representation The Skeleton-Cache comprises $|\mathcal{Y}_u|$ class-specific blocks, each capable of holding up to $K$ entries, where $\mathcal{Y}_u$ is the set of unseen classes. Each cache entry is a triplet $(e_k, e_y, e_h)$: - $e_k \in \mathbb{R}^{D\times N}$: Cache key, a concatenation of $(P+Z+1)$ descriptors each of dimension $N$, representing global, spatial (body-part), and temporal (motion phase) features. - $e_y \in \{0,1\}^{|\mathcal{Y}_u|}$: One-hot class label. - $e_h \in \mathbb{R}$: Prediction confidence scored as negative entropy. Descriptors are extracted from the frozen SZAR backbone. Given $F \in \mathbb{R}^{N \times T \times V}$, the backbone feature tensor: - Global descriptor: $$ \mathbf{g} = \frac{1}{VT} \sum_{t=1}^T \sum_{v=1}^V F_{:, t, v} \in \mathbb{R}^N $$ - Local spatial descriptors for $p = 1, \dots, P$ body-part groups: $$ \mathbf{s}_p = \frac{1}{|\mathcal{V}_p|T} \sum_{v \in \mathcal{V}_p} \sum_{t=1}^T F_{:, t, v} \in \mathbb{R}^N $$ - Local temporal descriptors for $z = 1, \dots, Z$ phases: $$ \mathbf{t}_z = \frac{1}{V|\mathcal{T}_z|} \sum_{t \in \mathcal{T}_z} \sum_{v=1}^V F_{:, t, v} \in \mathbb{R}^N $$ These are concatenated to form the cache key: $$ e_k = \mathrm{concat}\bigl(\mathbf{g}, \mathbf{s}_1, \ldots, \mathbf{s}_P, \mathbf{t}_1, \ldots, \mathbf{t}_Z\bigr) \in \mathbb{R}^{(P+Z+1)\times N} $$ The cache is constructed online during testing, with only high-confidence predictions used as exemplars. There are no gradient updates or access to training data; cache updates and lookups are computationally efficient. ## 3. Retrieval and Fusion Mechanism Test-time adaptation consists of querying the cache and fusing predictions: - For each incoming sample $x$, query embeddings $\{q^{(d)}\}_{d=0}^{P+Z}$ are computed as described above. - Affinity scores between the query descriptor and cached descriptors are computed using cosine similarity and an exponential temperature kernel: $$ a_{j,i}^{(d)} = \exp\left[-\beta\left(1 - \cos(q^{(d)}, k_{j,i}^{(d)})\right)\right] $$ where $k_{j,i}^{(d)}$ is the $d$-th descriptor for class $j$, slot $i$, and $\beta$ is a hyperparameter. - The descriptor-wise logit vector for each type $d$ is computed as: $$ o^{(d)} = a^{(d)} Y \in \mathbb{R}^{1 \times |\mathcal{Y}_u|} $$ where $Y$ is a label matrix of cached entries. - Aggregating over all descriptor types yields $O \in \mathbb{R}^{(P+Z+1)\times |\mathcal{Y}_u|}$. ## 4. Semantic Priors via Large Language Models Prior knowledge about the importance of global, spatial, and temporal descriptors for each action class is extracted using GPT-4, with a single prompt per class. The LLM is tasked to output: - "spatial": four nonnegative weights (over [Head, Torso, Arms, Legs]) summing to 1, - "temporal": three nonnegative weights (over [Begin, Middle, End]) summing to 1, - "gamma": $\gamma \in [0,1]$, global/locale trade-off. The resulting per-class weight vector is: $$ \tilde w^{(c)} = \left[\gamma, (1-\gamma) w_\text{spa}^{(1)}, \ldots, (1-\gamma) w_\text{spa}^{(P)}, (1-\gamma) w_\text{tmp}^{(1)}, \ldots, (1-\gamma) w_\text{tmp}^{(Z)}\right] $$ $\tilde w^{(c)}$ is then $\ell_1$-normalized to $w^{(c)} \in \mathbb{R}^{P+Z+1}$. These semantic priors are statically precomputed per class for all unseen actions. ## 5. End-to-End Inference and Adaptation Protocol For a test sample, the full protocol can be summarized as: 1. Extract skeleton descriptors ($\mathbf{g}, \{\mathbf{s}_p\}, \{\mathbf{t}_z\}$). 2. Compute original SZAR logits $\hat{\phi}$ and entropy $h$. 3. Update the cache for the predicted class $\hat{y}$ if the current slot count $