---
title: Zero-Shot Human Activity Recognition
url: https://www.emergentmind.com/topics/zero-shot-human-activity-recognition-har-d9096a8b-8f91-4b8b-b4c6-228a02f714dd
type: topic
---

# Zero-Shot Human Activity Recognition

Zero-shot Human Activity Recognition (HAR) is the paradigm in which systems are required to recognize human activities that are not encountered during training, using only auxiliary domain knowledge—such as language or semantic attributes—for transfer. This task addresses the core limitation of standard supervised HAR: the dependence on exhaustive labeled data for every possible activity, sensor modality, or environment. Recent progress has enabled reliable zero-shot HAR across sensor modalities (e.g., IMU, video, radar, LiDAR, smart-home event sensors), leveraging joint embedding spaces, cross-modal representations, retrieval-augmented reasoning, and the interpretability of large language models (LLMs). These advances radically improve HAR generalization to new subjects, activities, or domains, and foster transparency in predictions.

## 1. Zero-Shot Human Activity Recognition: Formalization and Motivation

Zero-shot HAR extends the zero-shot learning (ZSL) framework—originally studied for classification and vision tasks—to scenarios where no labeled data for certain target activities ("unseen classes") are available at training. Formally, consider an activity label set $\mathcal{Y} = \mathcal{Y}_\text{seen} \cup \mathcal{Y}_\text{unseen}$, with $\mathcal{Y}_\text{seen} \cap \mathcal{Y}_\text{unseen} = \emptyset$. The objective is to learn a function $f: X \to \mathcal{Y}$, such that for input $x$ (e.g., a raw IMU sequence or a video clip), $f(x) \in \mathcal{Y}_\text{unseen}$ is possible, even though no training instances of $\mathcal{Y}_\text{unseen}$ are present in labeled training data [2507.00050][1709.05107][2507.13482].

Zero-shot ability is essential for HAR deployment in fields such as healthcare, smart environments, and mobile sensing. Here, unseen user behaviors, new sensor configurations, or shifting environmental affordances can generate activities absent from pre-collected HAR datasets. The primary enablers for zero-shot HAR are (1) cross-modal/semantic transfer (e.g., from word embeddings, language models, or video primitives), and (2) robust evaluation on carefully partitioned datasets—ensuring that test activities (or subjects) are absent from any model adaptation or training.

## 2. Semantic and Cross-Modal Representations

Central to zero-shot HAR is the construction of a semantic embedding space that permits transfer between observed and unobserved classes. Several architectures have been reported:

- **Semantic Alignment via Word Embeddings or Descriptors:** Models such as SEZ-HARN and Joint Latent Ranking Embedding map sensor inputs and activity labels into a shared latent space, leveraging word embeddings (e.g., GloVe, Word2Vec) or text descriptors as semantic anchors. Feature similarity (typically dot-product or cosine) enables direct comparison, so that even when an activity is unseen, its embedding can serve as a query prototype [1709.05107][2507.00050].

- **Vision-Language Alignment and Video Semantics:** Approaches such as video-based autoencoders [2002.02265], TENT [2311.08245], and cross-modal pretraining [2507.13482] fuse features from videos and textual descriptors. For instance, TENT uses CLIP-style encoders for both sensor data and language, training with a contrastive InfoNCE objective, supplementing class labels with short descriptions and learned soft prompts to enhance generalization.

- **Natural Language Summarization and Embedding:** In smart-home settings, raw sensor events are heuristically summarized into natural language (human-readable) sentences, which, together with one-sentence activity descriptors, are embedded in a pretrained SBERT model. Cosine similarity between sensor summary and activity descriptor embeddings directly yields zero-shot predictions [2507.21964].

Table 1: Shared Semantic Embedding Paradigms

| Approach                | Modality         | Semantic Anchor           |
|-------------------------|------------------|--------------------------|
| Joint Latent Embedding  | Video            | Word2Vec, attributes     |
| SEZ-HARN                | IMU, video       | I3D video features       |
| SBERT Summary           | Event sensors    | Sentential descriptors   |
| TENT                    | Video, LiDAR     | CLIP text+desc+prompts   |
| PatchTST+VideoMAE       | IMU, video       | Joint cross-modal space  |

The diversity of representations (from deep feature embeddings to structured language) demonstrates the generality of the semantic transfer principle in zero-shot HAR.

## 3. Algorithms and Architectural Paradigms

Zero-shot HAR models fall into several principal design families:

- **Joint-Embedding Models:** These architectures map both sensor input and class descriptors to a shared latent space, enforcing proximity of semantically related pairs via alignment or ranking losses (e.g., cosine similarity, cross-modal ranking). Zero-shot classification operates as nearest-neighbor search or scoring in this space [2002.02265][1709.05107][2507.00050][2311.08245][2507.13482].

- **LLM-Prompted Classification:** Emerging works demonstrate that frozen large language models (GPT-4, Gemini) can be prompted with raw or preprocessed sensory input (e.g., IMU time series directly serialized as numeric tuples) plus candidate activities. Contextual prompting—particularly with "role play" and "step-by-step analysis" cues—enables robust zero-shot activity assignment without gradient updates [2403.02727].

- **Retrieval-Augmented and Agent-Based Reasoning:** Frameworks such as ZARA maintain a feature-level knowledge base for all class pairs and dynamically retrieve proto-typical examples via a frozen time-series embedder. A hierarchical agent pipeline embedded in an LLM then performs feature selection, evidence pruning, and chain-of-thought justification, achieving state-of-the-art zero-shot accuracy and interpretability [2508.04038].

- **Language Modeling via Embedding:** Instead of online prompting, sensor events and activities are converted into natural language representations and embedded using a static sentence-BERT model, allowing for deterministic and privacy-preserving zero-shot recognition [2507.21964].

## 4. Evaluation Protocols and Benchmarks

Proper zero-shot HAR assessment requires strict subject/activity partitioning and diverse metrics:

- **Partitioning:** Common protocols hold out a subset of activities as "unseen" (class-wise split), or exclude certain users/participants (subject-wise split). For example, HARGPT used test-unseen splits—20% of users never seen during training—for both Capture24 and HHAR datasets [2403.02727]. SEZ-HARN performs k-fold class-wise splits, reserving 20–30% of classes as $C_u$ [2507.00050]. Multi-label settings further restrict training and evaluation overlap by label or by instance [1709.05107].

- **Metrics:** Typical evaluation metrics include per-class/macro F1, balanced accuracy, mean average precision (mAP), Top-N retrieval accuracy, recall@K, and mean reciprocal rank (MRR). Macro-F1 and per-class accuracy guard against class imbalance [2508.04038][2507.13482][2403.02727][1709.05107].

- **Datasets:** Zero-shot HAR spans modalities and domains:
  - Video: Kinetics [2002.02265], Charades, Breakfast [1709.05107]
  - IMU: PAMAP2, DaLiAc, UTD-MHAD, MHEALTH [2507.00050], HHAR, Capture24 [2403.02727]
  - Multisensor/smart-home: CASAS, MARBLE, ARAS [2507.21964]
  - IoT: MM-Fi (video, LiDAR, radar) [2311.08245]
  - OOD: MMEA, Parkinson’s Disease IMU [2507.13482]
  - Time-series: Opportunity, UCI-HAR, WISDM, DSADS [2508.04038]
  - Wireless CSI: not detailed owing to lack of verbatim data [2312.07076]

## 5. Quantitative Results and Empirical Comparisons

Zero-shot HAR methods have demonstrated significant gains over classical or unimodal benchmarks, with state-of-the-art results supported by rigorous ablation studies.

- **LLM-Prompted Zero-Shot HAR:** HARGPT (GPT-4, Chain-of-Thought prompting) achieved macro F1 of 0.795 (Capture24, 4-class) and 0.790 (HHAR, 2-class) on unseen-subject splits, surpassing conventional DCNNs, LIMU-LSTM, RF, and SVMs. Direct-output (no CoT) prompts led to much lower F1, confirming the criticality of prompt engineering [2403.02727].

- **Self-Explainable Models:** SEZ-HARN matched or exceeded the strongest black-box ZS-HAR models on DaLiAc, UTD-MHAD, and MHEALTH, and was within 3% of the best model on PAMAP2. Explanation modules (skeleton video generation) incurred no meaningful accuracy degradation, validated both quantitatively (DTW, DFD) and via human studies (>80% correct super-class identification) [2507.00050].

- **Retrieval- and Agent-Based Approaches:** ZARA yielded macro F1 up to 92.5% (Opportunity) and 81.4% on average across eight benchmarks—2.53x improvement over best baselines (e.g., UniMTS). Ablations showed that each module (retrieval, prior knowledge, evidence pruning) contributed 10–20 percentage points to accuracy [2508.04038].

- **Cross-Modal Pretraining:** Self-supervised IMU-video alignment with PatchTST and VideoMAE improved zero-shot balanced accuracy to 33.9% on MMEA and 24.1% for PD, outperforming IMU2CLIP and vision-language variants [2507.13482]. Similar cross-modal alignment drives TENT’s advances in video, LiDAR, and radar-based HAR (absolute gains of 12–23.5% in top-1 accuracy over vision-language baselines) [2311.08245].

- **Language Modeling via Embedding:** Sentence-BERT smart-home pipeline delivered accuracy up to 81% (MARBLE) and 71% (CASAS Aruba), on par or better than prompt-based LLM solutions, with privacy and reliability advantages [2507.21964].

## 6. Explainability, Limitations, and Practical Considerations

Explainability is increasingly fundamental in zero-shot HAR:

- **Self-Explanation via Skeleton Generation:** SEZ-HARN uniquely provides skeleton video traces for each prediction, with reconstruction losses enforcing kinematic fidelity to auxiliary video semantics. Human studies and DTW/DFD metrics confirmed the interpretability and realism of generated explanations [2507.00050].

- **Structured Agent Rationales:** ZARA’s agent pipeline yields natural language rationales citing discriminative features and retrieved evidence, judged faithful in human evaluation [2508.04038].

- **Language Embedding Transparency:** Fixed SBERT encoders and handcrafted descriptors produce deterministic, inspectable decision rules without reliance on external APIs or variable black-box LLM outputs [2507.21964].

Limitations highlighted across recent literature include:
- Brittle or verbose LLM responses (particularly under variant prompt styles) [2403.02727].
- Manual burden in crafting activity descriptors and aggregating summaries [2507.21964].
- Dependence of cross-modal or retrieval-based models on embedding quality and coverage—rare or complex activities may yield ambiguous, less reliable predictions [2508.04038][2507.13482].
- Real-time performance constraints, especially for agent-based or multi-stage retrieval models [2508.04038].

## 7. Open Problems and Future Research Directions

The field continues to evolve along several axes:

- **Benchmark and Protocol Standardization:** There is an ongoing need for widely adopted zero-shot HAR benchmarks, strict split protocols, and unified metrics to foster fair comparison and reproducibility [2403.02727][2507.21964].

- **Multimodal and Few-shot Extensions:** Future work is aimed at fusing additional sensor modalities (e.g., audio, barometer, EMG, WiFi CSI) and combining zero-shot with few-shot strategies—using embedded class prototypes from sparse labeled instances for improved discrimination [2507.13482][2507.21964].

- **Automated Descriptor Generation:** Semi-automatic or privacy-preserving LLM-assisted generation of sensor summaries and activity descriptors can mitigate manual overhead and increase portability [2507.21964].

- **Adaptive and Plug-and-play Systems:** Retrieval-based and agent-oriented frameworks are being extended for dynamic knowledge base updating, online learning, and flexible sensor sets [2508.04038].

- **Transparent Reasoning and Trust:** Combining visual, textual, and tabular explanation modalities (e.g., skeleton video + natural language + attention overlays) is advocated to further enhance user trust and regulatory compliance in critical environments [2507.00050][2508.04038].

Zero-shot HAR remains a rapidly developing area, unifying advances in transfer learning, multimodal representation, language modeling, and human-centered AI, with strong empirical evidence for both performance and interpretability over an expanding range of sensor platforms and settings.

Source: https://www.emergentmind.com/topics/zero-shot-human-activity-recognition-har-d9096a8b-8f91-4b8b-b4c6-228a02f714dd