TPRL-DG: Temporal RL Domain Generalization

• The paper presents a framework that learns temporal, domain-invariant policies through RL, effectively addressing shifting sequential data without target-domain samples.
• It employs sequential feature construction using autoregressive transformers and PPO-based multi-objective rewards to achieve robust cross-domain generalization.
• Empirical results on HAR benchmarks demonstrate improved accuracy (up to 2-3% gains) over traditional methods like ERM and adversarial training.

Temporal-Preserving Reinforcement Learning Domain Generalization (TPRL-DG) refers to a class of methodologies and algorithms that seek to learn representations or policies in reinforcement learning (RL) which generalize robustly across dynamically shifting domains while rigorously preserving temporal dependencies inherent in sequential data. TPRL-DG techniques enable models to extract domain-invariant, temporally coherent features or behaviors that maintain performance when exposed to new environments, users, or time-evolving dynamics—without access to target-domain samples during training or deployment.

1. Problem Definition and Motivation

TPRL-DG formalizes the challenge of domain generalization in RL settings where the domain differences are not only spatial or appearance-based but also reside in temporal shifts, sequence structure, or evolving environment parameters. Given $M$ source domains $$\mathcal{S}_{\text{train}} = \{\mathcal{S}^1, ..., \mathcal{S}^M\}$$ and temporally dependent data, the goal is to learn a policy or feature extractor $h$ that minimizes the expected loss or maximizes return in a novel, unseen target domain $\mathcal{S}_{\text{test}}$, which may differ in dynamics, user behavior, or time-evolving context (Wang et al., 2021, Ye et al., 31 Aug 2025).

This objective is central in domains such as human activity recognition (HAR)—where sensor signals are nonstationary across users and over time (Ye et al., 31 Aug 2025)—and more broadly in control, robotics, and real-world RL applications subject to abrupt or gradual temporal drifts.

2. Key Methodological Innovations

TPRL-DG frameworks integrate several methodological advances that distinguish them from classical domain generalization or transfer learning:

1. Sequential Decision-Driven Feature Construction: Feature extraction is cast as a sequential decision problem where a policy iteratively constructs feature tokens, thereby embedding and preserving temporal dynamics within representations (Ye et al., 31 Aug 2025).
2. Autoregressive Transformer-Based Generators: A Transformer-based autoregressive architecture generates feature tokens, sequentially attending to prior context and sensor sequence, ensuring the resulting features track temporal order and transitions in the input (Ye et al., 31 Aug 2025).
3. Multi-Objective Reinforcement Learning Optimization: The feature-generation policy is trained with multi-objective reward signals that jointly enforce discriminability of activity classes and cross-domain (user/environment) invariance. This is formalized as:

$$J(\theta) = \mathbb{E}_{\pi}\left[w_{\text{cls}} \cdot R_{\text{cls}}(\mathcal{Z}) + w_{\text{inv}} \cdot R_{\text{inv}}(\mathcal{Z})\right]$$

where $R_{\text{cls}}$ maximizes inter-class distances in feature space and $R_{\text{inv}}$ encourages alignment across domains/users (Ye et al., 31 Aug 2025).

1. Label-Free Reward Design: The reward construction does not require target-domain (or user) labels. Rewards are computed using source-domain data only, and the RL training adjusts the feature extractor to optimize for invariance and class separation without per-domain calibration or annotation (Ye et al., 31 Aug 2025).

3. Formal Algorithms and Theoretical Principles

Various algorithmic primitives and theoretical foundations underlie TPRL-DG systems:

• Sequential Policy Modeling: The extractor is parameterized as $\pi_\theta(z_{i,j} \mid x_i, z_{i,1:j-1})$, generating each feature token $z_{i,j}$ conditioned on sample $x_i$ and past tokens, akin to an autoregressive decoder or LLM (Ye et al., 31 Aug 2025).
• Class Discriminability and User Invariance:
  • Class centroid separation is optimized via:

  $$R_{\text{cls}}(\mathcal{Z}) = \frac{1}{C(C-1)} \sum_{c \neq c'} \|\mu_c - \mu_{c'}\|^2_F$$

where $C$ is the number of classes, and $\mu_c$ is the mean token embedding for class $c$. - Invariance reward is computed as:

$$R_{\text{inv}}(\mathcal{Z}) = -\sum_{c=1}^C \left[\frac{1}{U} \sum_{u=1}^U \frac{1}{|\mathcal{Z}_c^u|} \sum_{z_i \in \mathcal{Z}_c^u}\|z_i - \mu_c^u\|_F^2 + \frac{1}{U(U-1)} \sum_{u \neq v}\|\mu_c^u - \mu_c^v\|_F^2\right]$$

where $U$ is the number of users, $\mu_c^u$ is class $c$'s centroid for user $u$ (Ye et al., 31 Aug 2025).

• Policy Gradient Training: Proximal Policy Optimization (PPO) is employed to train the sequential feature-generation policy in a high-dimensional, continuous action space (feature vector space), explicitly optimizing the multi-objective reward (Ye et al., 31 Aug 2025).
• Transformer Architecture with Positional Encoding: The encoder processes raw sensor streams $x_i$ with added sinusoidal position encodings, and the autoregressive decoder generates the temporal tokens, enforcing causality and temporal alignment in the output features (Ye et al., 31 Aug 2025).

4. Empirical Evaluation and Benchmarks

TPRL-DG demonstrates strong empirical generalization on HAR benchmarks characterized by substantial cross-user and temporal variability:

• DSADS Dataset: Achieves average accuracy of $88.29\%$ across 8 users, with an observed $2.79\%$ improvement over strong adversarial-domain generalization baselines in challenging transfer scenarios (e.g., multi-source to single-user test) (Ye et al., 31 Aug 2025).
• PAMAP2 Dataset: Reaches approximately $74.15\%$ average accuracy, outperforming competing models by around $2.20\%$ under leave-one-group-out validation (Ye et al., 31 Aug 2025).
• Comparison with Competing Methods: TPRL-DG surpasses empirical risk minimization (ERM), recurrent DG models (AdaRNN), and adversarial-invariant training (ACON) in cross-domain generalization and temporal consistency, without sacrificing within-domain performance (Ye et al., 31 Aug 2025).

5. Relation to Broader Domain Generalization Approaches

TPRL-DG builds upon and is distinguished from prior domain generalization and temporal-invariant reinforcement learning work:

• Meta-Learning for Domain Generalization: Model-agnostic meta-learning (MLDG) and related meta-optimization procedures simulate domain shifts during training, enforcing updates that yield improvements on unseen domains, but do not explicitly model temporal tokenization or reward-driven invariance (Li et al., 2017).
• Self-Supervised and Disentanglement Strategies: Auxiliary losses (e.g., temporal disentanglement (Dunion et al., 2022)) and domain adversarial objectives (Li et al., 2021) encourage invariant representations, but TPRL-DG directly leverages RL-driven sequential decision mechanisms to construct features tailored for temporal generalization.
• Theoretical Principles: Risk bounds via domain divergence (Wang et al., 2021), invariant risk minimization, and recent advances using Koopman theory for temporally evolving distributions (Zeng et al., 12 Feb 2024) all provide complementary justifications for learning representations that align cross-domain temporal dependencies. TPRL-DG contributes practically by embedding these aims in the RL-driven autoregressive architecture.

6. Practical Applications and Implications

TPRL-DG's architecture and reward-design facilitate deployment in settings that require generalization under temporal and domain shifts, including:

• Personalized Healthcare: Cross-patient monitoring for rare or abnormal activity patterns, where individualized calibration is impractical.
• Adaptive Fitness Tracking: Robust activity classification across diversified user styles and device placements, enabled by user-invariant temporal features.
• Smart and Context-Aware Environments: Real-time recognition of human activities across diverse environments and conditions, without requiring domain-specific model retraining (Ye et al., 31 Aug 2025).

The label-free reward, sequential policy design, and autoregressive feature construction collectively yield a framework that adapts to unseen users and time-varying domains without target-specific data, scaling effectively to large, heterogeneous populations.

7. Limitations and Future Research Directions

Current TPRL-DG approaches presuppose access to source-domain labels for reward computation, and the learned invariance is conditioned on the diversity of available source users and activities. Extending the framework to accommodate:

• Continuous domain adaptation in streaming settings,
• Integration with Koopman operator-based temporal alignment for nonstationary environments (Zeng et al., 12 Feb 2024),
• Self-supervised and test-time adaptation that further relaxes dependence on source labels,

are promising research directions. A plausible implication is that incorporating model-based forecasting of environmental temporal drift (via Koopman or prompt-based methods (Hosseini et al., 2023)) could further strengthen temporal generalization with minimal parameter overhead. Additionally, refinement of policy architectures to balance invariance and discriminability under more complex or structured reward functions remains an ongoing challenge.

---

Overall, Temporal-Preserving Reinforcement Learning Domain Generalization (TPRL-DG) defines a practical and methodologically rigorous paradigm for extracting temporally consistent, domain-invariant features or policies in RL, enabling scalable, robust deployment in user- and temporally-varying real-world settings (Ye et al., 31 Aug 2025).