Bayes-CPACE: Dual Approaches in FPCA & RL

• Bayes-CPACE is a dual-framework methodology that addresses uncertainty quantification in both functional data analysis and reinforcement learning.
• In density analysis, it uses a Monte Carlo EM algorithm in Bayes space to perform FPCA on sparsely sampled densities, yielding interpretable principal components.
• In reinforcement learning, it employs covering-based value estimation and optimistic nearest-neighbor strategies to achieve PAC-optimal exploration in continuous BAMDPs.

Bayes-CPACE is a nomenclature used for two independently developed methods within statistical learning and reinforcement learning, each addressing fundamental challenges in uncertainty quantification and data efficiency. In the context of functional data analysis, Bayes-CPACE (Bayes-space Compositional Principal Analysis by Conditional Expectation) implements functional principal component analysis (FPCA) for sparsely sampled densities via Bayes spaces. In reinforcement learning, Bayes-CPACE (PAC Optimal Exploration in Continuous Space Bayes-Adaptive Markov Decision Processes) is a sample-efficient algorithm for exploration in continuous-state and -action Bayesian Markov Decision Processes (BAMDPs) with formal Probably Approximately Correct (PAC) performance guarantees.

1. Bayes-CPACE in Functional Data Analysis: Bayes-space FPCA for Sparsely Sampled Densities

Bayes-CPACE for density data analysis is grounded in the geometry of Bayes spaces to address the challenge of FPCA when only sparse, individual samples from each underlying density are observed. The methodology bypasses pre-smoothing or histogram-based density estimation, directly modeling all sources of uncertainty while respecting the compositional constraints intrinsic to densities (unit integral and positivity) (Steyer et al., 2023).

Objects of interest are probability densities $f$ on a compact domain $\mathcal T \subset \mathbb R$, considered within the Bayes Hilbert space $\mathcal B$, an extension of Aitchison geometry to the infinite-dimensional case. The key is the centered log-ratio (clr) transform: $\clr(f)(x) = \log f(x) - \frac{1}{|\mathcal T|} \int_{\mathcal T} \log f(t)\,dt,$ establishing an isometric isomorphism $\clr:\mathcal B \rightarrow L_0^2(\mathcal T)$, the space of square-integrable mean-zero functions.

The statistical model assumes the transformed latent densities $\clr(f_i)$ are drawn independently from a Gaussian process (GP), truncated to an $N$-dimensional orthonormal basis. Observed data are raw samples from each density, with the principal component scores treated as missing data.

Estimation proceeds via a Monte Carlo Expectation-Maximization (MCEM) algorithm:

• E-step: Importance sampling estimates the conditional expectation with respect to the intractable posterior over principal scores.
• M-step: Weighted maximum likelihood updates for Gaussian parameters.
• Principal subspace recovery: The posterior covariance matrix is spectrally decomposed for principal directions in clr/Bayes space.
• Statistical inference: The approach handles all uncertainty sources jointly, providing interpretable, regularized principal modes.

Applications include trend analysis of temperature extremes and rent distributions, with empirically verified improvements in stability and interpretability of principal components over two-step, pre-smoothed alternatives, especially under sparse or heterogeneous sampling conditions.

2. Bayes-CPACE in Reinforcement Learning: PAC Optimal Exploration in Continuous BAMDPs

In the context of model-based reinforcement learning, Bayes-CPACE is the first algorithm achieving PAC (Probably Approximately Correct) guarantees for Bayes-Adaptive MDPs with continuous state and action spaces (Lee et al., 2018). BAMDPs extend the classical MDP by encoding model uncertainty through a latent random variable $\phi \in \Phi$ (model instances). The agent maintains a Bayesian posterior $b$ over $\Phi$, and the hyper-state is defined as $x = (s, b)$, where $s$ and $a$ are continuous.

The principal challenge is that the optimal Bayesian policy is intractable. Bayes-CPACE leverages covering-based value function approximations and exploits the Lipschitz continuity of the optimal value function:

• State-belief-action space covering: The algorithm maintains a finite sample set that forms an $\varepsilon$-cover of reachable tuples, allowing local value estimation to generalize with bounded error.
• Optimistic nearest-neighbor value estimation: Value is estimated by an average over nearest samples, optimistic up to problem-dependent bounds.
• Sample addition: If a query is 'unknown' (too far from the cover), a new transition is added and value iteration is recomputed on the cover.

Under mild assumptions (bounded rewards, Lipschitz continuity, belief contraction), Bayes-CPACE achieves sample complexity polynomial in the covering number $N(\varepsilon)$, $1/\varepsilon$, $1/\delta$, and $1/(1 - \gamma)$. A main result is that with probability at least $1 - \delta$, the number of $\varepsilon$-suboptimal steps is at most

$$O\left( \frac{N(\varepsilon)}{\varepsilon^2} \log(N(\varepsilon)/\delta) \log(Q_{\max}/\varepsilon) \right).$$

Empirical results on benchmark tasks (e.g., Tiger, Chain, Light–Dark Tiger) demonstrate Bayes-CPACE attains higher or comparable reward relative to model-free or myopic baselines, and efficiently discovers exploration-then-exploitation strategies.

3. Methodological Details and Algorithmic Frameworks

Variant	Domain	Core Algorithm
Density FPCA (Steyer et al., 2023)	Functional Data Analysis	MCEM for GP FPCA
RL Exploration (Lee et al., 2018)	Reinforcement Learning	Covering+Value Iter.

Functional Density Case:

• Latent GP prior: Principal component expansion in clr-transformed Bayes space.
• Monte Carlo EM: Approximates the intractable E-step using importance sampling; M-step updates the mean/covariance of latent GP scores.
• Numerical integration: Normalization integrals approximated via quadrature.
• Spectral decomposition: Determines principal directions of density variation in clr/Bayes space.

Reinforcement Learning Case:

• Sample set maintenance: Expands only when the state-belief-action triple is 'unknown.'
• Optimistic value estimation: Nearest-neighbor-based averaging, capped at theoretical reward maximum.
• Value iteration: Approximates the Bellman backup on the finite cover set.
• Efficiency enhancements: Use of upper-bound heuristics, batch updates, truncation of value iteration, and coverage reduction for 'known' belief vectors.

4. Empirical Applications and Results

Functional data applications demonstrate the utility of Bayes-CPACE for exploratory and dimensionality-reduction tasks in settings with sparse, non-uniform sampling. In the Berlin temperature series, Bayes-CPACE recovers trends in the frequency and magnitude of extreme heat, with principal component scores revealing long-term climate trends. In the Munich rental price study, interpretable density shifts and shrinkage towards the GP prior prevent overfitting in low-sample districts, outperforming naive two-step approaches (Steyer et al., 2023).

In reinforcement learning, Bayes-CPACE is evaluated on discrete and continuous BAMDPs. It outperforms or matches QMDP, POMDP-lite, SARSOP, and POMCPOW baselines on cumulative reward, with strictly higher exploration efficiency in continuous BAMDP tasks. The algorithm learns to efficiently gather information before committing to exploitation, as in the Light–Dark Tiger environment (Lee et al., 2018).

5. Limitations and Extensions

The FPCA method relies on the appropriateness of GP priors and the quality of the numerical quadrature for integrating density normalizations. In reinforcement learning, the primary limitation is the requirement of a finite or discretized set of latent MDPs $\Phi$; extension to continuous parameter spaces is possible by discretization, with explicit error bounds on the approximation. The covering number scaling poses computational challenges in high-dimensional state/action or belief spaces. For practical continuous-action maximization in RL, efficient heuristics are still an open problem. Richer priors, e.g., hierarchical or nonparametric, can be incorporated by summarizing beliefs onto tractable support.

6. Significance and Relation to the Literature

Bayes-CPACE establishes foundational techniques in two distinct areas. In functional data analysis, it operationalizes Bayes space geometry for density inference, generalizing Aitchison's compositional data analysis to infinite-dimensional densities. In reinforcement learning, Bayes-CPACE is the first PAC-optimal algorithm for continuous BAMDPs, combining Bayes-adaptive policy search, nearest-neighbor value approximation, and theoretical sample complexity guarantees. These contributions address persistent challenges related to uncertainty propagation, sample efficiency, and principled exploration in high-dimensional settings (Steyer et al., 2023, Lee et al., 2018).