Tensorized algorithms and scalable filtering methods for hidden Markov and factorial hidden Markov models

Abstract: A common method for the representation and analysis of time-series data is the hidden Markov model (HMM), where each observation is associated with a hidden state that evolves over time. However, many real-world systems are influenced by multiple independent factors, which are more naturally represented by factorial hidden Markov models (fHMM), where several hidden Markov chains jointly generate the observed data. Although an fHMM provides a richer and more realistic representation of many real-world systems, it can be reformulated as an equivalent HMM, but with a significantly larger state-space, leading to a severe increase in computational cost. In particular, the forward filtering algorithm, which is central to evaluation, decoding, and estimation tasks, becomes prohibitively expensive even for small systems. This work focuses on developing scalable methods for time-series analysis using tensor algebra to exploit the multidimensional structure of fHMM directly, without constructing intermediate HMM representations. Our novel filtering approach significantly improves computational performance and enables the efficient analysis of large systems and datasets, extending the scope of fHMM and providing a practical framework for data intensive applications.

• The paper presents tensorized algorithms that enable efficient forward filtering and Viterbi decoding for both HMMs and factorial HMMs.
• It leverages mode-wise tensor contractions to reduce computational complexity from exponential to nearly linear scaling in state-space size.
• Benchmark experiments show speedups over 4500x and demonstrate numerical equivalence with standard vectorized methods.

Tensorized Algorithms and Scalable Filtering for HMM and fHMM

Introduction

Hidden Markov models (HMMs) serve as foundational tools for probabilistic modeling of time-series data with latent state dynamics. However, many systems—such as those in speech, genomics, or biophysics—are influenced by multiple independent latent factors, motivating the use of factorial hidden Markov models (fHMMs), which feature several parallel Markov chains. The principal computational barrier for fHMMs is state-space explosion: naively, exact inference involves a cost exponential in the number of chains, limiting their practical applicability. This paper introduces tensorized algorithms for fHMM inference, exploiting the multilinear algebraic structure to achieve substantial computational and memory efficiency, and provides thorough benchmarking against the standard vectorized approaches (2607.07008).

Mathematical Formulation: HMMs and fHMMs in Tensor Language

The formalism is centered around the realization that the combinatorial state-space for $K$ parallel Markov chains is the Cartesian product of subsystem state-spaces. For $K$ subsystems with state sizes $M^k$, the joint state-space has cardinality $M = \prod_{k=1}^K M^k$. In standard implementations, forward filtering and Viterbi decoding involve matrix-vector operations over this exponential space.

This work systematically develops a tensor calculus for state and parameter representations. The forward algorithm is generalized to operate with tensors whose axes represent the factors (chains), allowing recursive filtering and decoding to be performed with mode-wise contractions and tensor products, rather than with Kronecker-flattened vectors and matrices. The authors employ vectorization and folding/unfolding mappings, mode-$k$ products, tensor contractions, and Kronecker product identities to formalize the recursions.

Tensorized Forward Filtering: Algorithmic Advances

For fHMMs, the standard approach—recasting as a single joint HMM—leads to computational cost per iteration scaling as $O(N M^2)$, i.e., $O(N \prod_{k=1}^K (M^k)^2)$, due to dense matrix-vector multiplications. By exploiting tensor contractions that mirror the independence structure, the tensorized algorithm achieves a per-step complexity of $O(N M \sum_k M^k)$ and memory requirements are dramatically reduced, since Kronecker-expanded objects are never instantiated.

In the tensorized algorithm, the recursive filtering step updates a $K$-mode tensor $\mathbb{A}_n$ (of shape $K$0):

• The initial step involves Hadamard products and mode-wise stretches of the emission tensor and initial state distributions.
• Each subsequent step contracts along each mode with the corresponding subsystem transition matrix and takes a Hadamard product with the new emission tensor slice.

This yields a speedup scaling as $K$1 for $K$2 subsystems of size $K$3, relative to naive vectorization—transformative for practical settings with $K$4.

Numerical Results: Accuracy and Performance Benchmarks

To validate both correctness and efficiency, the authors conduct extensive benchmark experiments using synthetic fHMMs of varying dimension and number of chains. Numerical accuracy is verified: over $K$5 random trials, maximum discrepancies between tensorized and vectorized outputs are at the scale of machine epsilon, demonstrating that reformulating the recursions in tensor algebra introduces no detectable numerical artifacts.

For performance, runtime scaling is assessed for different system sizes:

• When increasing a single subsystem's state size (keeping others fixed), the standard vectorized and tensorized methods both grow predictably, but the tensorized scales much more gently, only linearly in total state-size.
• When increasing all $K$6 subsystems' sizes simultaneously ($K$7), runtimes again track theoretical predictions: vectorized scales as $K$8, tensorized as $K$9.
• In high-dimensional settings, the tensorized algorithm achieves speedups exceeding $M^k$0 in some benchmarks, and remains tractable for subsystem sizes as large as $M^k$1, whereas the vectorized approach fails due to memory exhaustion.

  Figure 1: Assessment of the runtime performance of the vectorized and tensorized filtering algorithms. Only one of the sizes $M^k$2 or $M^k$3 varies at a time; asymptotic scaling is quadratic or linear, respectively.

The memory efficiency is an equally significant achievement; in practical implementations, the tensorized method enables analyses of models previously excluded from consideration.

Additional arrangement experiments show that the order of tensor modes (i.e., placement of large state spaces in the contraction sequence) may yield minor performance benefits, with the fastest execution when larger subsystems are placed on the early tensor modes. This insight can inform practical implementation choices for further efficiency.

Figure 2: Assessment of the runtime performance with all subsystem sizes equal ($M^k$4). Tensorized approaches scale quartically; vectorized methods scale sextically.

Decoding Algorithms and Exactness Guarantees

The tensorized formalism is extended to the Viterbi decoding problem, with mode-wise maximizations replacing vectorized argmax operations. The resulting algorithm provably yields identical sequences to the vectorized approach, as confirmed by zero Hamming distances across $M^k$5 synthetic benchmarks.

Practical and Theoretical Implications

This work demonstrates that tensor algebra can be used to rigorously and efficiently realize exact inference for factorial Markovian models, bypassing both prohibitive runtime and memory constraints of conventional approaches. Importantly, the algorithms maintain strict numerical equivalence to the standard vector operations, with no loss of stability or precision.

The tensorized schemes align naturally with modern scientific computing tools that support multidimensional arrays and optimized tensor contractions (e.g., NumPy, MATLAB), facilitating their adoption in large-scale data pipelines and nonparametric Bayesian inference workflows.

Beyond the immediate task of forward filtering and decoding, this approach lays groundwork for several advances:

• Scaling Bayesian inference methods (e.g., generic Metropolis–Hastings MCMC) for sequential models with non-conjugate likelihoods, rare transitions, or highly structured latent spaces.
• Enabling principled model selection/comparison where repeated likelihood evaluations are required, e.g., for evidence-based dynamical system identification or hidden process partitioning.
• Extending the approach to nonlinear or non-Gaussian state-space models that also exhibit factorial structure.

Conclusion

The proposed tensorized filtering and decoding algorithms for fHMMs offer a scalable, memory-efficient, and numerically robust alternative to standard vectorized implementations. By leveraging the multi-way structure inherent in factorial models, the algorithms enable principled time-series inference for dimensionalities previously considered intractable, with broad applicability in domains such as genomics, signal processing, and biophysics. The availability of fast, exact algorithms for these models is likely to catalyze further developments in scalable Bayesian inference, sequential data modeling, and resource-constrained applications requiring interpretable latent variable modeling.