Non-Sequential Markov Modeling

• Non-Sequential Markov Modeling is a framework that adapts traditional Markov chains to analyze processes with history dependence, non-geometric durations, and irregular observation intervals.
• It employs methods like first-order equivalent Markov chains and explicit-duration models to preserve one-dimensional marginal distributions while relaxing standard assumptions.
• These models enhance practical inference in applications such as behavioral analysis, regime-switching time series, and segmentation by converting non-Markovian dynamics into tractable representations.

“Non-Sequential Markov Modeling” (Editor's term) denotes a family of constructions in which Markov machinery is used outside the narrow setting of an ordinary first-order, memoryless, step-by-step chain. In this literature, the departure from standard sequential Markov modeling takes several forms: the source process may itself be non-Markovian but analyzed through a first-order Markov surrogate; latent dynamics may remain Markovian while durations, intervals, endpoints, or observation dynamics violate ordinary hidden-Markov assumptions; or a sequential model may be used primarily as a scoring, segmentation, or feature-extraction device rather than as a token-by-token generator (Farago, 2020, Rezaie et al., 2018, Kawawa-Beaudan et al., 2024, Mews et al., 2024). The common thread is not the abandonment of dependence structure, but its reorganization into a tractable Markov representation adapted to memory, irregular timing, boundary conditioning, partial observation, or non-autoregressive inference.

1. Conceptual scope and departures from the ordinary Markov assumption

The classical first-order Markov model assumes that the next state depends only on the current state. Several of the cited works take this assumption as the reference point and then modify one of its components. One line of work addresses processes for which the Markov property itself fails because transition probabilities depend on earlier history. Another retains Markov latent states but replaces memoryless dwell times by explicit duration models, or augments transitions with state intervals. A third introduces endpoint-conditioned or reciprocal dependence, so that the process is constrained by a destination, an origin, or both. A fourth keeps the hidden process Markovian but makes the observations themselves evolve as a Markov chain conditioned on the hidden state. A fifth uses Markov models as sequence encoders: likelihood profiles, pairwise comparisons, or segment-level mixtures become the primary inferential objects rather than decoded hidden paths (Farago, 2020, Johnson et al., 2012, Rezaie et al., 2018, Kouritzin, 2022, Kawawa-Beaudan et al., 2024).

This diversity implies that “non-sequential” is not a single technical category. In some cases it means genuinely non-Markovian source dynamics; in others it means non-geometric durations, acausal or boundary-conditioned dependence, or inference that is sequence-based but not autoregressive at prediction time. A recurring misconception is that such models discard order. The ensemble HMM framework for behavioral modeling explicitly rejects that interpretation: it is sequence-based but non-autoregressive at inference, because each sequence is converted into a scalar score or a likelihood-feature vector while preserving temporal order in the underlying HMM likelihood computation (Kawawa-Beaudan et al., 2024). A related misconception is that any successful Markov surrogate renders the original process Markovian; the first-order equivalent Markov-chain construction is explicit that this is not the case (Farago, 2020).

2. First-order equivalent Markov chains for non-Markov processes

A canonical formulation of non-Markovianity is history dependence:

$$\Pr(\xi_{t+1}=b \mid \xi_t=a)\neq \Pr(\xi_{t+1}=b \mid \xi_t=a,\xi_{t-1}=a_{t-1},\ldots,\xi_0=a_0).$$

When this occurs, direct use of ordinary Markov-chain analysis is invalid. The proposed remedy in “On Non-Markovian Performance Models” is the “First-Order Equivalent Markov Chain” (1-EMC), built from the one-step transition probabilities of the original process:

$$P_t = [P_t(a,b)]_{a,b\in S} = [\Pr(\xi_{t+1}=b\mid \xi_t=a)]_{a,b\in S}.$$

If these matrices are time-independent, the process is first-order homogeneous with transition matrix

$P = [p(a,b)]_{a,b\in S}.$

The 1-EMC is then the Markov chain generated by these one-step probabilities, independently of earlier path history (Farago, 2020).

The central result is that the 1-EMC and the original process have the same one-dimensional distributions at every time:

$\hat{T}_t = T_t \quad \text{for all } t.$

This yields the trajectory-summation formula

$T_t = T_0 \prod_{i=0}^{t-1} P_i,$

and, in the first-order homogeneous case,

$T_t = T_0 P^t.$

The construction therefore preserves marginal evolution and allows standard Markov-chain formulas to be transferred to a non-Markovian process. The paper further shows that if the process is irreducible and aperiodic, then it is ergodic and has a unique stationary distribution identical to that of the 1-EMC, with convergence relation

$T_t - T = \hat{T}_t - \hat{T}.$

Censoring results also carry over: for a subset $A \subset S$, the censored chain inherits the conditional stationary distribution

$$\pi_A(x)= \begin{cases} \pi(x)/\pi(A), & x\in A,\ 0, & x\notin A. \end{cases}$$

(Farago, 2020)

The limitation is equally important. For a non-Markov process, the probability of a specific trajectory is not necessarily equal to the product of one-step transitions along that trajectory, because history dependence can alter the actual path probability. The 1-EMC is therefore neither a hidden-state construction nor a state-augmentation method, and it does not recover higher-order statistics or pathwise behavior. Its validity is restricted to quantities determined by first-order marginals, stationarity, and some transient distributions. This makes it a principled reduction, not a full equivalence (Farago, 2020).

3. Semi-Markov, duration, and interval representations

A second major departure from ordinary Markov modeling concerns dwell times. In a standard HMM, latent-state durations are geometric, because self-transition probabilities are constant at each time step. The HSMM replaces this memoryless mechanism by explicit durations. In the segment formulation used by the HDP-HSMM, a super-state $z_s$ and its duration $$P_t = [P_t(a,b)]_{a,b\in S} = [\Pr(\xi_{t+1}=b\mid \xi_t=a)]_{a,b\in S}.$$0 generate contiguous runs,

$$P_t = [P_t(a,b)]_{a,b\in S} = [\Pr(\xi_{t+1}=b\mid \xi_t=a)]_{a,b\in S}.$$1

with observations emitted over the whole segment. The explicit-duration Hierarchical Dirichlet Process Hidden semi-Markov Model (HDP-HSMM) combines this with an HDP prior,

$$P_t = [P_t(a,b)]_{a,b\in S} = [\Pr(\xi_{t+1}=b\mid \xi_t=a)]_{a,b\in S}.$$2

and removes self-transitions at the segment level so that duration is handled entirely by $$P_t = [P_t(a,b)]_{a,b\in S} = [\Pr(\xi_{t+1}=b\mid \xi_t=a)]_{a,b\in S}.$$3 (Johnson et al., 2012).

The distinction from sticky HMM variants is precise. A self-transition bias $$P_t = [P_t(a,b)]_{a,b\in S} = [\Pr(\xi_{t+1}=b\mid \xi_t=a)]_{a,b\in S}.$$4 in a sticky HDP-HMM only nudges the geometric duration distribution; it does not replace it with an explicit duration law and does not allow direct state-specific duration priors. The HDP-HSMM was introduced specifically to address this mismatch, and its empirical discussion emphasizes that the semi-Markov formulation can adaptively reduce to an HMM-like explanation when the duration family includes geometric durations and the data support them (Johnson et al., 2012).

Interval modeling extends this logic from “how long a state lasts” to “how long the process waits before the next state begins.” DI-HMM augments HSMM with an interval variable $$P_t = [P_t(a,b)]_{a,b\in S} = [\Pr(\xi_{t+1}=b\mid \xi_t=a)]_{a,b\in S}.$$5 and a Gaussian interval density,

$$P_t = [P_t(a,b)]_{a,b\in S} = [\Pr(\xi_{t+1}=b\mid \xi_t=a)]_{a,b\in S}.$$6

which is multiplied into the HSMM dynamic program. The practical consequence is that sequences with the same state order can still be distinguished by state duration and inter-state gaps. The paper argues that interval modeling should not be implemented by inserting a dummy interval state between real states, because this can degrade discrimination accuracy (Narimatsu et al., 2015).

A related line of work proposes two explicit interval extensions of HSMM: the interval state hidden semi-Markov model (IS-HSMM), which inserts a designated “interval state node” but corrects transition distortion by using second-order structure around the interval state, and the interval length probability hidden semi-Markov model (ILP-HSMM), which adds an interval length probability matrix $$P_t = [P_t(a,b)]_{a,b\in S} = [\Pr(\xi_{t+1}=b\mid \xi_t=a)]_{a,b\in S}.$$7 to the parameter set,

$$P_t = [P_t(a,b)]_{a,b\in S} = [\Pr(\xi_{t+1}=b\mid \xi_t=a)]_{a,b\in S}.$$8

Its forward recursion includes the interval term directly:

$$P_t = [P_t(a,b)]_{a,b\in S} = [\Pr(\xi_{t+1}=b\mid \xi_t=a)]_{a,b\in S}.$$9

Both models were introduced as the first reported extensions of HMM to support state interval representation as well as state duration representation (Narimatsu et al., 2016).

Sequential Bayesian learning for HSMMs addresses the associated computational burden. Because HSMM likelihoods involve summation over durations, their batch cost is substantially higher than the $P = [p(a,b)]_{a,b\in S}.$0 regime typical of HMMs. Particle filtering, particle Gibbs, and SMC$P = [p(a,b)]_{a,b\in S}.$1 were proposed to make HSMM inference sequential, computationally feasible, and exact up to Monte Carlo error. In the VIX application, AR(1)-HSMMs with Negative Binomial or Poisson durations are used for regime switching, model selection, and clustering, with the AR(1) HSMM with Negative Binomial duration reported as the winning model (Aschermayr et al., 2023).

4. Endpoint-conditioned, reciprocal, and observation-side generalizations

Conditionally Markov (CM) and reciprocal models depart from standard sequential Markov structure by conditioning on boundaries. A sequence $P = [p(a,b)]_{a,b\in S}.$2 is $P = [p(a,b)]_{a,b\in S}.$3, $P = [p(a,b)]_{a,b\in S}.$4, if conditioned on endpoint $P = [p(a,b)]_{a,b\in S}.$5, the remaining sequence is Markov:

$P = [p(a,b)]_{a,b\in S}.$6

This yields the two standard classes: $P = [p(a,b)]_{a,b\in S}.$7, conditioned on the first time, and $P = [p(a,b)]_{a,b\in S}.$8, conditioned on the last time. Reciprocal sequences are more general: conditioned on two boundary points, the inside and outside of an interval become independent, with characterization

$P = [p(a,b)]_{a,b\in S}.$9

For nonsingular Gaussian sequences, reciprocality is equivalent to being both $\hat{T}_t = T_t \quad \text{for all } t.$0 and $\hat{T}_t = T_t \quad \text{for all } t.$1 (Rezaie et al., 2018).

The Markov-induced CM model establishes that every reciprocal $\hat{T}_t = T_t \quad \text{for all } t.$2 model can be induced by a Markov model. Starting from a ZMNG Markov process

$\hat{T}_t = T_t \quad \text{for all } t.$3

the conditional distribution $\hat{T}_t = T_t \quad \text{for all } t.$4 is Gaussian with parameters

$\hat{T}_t = T_t \quad \text{for all } t.$5

$\hat{T}_t = T_t \quad \text{for all } t.$6

$\hat{T}_t = T_t \quad \text{for all } t.$7

More generally, a ZMNG sequence is $\hat{T}_t = T_t \quad \text{for all } t.$8 iff it can be represented as

$\hat{T}_t = T_t \quad \text{for all } t.$9

where $T_t = T_0 \prod_{i=0}^{t-1} P_i,$0 is a ZMNG Markov sequence and $T_t = T_0 \prod_{i=0}^{t-1} P_i,$1 is an independent ZMNG vector. This provides a parameter-design method for acausal, endpoint-conditioned models in trajectory modeling, image processing, intelligent systems, and other endpoint-conditioned phenomena (Rezaie et al., 2018).

A distinct generalization is the Markov Observation Model (MOM), which keeps the hidden chain $T_t = T_0 \prod_{i=0}^{t-1} P_i,$2 Markov but makes the observations $T_t = T_0 \prod_{i=0}^{t-1} P_i,$3 themselves a Markov chain with transition probabilities depending on the hidden state:

$T_t = T_0 \prod_{i=0}^{t-1} P_i,$4

The joint pair $T_t = T_0 \prod_{i=0}^{t-1} P_i,$5 is then Markov with transition

$T_t = T_0 \prod_{i=0}^{t-1} P_i,$6

This change propagates through inference: the paper derives a Baum–Welch analog, a believe state or filter recursion, and a Viterbi analog adapted to observation transitions (Kouritzin, 2022).

A further extension treats continuous latent dynamics by discretization into a finite-state Markov-modulated Poisson process. For an Ornstein–Uhlenbeck latent process,

$T_t = T_0 \prod_{i=0}^{t-1} P_i,$7

the state space is partitioned into bins, transition probabilities are approximated from the SDE transition density, and a generator $T_t = T_0 \prod_{i=0}^{t-1} P_i,$8 is obtained through a small-step finite difference. The resulting semigroup $T_t = T_0 \prod_{i=0}^{t-1} P_i,$9 enables recursive latent-Markov likelihood evaluation. This is not acausal, but it shows how hidden continuous structure can be converted into a tractable finite-state Markov representation (Mews et al., 2024).

5. Regime switching, segment mixtures, and sequence-to-feature encodings

Some models depart from ordinary sequential Markov modeling by making regimes or segments, rather than single-step states, the primary inferential objects. Markovian RNN addresses nonstationary time series by combining nonlinear recurrent dynamics with HMM-based switching. A latent regime $T_t = T_0 P^t.$0 evolves as a first-order Markov chain, each regime has its own recurrent parameters, and the hidden state is a soft mixture of regime-specific states:

$T_t = T_0 P^t.$1

The regime beliefs follow the standard HMM filtering recursion,

$T_t = T_0 P^t.$2

with online covariance updates controlling switching sensitivity. Empirically, the model is reported to achieve the best RMSE and MAE in all settings considered, outperforming vanilla RNN and classical Markov-switching baselines on synthetic and currency datasets (Ilhan et al., 2020).

The Infinite Mixture Model of Markov Chains (IMMC) pushes the same idea to segment-level clustering. A sequence is treated as a succession of segments, each generated by one underlying Markov chain, so that a single series may switch multiple times between different chain dynamics. The model combines a sticky HDP-HMM-style backbone over “super states” with internal Markov transitions over sub-states. Its significance lies in separating high-level regime selection from low-level categorical dynamics, thereby supporting segmentation, interpretable pattern discovery, and next-observation prediction. The reported Facebook navigation experiment attributes about $T_t = T_0 P^t.$3 predictive accuracy to IMMC, compared with about $T_t = T_0 P^t.$4 for FMMC and about $T_t = T_0 P^t.$5 for plain Markov models (Reubold et al., 2017).

The behavioral sequence ensemble framework uses HMMs in a more explicitly non-autoregressive way at inference time. For binary classification, singleton HMMs compare $T_t = T_0 P^t.$6 and $T_t = T_0 P^t.$7. The ensemble version trains many class-specific HMMs on random subsets and defines the score

$T_t = T_0 P^t.$8

Alternatively, the full likelihood vector

$T_t = T_0 P^t.$9

is $T_t - T = \hat{T}_t - \hat{T}.$0-normalized and used as a feature embedding. The framework explicitly describes itself as lightweight, interpretable, and efficient, and emphasizes that it is sequence-based but non-autoregressive at inference rather than non-sequential in the sense of discarding order (Kawawa-Beaudan et al., 2024).

These constructions illustrate a broader shift. The Markov structure remains central, but it is often moved from the surface observation sequence to latent regimes, segments, or prototype likelihoods. A plausible implication is that “non-sequential” in this subliterature often means that the final predictive object is not the next token or the full decoded path, but a segment label, regime belief vector, likelihood profile, or classification score.

6. Estimation regimes, applications, and limitations

Non-sequential Markov modeling is also defined by the estimation regime. When full trajectories are unavailable, simple transition counting fails. In discrete stationary finite-state chains with incomplete sequences, aggregate counts, or one observation per individual, Bayesian inference proceeds through data augmentation, Gibbs sampling, or Metropolis–Hastings. For complete data, each row of the transition matrix has a Dirichlet posterior. With incomplete sequences, missing states are imputed conditional on neighboring states; with aggregate data, the posterior is sampled by MH. An adaptive Gaussian-copula proposal was introduced to exploit dependence between rows of the transition matrix and accelerate mixing (Pasanisi et al., 2010).

When the state space is countably infinite or dynamically expanding, the transition matrix becomes an infinite-dimensional object,

$T_t - T = \hat{T}_t - \hat{T}.$1

The Generalized Hierarchical Stick-Breaking prior was introduced to estimate such matrices without hard truncation bias. Global weights are defined by

$T_t - T = \hat{T}_t - \hat{T}.$2

and row distributions are drawn via

$T_t - T = \hat{T}_t - \hat{T}.$3

The reported simulations compare GHSBP against MLE and standard HSBP and state that GHSBP achieves the lowest MAE across many hyperparameter choices (Saha et al., 10 Jul 2025).

Symbolic analysis provides another route when the original data are continuous. The procedure is to discretize $T_t - T = \hat{T}_t - \hat{T}.$4 into symbols $T_t - T = \hat{T}_t - \hat{T}.$5 via a partition $T_t - T = \hat{T}_t - \hat{T}.$6, then estimate the memory depth $T_t - T = \hat{T}_t - \hat{T}.$7 of the symbolic process and fit a $T_t - T = \hat{T}_t - \hat{T}.$8-Markov machine or PFSA. In the thermoacoustic instability study, maximum entropy partitioning with ternary alphabet $T_t - T = \hat{T}_t - \hat{T}.$9 and spectral order estimation yields $A \subset S$0 for stable combustion and $A \subset S$1 for unstable combustion at $A \subset S$2 (Jha, 2021).

Applications also include domains in which memory enters policy rather than probabilistic state evolution. In robot bin picking, ordinary Dex-Net-style policies are Markov because they use only the current RGBD image. Sequential failure objects violate the sufficiency of the current observation: a failed grasp often leaves the bin visually unchanged, so the same failed action is retried. Three non-Markov policies—cluster/cache, circle, and swap—add memory of past failures. In physical experiments on 50 heaps of 12 SFOs, the swap policy increased MPPH over the Dex-Net Markov policy by $A \subset S$3, with MPPH improving from $A \subset S$4 to $A \subset S$5, SFR reduced from $A \subset S$6 to $A \subset S$7, and MSL reduced from $A \subset S$8 to $A \subset S$9 (Sanders et al., 2020).

Psychological testing provides a contrasting application in which the model remains first-order Markov but challenges the assumption that responses are conditionally independent across items. Response categories are treated as states,

$$\pi_A(x)= \begin{cases} \pi(x)/\pi(A), & x\in A,\ 0, & x\notin A. \end{cases}$$0

with transition probabilities

$$\pi_A(x)= \begin{cases} \pi(x)/\pi(A), & x\in A,\ 0, & x\notin A. \end{cases}$$1

This is used to analyze path-dependency, first-order autocorrelation, state-dependency, hysteresis, and order effects in test response dynamics, and to construct likelihood-ratio classifiers between group-specific transition models (Bosco, 2024).

Across these domains, the main limitations are consistent. First-order surrogates preserve marginals but not higher-order dependence (Farago, 2020). Sticky self-transition biases do not replace explicit duration laws (Johnson et al., 2012). Interval-state insertion can distort transition statistics unless corrected (Narimatsu et al., 2016). Sequence-to-feature encodings preserve order only indirectly, through likelihoods under underlying sequential models (Kawawa-Beaudan et al., 2024). Memory-based control policies may depend on heuristics such as masking, tracking, or similarity matching rather than learned latent failure models (Sanders et al., 2020). The field therefore does not present a single alternative to Markov modeling; it presents a set of ways to preserve the tractability of Markov structure while relaxing the assumptions that are most often violated in practice.