Asset allocation using a Markov process of clustered efficient frontier coefficients states

Abstract: We propose a novel asset allocation model using a Markov process of states defined by clustered efficient frontier coefficients. While most research in Markov models of the market characterize regimes using return and volatility, we instead propose characterizing these states using efficient frontiers, which provide more information on the interactions of underlying assets that comprise the market. Efficient frontiers can be decomposed to their functional form, a square-root second-order polynomial defined by three coefficients, to provide a dimensionality reduction of the return vector and covariance matrix. Each month, the proposed model hierarchically clusters the monthly coefficients data up to the current month, to characterize the market states, then defines a Markov process on the sequence of states. To incorporate these states into portfolio optimization, for each state, we calculate the tangency portfolio using only return data in that state. We then take the expectation of these weights for each state, weighted by the probability of transitioning from the current state to each state. To empirically validate our proposed model, we employ three sets of assets that span the market, and show that our proposed model significantly outperforms benchmark portfolios.

• The paper presents a regime-switching asset allocation model that clusters efficient frontier coefficients to define observable market states.
• It employs hierarchical clustering with dynamic time warping and a first-order Markov process to construct state-specific tangency portfolios, demonstrating improved performance.
• Empirical results reveal enhanced Sharpe ratios and reduced drawdowns, underscoring the model’s practical impact on dynamic portfolio management.

Asset Allocation via Markov Process of Clustered Efficient Frontier States

Introduction

This paper introduces a regime-switching asset allocation model that leverages a novel characterization of market states: clusters of efficient frontier (EF) coefficients derived from Modern Portfolio Theory (MPT). Rather than relying on conventional Markov regime definitions based on asset returns or volatilities, the approach defines observable market regimes via unsupervised clustering of monthly EF coefficients, followed by modeling their evolution as a first-order Markov process. Each state is mapped to a tangency portfolio constructed using returns within that state; allocations are then formed by computing the expected tangency portfolio weights, weighted by the Markov transition probabilities. The methodology is empirically validated across three asset universes, with significant outperformance relative to standard benchmarks.

Methodological Framework

Efficient Frontier Coefficient Representation

The core of the approach is the dimensionality reduction of the return–covariance space using three EF coefficients: the return and standard deviation at minimum variance (TMVP, OMVP), and the EF curvature parameter (u), as derived via Merton’s analytic solution to the efficient frontier. These interpretable coefficients succinctly capture market structure, encoding joint distribution shifts that are elusive to first-moment or second-moment statistics alone. They allow for regime characterization at the level of the global risk-reward trade-off, rather than at the asset or sector level.

Temporal Clustering and State Identification

Given the time series of monthly EF coefficient vectors, states are identified by applying hierarchical clustering with the dynamic time warping (DTW) distance, enabling robust handling of the shifts and distortions common in financial data. Hierarchical clustering is preferred over k-means or spectral clustering for its ability to capture the hierarchical, non-spherical structure of financial regimes and its interpretability in terms of market taxonomy. The DTW metric accommodates temporal structure, enhancing regime detection fidelity.

Markov Modeling and Portfolio Mapping

States, once assigned, are used to estimate the empirical Markov transition matrix. At each month, tangency portfolios are computed using returns from periods belonging to the current state, under both fully-invested and market-neutral constraints. The final allocation at time $t$ is the expectation of all state-conditioned tangency portfolios, weighted by the predicted transition probabilities from the present state. This yields dynamic allocations that directly account for anticipated regime changes without resorting to hidden state layers as in HMMs.

Online Updating and Practical Constraints

An expanding online backtest framework is used to ensure real-time information flow and robust estimation. To maintain realism, portfolio leverage is capped at 1.5x, rebalancing is performed daily, and a 1% per-transaction fee is introduced. The method’s hyperparameter—the number of clusters—is selected via dendrogram analysis, and sensitivity analysis shows stable performance across 3 to 5 clusters.

Empirical Results

Performance Across Asset Universes

Out-of-sample backtests are performed for three diversified universes: (1) US mutual funds spanning growth/value and market cap segments (GVMC); (2) S&P 9-sector ETFs; (3) five major developed equity indices. Across splits starting from both 2000 and 2008, the proposed Markov Markowitz model consistently achieves higher Sharpe ratios, annualized returns, and superior return-to-max-drawdown profiles compared to equal-weight, S&P 500, and static tangency portfolio benchmarks. Notably, for US sectors, the Sharpe increases to 0.73 (2008–2022), with max drawdown sharply reduced versus baselines.

Alpha regressions against all benchmarks show statistically significant outperformance (p < 0.05 in US asset universes), robust to changes in test period start date and cluster count. The only exception is developed markets (p ≈ 0.13–0.20), where lack of total return data limits statistical power.

Sensitivity to Universe Size and Covariance Estimation

When combining all three universes into a single asset list, performance deteriorates with Sharpe ratios decreasing and underperformance versus benchmarks. The root cause is the high ratio of asset dimensionality to monthly sample size, which impairs covariance estimation accuracy for EF coefficient computation. This limitation highlights the necessity for restricted universe sizes or higher-frequency (intraday) sampling to preserve estimation reliability.

Regime Interpretability and Market Insights

A major advantage of observable regime clustering is transparency: the assignment and characteristics of market states can be directly analyzed. Cluster assignments are shown to correspond well with known financial shocks and recessions—bearish clusters subsume crisis periods, bullish clusters partition into states with different transition risks to downturn. Heatmaps of state-conditioned tangency portfolios reveal distinct allocation regimes, with market-neutral stances in bearish regimes and differentiated sector exposures in bullish regimes. Markov transition matrices uncover persistent regime dynamics: bearish states tend to recur with high probability, especially in US-centric universes, offering new insights for risk management.

Theoretical and Practical Implications

This model introduces several theoretical innovations:

• Non-Return Based Regime Definition: By clustering EF coefficients, the approach discards the limiting assumption that market states are adequately described by return/volatility alone, instead modeling higher-level portfolio opportunity sets.
• Observable, Interpretable States: Unlike HMMs, states and transitions are interpretable and disconnected from the limitations of maximum likelihood fitting sensitive to minor misclassifications.
• Unified Regime-Switching Portfolio Construction: The direct mapping of state to tangency portfolio, with probability aggregation, allows both stable estimation (using full data within each state) and dynamic responsiveness to expected regime evolution.

From a practical asset management standpoint, the model demonstrates applicability to liquid, moderate-cardinality universes—but underscores the need for refined covariance estimation methods (e.g., Ledoit-Wolf shrinkage for small samples) or access to higher-frequency data for larger universes.

Future Directions

Potential research extensions include:

• Application to additional asset classes (commodities, FX, fixed income) and incorporation of alternative EF coefficient parameterizations.
• Investigation of intraday EF coefficient clustering for higher frequency (e.g., daily) regime adaptations.
• Integration with advanced covariance estimators or Bayesian shrinkage to support expansion to larger universes with fewer estimation errors.
• Systematic comparison of clustered regime transitions with annotated macroeconomic/financial crisis timelines for causal insight.

Conclusion

The Markov process of clustered efficient frontier coefficient states constitutes a substantive development in dynamic asset allocation. By embedding regime identification in the structure of the efficient frontier, the approach enables both interpretable and empirically robust allocations that adapt dynamically to market state evolution. The model is especially well-suited for allocation tasks over moderate-size universes with reliable return and covariance estimation, and offers new avenues for regime-aware portfolio construction and financial market interpretability.