PortBench: LLM-Driven Portfolio Benchmark

• PortBench is a correlation-aware benchmark that evaluates LLMs on end-to-end multi-asset portfolio management using a 10-year dataset and dual evaluation layers.
• It employs a dynamic five-stage pipeline alongside a static QA layer to assess market interpretation, risk management, and diversified asset allocation across various stress scenarios.
• The design emphasizes challenges like error propagation, investor-profile alignment, and effective exploitation of asset correlations, guiding future advances in financial LLM applications.

Searching arXiv for "PortBench" and closely related uses to ground the article in the relevant papers. PortBench is a correlation-aware, full-pipeline benchmark for LLM-driven portfolio management that evaluates whether models can perform end-to-end multi-asset portfolio management under realistic market conditions, investor constraints, and stress regimes (Zhao et al., 27 May 2026). In the cited formulation, PortBench combines a 10-year, 183-instrument market dataset across six asset classes with two complementary evaluation layers: a static QA dataset of 6,269 automatically generated, correlation-based questions across seven task templates, and a dynamic five-stage allocation pipeline that mirrors the portfolio management decision cycle (Zhao et al., 27 May 2026). The benchmark was introduced to address two gaps identified in prior financial LLM evaluation: the absence of correlation awareness and the absence of full decision-pipeline evaluation (Zhao et al., 27 May 2026). In adjacent literatures, the term has also appeared in broader “PortBench-like” senses, including systematic x86 instruction port-usage characterization built on nanoBench (Abel et al., 2019) and a proposed open repository for performance portability benchmarking (Marowka, 2023). However, the canonical use of “PortBench” as a named benchmark refers to the portfolio-management benchmark introduced in 2026 (Zhao et al., 27 May 2026).

1. Definition and scope

PortBench is designed to answer a specific question: if a modern LLM is given realistic multi-asset market data and asked to run an entire portfolio management process end-to-end, how well it performs, especially when correlations change and markets are stressed (Zhao et al., 27 May 2026). Its stated structure consists of a 10-year, 183-instrument market dataset across six heterogeneous asset classes, a dual evaluation layer, three historical stress regimes, and three investor risk profiles (Zhao et al., 27 May 2026).

The asset universe comprises 183 instruments in six classes: 126 equity tickers, 16 commodity tickers, 15 bond series, 12 cryptocurrency tickers, 10 real-estate series, and 4 cash-equivalent series, over January 2015 to December 2025 (Zhao et al., 27 May 2026). The underlying data include daily prices, returns, macro indicators such as rates, inflation, spreads, and VIX, as well as news and filings (Zhao et al., 27 May 2026).

A central design premise is that intra-class correlations are high, while inter-class correlations are generally low; accordingly, diversification requires going across classes rather than merely across tickers (Zhao et al., 27 May 2026). This correlation structure is not only visible to the model but is directly encoded into the scoring system (Zhao et al., 27 May 2026). This distinguishes PortBench from benchmarks that treat assets independently or reduce portfolio management to isolated forecasting or QA tasks (Zhao et al., 27 May 2026).

2. Motivation and benchmark construction

The benchmark identifies two major deficiencies in earlier financial LLM benchmarks. First, prior benchmarks often ignore cross-asset correlation structures and therefore fail to distinguish genuinely diversified portfolios from concentrated portfolios (Zhao et al., 27 May 2026). Second, prior benchmarks often evaluate static tasks rather than the full sequential portfolio management process in which interpretation, signal formation, optimization, execution, and monitoring interact over time (Zhao et al., 27 May 2026).

PortBench addresses these deficiencies through a dual-layer design. The static layer is a QA dataset with 6,269 automatically generated questions across seven templates (Zhao et al., 27 May 2026). The dynamic layer replays the historical dataset over time and asks the model to operate within a five-stage portfolio management workflow on monthly rebalance dates for the main experiments, and across all dates in stress windows for stress evaluation (Zhao et al., 27 May 2026).

The benchmark further incorporates three historical stress regimes—2015 China Shock, 2020 COVID Crash, and 2022 Crypto Collapse—and three investor profiles—conservative, balanced, and aggressive—expressed as natural-language constraints in prompts (Zhao et al., 27 May 2026). This design allows evaluation not only of nominal portfolio performance, but also of robustness and investor-profile alignment (Zhao et al., 27 May 2026).

A plausible implication is that PortBench is structured less as a pure prediction benchmark than as a controlled environment for studying whether LLMs can translate market interpretation into coherent, risk-constrained allocation behavior. That implication is supported by the explicit separation of static formula-driven tasks from the stateful pipeline evaluation (Zhao et al., 27 May 2026).

3. Static QA layer

The static QA layer contains 6,269 correlation-based questions generated automatically from historical data via analytical formulas and optimizations, without manual labeling (Zhao et al., 27 May 2026). The seven templates span both analytical and judgmental tasks.

Template	Task	Primary focus
T1	Return direction prediction	Single-asset trend/regime understanding
T2	Risk assessment (VaR)	Numerical risk estimation
T3	Position sizing (Kelly-style)	Risk-to-size conversion
T4	Pairwise minimum-variance allocation	Covariance with return floor
T5	Multi-asset maximum-Sharpe allocation	Markowitz-style optimization
T6	Rebalancing decision with trade specification	Portfolio constraints and trade planning
T7	Regime detection and allocation	Regime-aware multi-asset reallocation

For T2, the benchmark defines one-day VaR at level $\alpha$ using historical simulation from the trailing window:

$$\text{VaR}_\alpha = \text{quantile}(r_{1:252},\, 1-\alpha)$$

(Zhao et al., 27 May 2026)

For T3, position sizing uses the simplified fixed-fraction formula

$$f^* = \min\!\left(1.0,\; \frac{\delta_\text{max}}{|\text{VaR}_{99\%}|}\right)$$

where $\delta_\text{max}$ is the maximum drawdown allowed by the question (Zhao et al., 27 May 2026).

For T4, the unconstrained minimum-variance weight for the first asset is

$$w_1^\text{mv} = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}},\quad w_2 = 1 - w_1$$

with an additional return-floor condition that shifts weights if the unconstrained portfolio return falls below the required floor (Zhao et al., 27 May 2026).

For T5, the optimization objective is

$$\max_{\mathbf{w}} \frac{\mathbf{w}^\top \boldsymbol{\mu} - r_f}{\sqrt{\mathbf{w}^\top \boldsymbol{\Sigma}\mathbf{w}}} \quad\text{s.t.}\quad \sum_i w_i = 1,\; w_i \ge 0$$

with $r_f = 4\%$ per annum (Zhao et al., 27 May 2026).

T6 evaluates whether a model decides to rebalance when the maximum deviation exceeds 5%, and, if so, whether it specifies the largest trade correctly in asset, size, and direction (Zhao et al., 27 May 2026). T7 requires regime classification into bull, bear, sideways, or crisis, followed by recommended increase, decrease, or hold decisions for exposures across equities, bonds, commodities, real estate, crypto, and cash according to stylized “flight-to-quality” rules (Zhao et al., 27 May 2026).

The static layer is therefore explicitly heterogeneous: some tasks are formula-driven, while others depend on open-ended financial reasoning. This distinction becomes central to the benchmark’s empirical findings (Zhao et al., 27 May 2026).

4. Dynamic five-stage pipeline

The dynamic layer replays the base dataset over time and evaluates a stateful five-stage workflow (Zhao et al., 27 May 2026). On each rebalance date, the LLM is called in the first three stages, while the last two are deterministic (Zhao et al., 27 May 2026).

The five stages are as follows.

1. S1 – Market Interpretation: The input is a MarketSnapshot containing 60-day history of all assets, intra- and inter-class correlation matrices, macro indicators, news, and the current portfolio. The output is a continuous sentiment score $v_i \in [-1,1]$ per asset (Zhao et al., 27 May 2026). The stage score is

$$\sigma_1 = 1 - \frac{1}{2n}\sum_{i=1}^n |v_i - v_i^*|$$

where $v_i^*$ is the ground-truth view scaled from realized forward returns (Zhao et al., 27 May 2026).

2. S2 – Signal Generation: The input is the S1 scores, and the output is a discrete buy, hold, or sell signal per asset using thresholds $$\text{VaR}_\alpha = \text{quantile}(r_{1:252},\, 1-\alpha)$$0 for buy and $$\text{VaR}_\alpha = \text{quantile}(r_{1:252},\, 1-\alpha)$$1 for sell (Zhao et al., 27 May 2026). The score is

$$\text{VaR}_\alpha = \text{quantile}(r_{1:252},\, 1-\alpha)$$2

(Zhao et al., 27 May 2026).

3. S3 – Weight Optimization: The input consists of buy signals from S2, class-level correlation summaries, and investor profile constraints. The output is a weight vector $$\text{VaR}_\alpha = \text{quantile}(r_{1:252},\, 1-\alpha)$$3 across buy-signal assets, with zero weight on all others (Zhao et al., 27 May 2026). The ground-truth portfolio is the ex-post signal-constrained maximum-Sharpe portfolio

$$\text{VaR}_\alpha = \text{quantile}(r_{1:252},\, 1-\alpha)$$4

subject to

$$\text{VaR}_\alpha = \text{quantile}(r_{1:252},\, 1-\alpha)$$5

where $$\text{VaR}_\alpha = \text{quantile}(r_{1:252},\, 1-\alpha)$$6 is the set of buy-signal assets, $$\text{VaR}_\alpha = \text{quantile}(r_{1:252},\, 1-\alpha)$$7 comes from future returns, and $$\text{VaR}_\alpha = \text{quantile}(r_{1:252},\, 1-\alpha)$$8 from the lookback window (Zhao et al., 27 May 2026).

4. S4 – Execution Simulation: There is no new LLM call; S4 deterministically applies S3 weights with fixed transaction costs and compares actual turnover $$\text{VaR}_\alpha = \text{quantile}(r_{1:252},\, 1-\alpha)$$9 to oracle turnover $$f^* = \min\!\left(1.0,\; \frac{\delta_\text{max}}{|\text{VaR}_{99\%}|}\right)$$0 (Zhao et al., 27 May 2026). The score is

$$f^* = \min\!\left(1.0,\; \frac{\delta_\text{max}}{|\text{VaR}_{99\%}|}\right)$$1

(Zhao et al., 27 May 2026).

5. S5 – Risk Monitoring: This stage is also deterministic. Given executed weights and return history, the sandbox computes VaR, maximum drawdown, and weight drift, then decides whether to rebalance for risk (Zhao et al., 27 May 2026). The score is

$$f^* = \min\!\left(1.0,\; \frac{\delta_\text{max}}{|\text{VaR}_{99\%}|}\right)$$2

with

$$f^* = \min\!\left(1.0,\; \frac{\delta_\text{max}}{|\text{VaR}_{99\%}|}\right)$$3

(Zhao et al., 27 May 2026).

Because the pipeline is stateful, the outputs from one date update the portfolio and become inputs at the next date, yielding a NAV trajectory over time and per-stage scores (Zhao et al., 27 May 2026). This statefulness is essential to the benchmark’s emphasis on error propagation (Zhao et al., 27 May 2026).

5. Correlation-aware scoring and CEPS

PortBench’s distinctive technical contribution is its explicit correlation-aware scoring in S3 together with CEPS, the Cross-stage Error Propagation Score (Zhao et al., 27 May 2026).

The S3 score is

$$f^* = \min\!\left(1.0,\; \frac{\delta_\text{max}}{|\text{VaR}_{99\%}|}\right)$$4

where the accuracy term is

$$f^* = \min\!\left(1.0,\; \frac{\delta_\text{max}}{|\text{VaR}_{99\%}|}\right)$$5

and the correlation term is

$$f^* = \min\!\left(1.0,\; \frac{\delta_\text{max}}{|\text{VaR}_{99\%}|}\right)$$6

(Zhao et al., 27 May 2026).

For intra-class concentration, the score defines class weight and mean within-class off-diagonal correlation as

$$f^* = \min\!\left(1.0,\; \frac{\delta_\text{max}}{|\text{VaR}_{99\%}|}\right)$$7

and then uses

$$f^* = \min\!\left(1.0,\; \frac{\delta_\text{max}}{|\text{VaR}_{99\%}|}\right)$$8

so concentration in a highly correlated class reduces the score (Zhao et al., 27 May 2026).

For inter-class hedging, the benchmark computes weighted average cross-class correlation

$$f^* = \min\!\left(1.0,\; \frac{\delta_\text{max}}{|\text{VaR}_{99\%}|}\right)$$9

and maps it to

$\delta_\text{max}$0

so negative or low cross-class correlations increase the score (Zhao et al., 27 May 2026).

This decomposition formally distinguishes portfolios diversified across low- or negatively correlated classes from portfolios that distribute weight across many instruments within a highly correlated class (Zhao et al., 27 May 2026). That distinction is one of the benchmark’s central claims of novelty (Zhao et al., 27 May 2026).

CEPS measures whether performance degrades across adjacent stages rather than merely averaging stage quality. Let $\delta_\text{max}$1 be the score of stage $\delta_\text{max}$2. Then

$\delta_\text{max}$3

and

$\delta_\text{max}$4

so that

$\delta_\text{max}$5

(Zhao et al., 27 May 2026).

The intended interpretation is explicit in the formulation: uniformly mediocre pipelines can score higher than pipelines that begin strongly but collapse in later stages (Zhao et al., 27 May 2026). This suggests that PortBench treats portfolio management competence as a sequential reliability problem rather than a collection of disconnected subtasks.

6. Stress regimes, investor profiles, and empirical results

PortBench evaluates robustness under three historical stress windows: the 2015 China Shock from August 2015 to February 2016, the 2020 COVID Crash from February 2020 to May 2020, and the 2022 Crypto Collapse from May 2022 to December 2022 (Zhao et al., 27 May 2026). The benchmark notes that inter-class correlations rise in stress, reducing diversification benefits (Zhao et al., 27 May 2026).

Three investor profiles are implemented as natural-language prompt constraints (Zhao et al., 27 May 2026). Conservative requires maximum equity plus crypto weight $\delta_\text{max}$6, minimum bond plus cash weight $\delta_\text{max}$7, maximum drawdown tolerance $\delta_\text{max}$8, and daily VaR limit $\delta_\text{max}$9 (Zhao et al., 27 May 2026). Balanced uses $$w_1^\text{mv} = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}},\quad w_2 = 1 - w_1$$0, $$w_1^\text{mv} = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}},\quad w_2 = 1 - w_1$$1, $$w_1^\text{mv} = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}},\quad w_2 = 1 - w_1$$2, and $$w_1^\text{mv} = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}},\quad w_2 = 1 - w_1$$3 (Zhao et al., 27 May 2026). Aggressive uses $$w_1^\text{mv} = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}},\quad w_2 = 1 - w_1$$4, $$w_1^\text{mv} = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}},\quad w_2 = 1 - w_1$$5, $$w_1^\text{mv} = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}},\quad w_2 = 1 - w_1$$6, and $$w_1^\text{mv} = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}},\quad w_2 = 1 - w_1$$7 (Zhao et al., 27 May 2026).

The benchmark also defines a Profile Alignment Score and an Adaptation Score:

$$w_1^\text{mv} = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}},\quad w_2 = 1 - w_1$$8

where higher AdaptScore indicates more differentiated behavior across profiles (Zhao et al., 27 May 2026). A separate stress gate is passed when the worst maximum drawdown across the three stress regimes remains within the profile’s drawdown tolerance (Zhao et al., 27 May 2026).

Ten frontier LLMs were evaluated: DeepSeek-V4-Flash, DeepSeek-V4-Pro, Qwen3.7-Max, Qwen3.6-Plus, Qwen3.6-35B-A3B, GLM-5.1, Doubao-Seed-2.0-Lite, Doubao-Seed-2.0-Pro, Hunyuan3-Preview, and Kimi-K2.6 (Zhao et al., 27 May 2026). The protocol used temperature $$w_1^\text{mv} = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}},\quad w_2 = 1 - w_1$$9 and maximum output length 4096 tokens (Zhao et al., 27 May 2026).

The main reported findings are unfavorable to current models. On static QA, formula-driven templates T3, T4, and T5 often yield very high accuracy, while T1 does not exceed 0.52 accuracy for any model, and T6 shows a wide spread from 0.468 to 0.882 (Zhao et al., 27 May 2026). The paper reports an average formula-versus-judgment gap of approximately 0.21 (Zhao et al., 27 May 2026). When covariance matrices are removed from T4 and T5 prompts, most models do not degrade and often improve; the paper gives the example that Kimi-K2.6 T5 rises from 0.28 to 0.71 under the restricted condition, while only DeepSeek models use the covariance matrix productively (Zhao et al., 27 May 2026).

In the pipeline setting, S1 is consistently high, S2 moderate, S3 compressed, S4 the worst stage, and S5 highly variable (Zhao et al., 27 May 2026). The paper reports that GLM-5.1 ranks seventh on QA mean but first on CEPS for the balanced profile, Kimi-K2.6 ranks tenth on QA but third on CEPS, Doubao-Lite ranks fourth on QA but last on CEPS, and the Spearman correlation between QA rank and CEPS rank is $$\max_{\mathbf{w}} \frac{\mathbf{w}^\top \boldsymbol{\mu} - r_f}{\sqrt{\mathbf{w}^\top \boldsymbol{\Sigma}\mathbf{w}}} \quad\text{s.t.}\quad \sum_i w_i = 1,\; w_i \ge 0$$0 (Zhao et al., 27 May 2026). The paper therefore argues that static QA is a poor predictor of end-to-end portfolio competence (Zhao et al., 27 May 2026).

Against financial baselines, the paper evaluates Equal-Weight, 60/40, Risk Parity, Covariance Risk Parity, and Minimum Variance (Zhao et al., 27 May 2026). Across 30 model-profile combinations, 27 of 30 fail to beat Equal-Weight on Sharpe (Zhao et al., 27 May 2026). Only one model per profile beats Equal-Weight on Sharpe: GLM-5.1 for conservative, Qwen3.6-Plus for balanced, and DeepSeek-V4-Pro for aggressive (Zhao et al., 27 May 2026). Only Qwen3.6-Plus in the balanced profile both beats Equal-Weight on Sharpe and passes all stress gates (Zhao et al., 27 May 2026).

The benchmark attributes these failures to several recurring modes: universal under-trading or execution collapse, effective disregard of correlation matrices by most models, stress fragility despite procedural compliance, dissociation between normal and stress performance, and inconsistent profile adaptation (Zhao et al., 27 May 2026). The paper’s summary conclusion is that LLMs exhibit strong local competence on formula execution and static QA but poor global competence in constructing diversified, robust portfolios under stress and constraints (Zhao et al., 27 May 2026).

7. Related and ambiguous uses of the name

The name “PortBench” has adjacent meanings in other technical contexts, and distinguishing them is useful for bibliographic precision.

In x86 microarchitecture research, the 2019 nanoBench paper describes a first case study devoted to determining the latency, throughput, and port usage of more than 13,000 instruction variants on recent x86 processors, with results published at uops.info (Abel et al., 2019). The accompanying discussion characterizes this as “PortBench-style work,” although the paper itself does not brand the framework as PortBench (Abel et al., 2019). In that literature, the term denotes systematic instruction-characterization and port-mapping infrastructure rather than portfolio management (Abel et al., 2019).

In HPC performance-portability research, a 2023 proposal outlines an open repository of performance portability of applications, benchmarks, and models, effectively a “PortBench-like” repository governed by standardized metrics, run-rules, and metadata requirements (Marowka, 2023). That work recommends the arithmetic-mean performance-portability metric

$$\max_{\mathbf{w}} \frac{\mathbf{w}^\top \boldsymbol{\mu} - r_f}{\sqrt{\mathbf{w}^\top \boldsymbol{\Sigma}\mathbf{w}}} \quad\text{s.t.}\quad \sum_i w_i = 1,\; w_i \ge 0$$1

as a primary score for repository standardization (Marowka, 2023). Here again, the topic is performance portability across heterogeneous systems rather than finance (Marowka, 2023).

These neighboring uses do not alter the primary meaning of PortBench as a named benchmark in current arXiv usage. Rather, they show that the compound “PortBench” or “PortBench-like” has been independently associated with port-usage benchmarking, performance-portability benchmarking, and portfolio-management benchmarking. In contemporary citation practice, the unqualified title “PortBench” refers to the correlation-aware benchmark for LLM-driven portfolio management introduced in 2026 (Zhao et al., 27 May 2026).