From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets

Abstract: Evaluating whether LLM agents can profit in capital markets is increasingly framed as end-to-end trading: place an agent in a historical market, let it trade, and measure portfolio returns. This setup is vulnerable to two evaluation failures. First, long backtests often overlap with the knowledge cutoffs of frontier LLMs, allowing memorized tickers, dates, prices, and market narratives to substitute for investment reasoning. Second, raw returns are a noisy proxy for stock-selection ability, since positive performance may come from market beta, style exposure, or favorable regimes rather than genuine alpha. We introduce KTD-Fin (Knowing-To-Doing Financial Benchmark), an end-to-end stock-market trading benchmark that addresses both issues. KTD-Fin uses a data-side masking protocol to anonymize key identifiers and calendar information consistently across prompts and tools, separating historical market memory from investment decision-making. It also incorporates a Barra-style performance attribution framework that decomposes portfolio returns into market, style, and stock-selection alpha components. Across ten frontier LLM agents evaluated on the Chinese CSI300 over a 2024--2026 window, masking substantially changes agent rationales, pushing them towards anonymized factor-based reasoning. Attribution analysis further shows that LLM agents' cumulative returns under leakage-controlled evaluation are largely explained by passive market and style exposure, with limited evidence of persistent stock-selection alpha. These findings suggest that financial LLM benchmarks should evaluate not only whether an agent makes money, but also whether the source of returns reflects transferable investment skill. We release KTD-Fin as a reproducible template for leakage-controlled and attribution-aware evaluation of LLM trading agents.

• The paper introduces KTD-Fin, a leakage-controlled benchmark that combines ticker and date masking with Barra-style factor attribution to separate memory-based trading from genuine stock-selection skill.
• Across ten frontier LLM agents, nine produced negative factor-adjusted selection alpha, while cumulative returns were often driven by market and style exposures rather than transferable investment ability.
• The benchmark shows that data-side anonymization outperforms prompt instructions for preventing pretraining-memory leakage, while supervised Qlib factor models achieved stronger risk-adjusted results and lower drawdowns than LLM agents.

End-to-end trading benchmarks have become the standard instrument for asking whether LLM agents can profit in capital markets, but the paper argues that this framing suffers from two evaluation failures: pretraining-memory leakage through real tickers and dates, and the conflation of cumulative return with stock-selection skill. KTD-Fin (Knowing-To-Doing Financial Benchmark) (2605.28359) addresses both within a single protocol, instantiated on the Chinese A-share CSI300 universe over a 2024–2026 window, and its headline finding is stark: under leakage-controlled evaluation, ten frontier LLM agents' returns are largely explained by passive market and style exposure, with nine of ten posting negative factor-adjusted selection alpha.

Benchmark design

KTD-Fin is built on Qlib conventions and simulates A-share trading at realistic granularity: long-only positions from CNY 1,000,000, T+1 sell availability, 5 bps buy / 15 bps sell costs, next-day-open execution, and board-differentiated price-limit filtering (±10% main board, ±20% ChiNext/STAR, ±30% BSE). The observation channel is deliberately price-only—OHLCV histories and derived technical features—with no news, filings, or fundamentals. This design choice is diagnostic: any rationale that names brand, sector, or fundamentals must originate from pretraining priors rather than observation.

Three decision modes decompose trading competence along orthogonal dimensions. Memory-only provides only stock identifiers and account state, so any non-trivial trade must come from priors; fixed-candidate supplies a curated OHLCV+factor table (the typical prior-work setting); open-research exposes six read-only retrieval tools in a ReAct-style research-then-submission loop, with tool arguments un-masked before querying the data layer and re-masked before returning to the model.

Four-level masking protocol

The leakage control is data-side rather than prompt-side. Four levels—bright (real ticker, real date), stock-blind, date-blind, and blinded (both anonymized)—anonymize tickers, calendar dates, and tool-return timestamps consistently across prompts and tools. Aliases are stable within an episode but shuffled across episodes; all numeric features are computed on original series so only identifying handles change.

Two results establish that masking matters. First, in memory-only mode the anchor model (Step-3.5-Flash) trades actively under bright ($-0.16\%$ return) but holds cash at exactly $0.00\%$ under blinded—the disappearing trades are precisely those attributable to identity priors. Second, verbatim rationale analysis shows brand-driven narratives under bright ("Kweichow Moutai, liquor leader"; "the Oct-18 limit-up move" for SMIC, a date unrecoverable from the price channel) collapsing to pure factor ranks under blinded. The intermediate conditions localize the effect: date-blind retains sector priors while stock-blind removes them, indicating the ticker handle carries the dominant pretraining prior.

An independent de-anonymization probe certifies the mask: ten frontier LLMs acting as attackers on 200 masked probes achieve top-1 ticker recovery of at most $3.0\%$ (random baseline $0.3\%$) and strict joint success (ticker top-5 plus date within ±7 trading days) of at most $1.5\%$. Blinded is therefore treated as a near-clean reasoning condition. Notably, prompt-level "do not use external memory" instructions do not reproduce these ablations, consistent with prior evidence from static forecasting settings.

Barra-style attribution

To separate skill from factor luck, each day's cross-section is fit by weighted least squares regressing stock returns on a unit-exposure common component and nine VIF-screened style factors (momentum, volatility, illiquidity, reversal, skewness, volume–price interaction), with exposures computed strictly before day $t$. Aggregating residuals under the agent's weights yields factor-adjusted selection alpha $\alpha^p_t = \sum_i w^p_{i,t-1}\varepsilon_{i,t}$, with Common + Style + Selection Alpha = Portfolio by construction.

The attribution concentrates nearly all cross-agent variance in selection alpha: market contributions span 13pp and style contributions 17pp across agents, while selection alpha spans $+0.2\%$ to $-77.8\%$—a 78pp range larger than the other two combined. Every agent, including the worst performers, harvests $+11\%$ to $0.00\%$0 from systematic factor tilts. The implied ranking diverges sharply from cumulative return: the total-return leaders (Qwen3.6-Plus at $0.00\%$1, GPT-5.5, Doubao-Seed-2.0) post negative selection alpha ($0.00\%$2, $0.00\%$3, $0.00\%$4 respectively), and only Claude Opus 4.7 reaches par at $0.00\%$5. The bottom four agents share the same broad factor tilts as the top group; their failure is within-factor selection, not stylistic misallocation. A benchmark scoring solely on cumulative return would systematically over-credit agents that converge on the prevailing factor portfolio.

Model	Total ret.	Sel. α	α-rank
Claude Opus 4.7	$0.00\%$6	$0.00\%$7	1
Qwen3.6-Plus	$0.00\%$8	$0.00\%$9	2
Doubao-Seed-2.0	$3.0\%$0	$3.0\%$1	3
Kimi-K2.6	$3.0\%$2	$3.0\%$3	10

Cross-model results and baselines

Under blinded × open-research on the contiguous 2024-01-01 to 2026-04-10 window (548 trading days), five of ten LLMs beat CSI300 buy-and-hold ($3.0\%$4), led by Qwen3.6-Plus at $3.0\%$5 (Sharpe 1.13) and GPT-5.5 at $3.0\%$6. Against eighteen trained Qlib factor models sharing the same Alpha9 features (trained 2008–2022, validated 2023), however, the supervised baselines dominate: SFM achieves $3.0\%$7 with Sharpe 2.02 and maximum drawdown of only $3.0\%$8, versus drawdowns of $3.0\%$9 to $0.3\%$0 for LLM agents. LLM behavior also differs structurally: cohort-level exposure analysis shows ML books form classical low-volatility momentum portfolios (net short realized volatility, amount variability, intraday momentum; 15 of 18 models share sign across all nine factors), while LLM books flip those tilts, loading most heavily on intraday momentum ($0.3\%$1) and high-volatility, recently moving names. The two cohorts agree only on long-horizon momentum, the 52-week-high anchor, and illiquidity aversion.

Reliability metrics reveal further weaknesses: expected calibration error ranges from 0.246 to 0.425 across agents, and turnover varies from 14× to over 100× annualized, with Gemini-3.1-Pro and GLM-5.1 churning their books without commensurate return.

Limitations and open questions

The authors concede several constraints. Coverage is limited to the CSI300 A-share universe with price-only observations; US/Hong Kong equities and multimodal inputs (news, filings, macro series) are out of scope, though the masking-plus-attribution pair is claimed to transfer to such settings. Only the anchor model sweeps the full mask × mode × window grid; the other nine frontier LLMs run solely under blinded × open-research, so mask sensitivity on non-anchor models rests on fragmentary replication over three additional models on short windows. Attribution depends on the assumption that the residual term isolates selection skill—a proxy the paper treats as "closer" rather than exact—and the intercept is a common-return component rather than an externally observed index return. Whether the near-zero alpha of the best agent reflects a genuine capability ceiling or insufficient agentic scaffolding remains open, as does whether the pattern holds outside a bull-leaning window where passive factor harvesting flatters all agents.

Conclusion

KTD-Fin contributes a reproducible template for leakage-controlled, attribution-aware evaluation of LLM trading agents. Its two central empirical claims—that data-side masking eliminates identifier-based memory leakage where prompt instructions fail, and that cumulative return substantially overstates stock-selection ability for frontier LLMs—are supported by an independently certified mask and a decomposition showing negative selection alpha for nine of ten agents. The benchmark reframes the evaluation question from whether an agent makes money to whether the source of returns reflects transferable investment skill.