Plan Drift Diagnostics

• Plan drift diagnostics are methods to identify deviations in a system’s intended plan, distinguishing permissible adaptations from unsafe changes.
• They integrate frozen specifications, conformance checking, and statistical monitoring to ensure consistency and auditability.
• Applications span LLM-generated trading systems, interactive agents, process mining, generative retrieval, and robotics for enhanced system reliability.

Plan drift diagnostics comprises methods for determining whether a system that is supposed to execute a fixed plan is silently changing what plan it is executing, inferring a different plan from ambiguous instructions, or entering a regime in which previously valid plans, policies, or predictive mappings no longer remain stable. Recent work instantiates this problem in LLM-generated trading systems as plan or strategy drift, in agent interaction as divergence from the user’s intended plan under input faults, in process mining as control-flow drift in execution traces, in generative retrieval as instability of the planner’s candidate set and token evidence, in long-horizon robotics as instruction drift induced by sampling, and in streaming prediction as monitored changes in score vectors, calibration pools, or multimodal data concordance (Cao et al., 6 Apr 2026, Bao et al., 2 Feb 2026, Gallego-Fontenla et al., 2019, Mekonnen et al., 25 Apr 2026, Chen et al., 10 May 2026, Zhang et al., 2020, Opoku et al., 14 Jun 2026, Merkow et al., 2024).

1. Conceptual scope and drift taxonomies

A recurring distinction in the literature is between changes in what a system is intended to do and changes in how it is implemented. In SysTradeBench, drift is explicitly not market drift or model drift in the ML sense; it is plan/strategy drift, meaning that the trading system silently changes what strategy it is implementing rather than merely fixing how it is implemented. The benchmark therefore freezes a Base Strategy Doc and a SHA256-frozen canonical JSON strategy card, and allows only implementation refinements such as NaN handling, corrected stop-loss logic under the same declarative rules, and improved logging (Cao et al., 6 Apr 2026).

Drift-Bench generalizes the problem to interactive agents by treating LLM-agent interaction as cooperative and defining cooperative breakdowns as systematic failures where the agent’s inferred task or sequence of actions no longer matches the user’s intended plan. It organizes these failures into four fault classes: Flaw of Intention, Flaw of Premise, Flaw of Parameter, and Flaw of Expression. This yields a plan-level decomposition into goal, world-state assumptions, concrete arguments, and linguistic realization (Bao et al., 2 Feb 2026).

Process-mining work formalizes an analogous notion at the execution-trace level. In C2D2, a process model is a Petri net $N=(P,T,F)$, and a control-flow drift exists when the activity set changes, $T \neq T^*$, or when the relation set changes, $F \neq F^*$. Here the “plan” is the process model itself, and drift is a structural change in the allowed ordering, branching, or activity inventory of executions (Gallego-Fontenla et al., 2019).

In planning-guided generative retrieval, the plan is neither a symbolic workflow nor a frozen specification but the planner’s top-$n$ candidate set and high-weight planner tokens. The paper on PAG defines plan drift as instability in the planner’s top-$n$ candidate set and high-weight planner tokens under intent-preserving query variation, and plan collapse as the regime in which this instability coincides with a substantial drop in planning-only retrieval effectiveness (Mekonnen et al., 25 Apr 2026).

2. Frozen specifications, conformance checking, and semantic equivalence

Specification-anchored diagnostics attempt to make the intended plan machine-checkable. SysTradeBench does this with Frozen Semantics. If $C_0$ is the original strategy card and $C_k$ the card at iteration $k$, semantic equivalence is defined by hash identity of core_logic, tolerance-bounded equality of parameters, and exact equality of constraints: $$\text{Equiv}(C_0, C_k) := \text{Hash}(C_0.\text{core\_logic}) = \text{Hash}(C_k.\text{core\_logic})$$ together with parameter tolerances $\epsilon_p$ and exact constraint matching. If $T \neq T^*$0, the iteration is marked as semantic drift, the D1 Spec Fidelity score is set to $T \neq T^*$1, and the patch is invalidated. SysTradeBench then adds a second layer: trace-based behavioral drift detection on 50-bar micro-scenarios using

$T \neq T^*$2

Thresholds are explicit: $T \neq T^*$3 is treated as a legitimate refinement, $T \neq T^*$4 is a warning, and $T \neq T^*$5 is suspicious (Cao et al., 6 Apr 2026).

The same section of SysTradeBench ties semantic checks to execution validity gates. Determinism is verified by running the strategy three times with different seeds and hashing trade-level outcomes; anti-leakage is enforced by static AST analysis and a LeakageGuard proxy that raises exceptions on future-bar access. This makes plan drift detection inseparable from reproducibility and temporal integrity, because non-determinism or leakage can mask or mimic drift (Cao et al., 6 Apr 2026).

Conformance-checking approaches in business processes solve a related problem when no frozen JSON specification exists but an executable process model does. C2D2 discovers a reference model from an initial window, slides a window over the time-ordered trace log, computes fitness and precision between the current window and the reference model, and then detects sustained changes via linear regression over recent measurements. A drift is confirmed when the last $T \neq T^*$6 windows are all marked as candidates for either the fitness or the precision series. The method therefore distinguishes novelty-like drift, where new behavior stops replaying well against the old model, from obsolescence-like drift, where old model branches cease to appear in the log (Gallego-Fontenla et al., 2019).

A plausible implication is that frozen-specification methods and conformance-checking methods occupy complementary ends of the same design space. Frozen semantics provide exact rule-level invariants, whereas conformance checking provides execution-level structural invariants when the “plan” is expressed as a process model rather than a declarative contract.

3. Behavioral, interactional, and search-time diagnostics

When plans are inferred interactively rather than read from a frozen specification, diagnostics shift toward clarification behavior, trajectory divergence, and search dynamics. Drift-Bench evaluates this with the RISE protocol. Robustness is measured as

$T \neq T^*$7

Intelligence as clarification gain

$T \neq T^*$8

Safety as Safe Action Rate based on whether high-risk execution occurs only after effective clarification, and Efficiency as Average Interaction Rounds over successful tasks. The benchmark shows that clarification often improves performance in state-oriented environments but can reduce performance in service-oriented environments, a result the paper terms the Clarification Paradox (Bao et al., 2 Feb 2026).

The concrete failure modes are explicitly plan-diagnostic. Drift-Bench documents Over-Speculation (Blind Execution), Contextual Hallucination, and Task Drift, and reports that Safe Action Rate is approximately $T \neq T^*$9 for intention faults but approximately $F \neq F^*$0 for premise and parameter faults. This means that in more than $F \neq F^*$1 of cases with false assumptions or missing parameters, agents execute high-risk actions before clarifying (Bao et al., 2 Feb 2026).

In generative retrieval, planning is embedded in decoding. The PAG stress test defines

$F \neq F^*$2

for the planner’s top-$F \neq F^*$3 candidate stability and

$F \neq F^*$4

for planner-token stability. It then measures PlanSwapDrop by decoding a perturbed query $F \neq F^*$5 with either its own plan or the clean query’s plan. Negative PlanSwapDrop indicates that the perturbed plan is causally harmful. The paper shows that intent-preserving typos can trigger plan collapse, under which the look-ahead bonus provides little useful guidance and decoding effectively reverts toward weaker unguided search (Mekonnen et al., 25 Apr 2026).

Long-horizon robotics offers a different but related view. CAPS argues that instruction drift is fundamentally a systematic sampling error: local greedy decoding collapses into Negative Pivotal Windows, where a locally probable action severs all paths to global success. It defines

$F \neq F^*$6

as a power distribution over trajectories and uses an entropy-based proxy for signal-to-noise ratio,

$F \neq F^*$7

to trigger adaptive MCMC search only when drift risk is high (Chen et al., 10 May 2026).

4. Statistical monitoring and uncertainty quantification

Another research strand treats plan drift diagnostically as a change in monitored statistics rather than as an explicit symbolic mismatch. The score-vector framework monitors the Fisher score

$F \neq F^*$8

under a fitted parametric model. Under stationarity, the mean score is zero; under concept drift, the mean shifts. The paper applies a multivariate exponentially weighted moving average,

$F \neq F^*$9

and a Hotelling statistic

$n$0

to detect changes, then uses Fisher-information-based decoupling to localize which parameters actually shifted (Zhang et al., 2020).

A complementary label-free approach monitors model confidence rather than score vectors. The sequential change-point framework of “Detection of data drift and outliers affecting machine learning model performance over time” uses $n$1 and applies Change Point Models to compare pre- and post-change confidence distributions while controlling false alarms under sequential testing. After an alarm, it mines local density increases in confidence space to surface suspicious observations for diagnosis (Ackerman et al., 2020).

For non-exchangeable streams, DASC defines a transport-based drift score

$n$2

where $n$3 is the empirical current-window distribution and $n$4 is the weighted calibration distribution. It then tracks the effective sample size

$n$5

and proves an approximate coverage bound in which coverage loss decomposes into drift, residual mismatch, and weighted effective sample size: $n$6 This turns drift diagnostics into a calibrated monitoring problem over both representativeness and statistical fragility (Opoku et al., 14 Jun 2026).

MMC+ extends this statistical perspective to medical imaging AI by aggregating Hellinger distances for discrete metadata, Wasserstein distances for continuous variables and multivariate embeddings, and uncertainty bounds from a Flapjack-inspired resampling procedure. Its final score is a weighted z-normalized sum of component distances,

$n$7

where weights are absolute Pearson correlations between component drift and AUROC in the reference period. The framework does not directly predict performance degradation; it serves as an early warning system indicating when systems may deviate from acceptable performance bounds (Merkow et al., 2024).

5. Auditability, visualization, and operational workflows

Several systems place auditability at the center of plan drift diagnostics. SysTradeBench requires each submission to produce a strategy_card.json, trade_log.csv, and audit_log.csv. trade_log.csv must contain entry_datetime, exit_datetime, side, entry_price, exit_price, quantity, pnl, and reason. audit_log.csv must contain datetime, close, strategy indicators, signal, position_state, equity, and constraint_check. Audit completeness is checked through

$n$8

and traceability tests require sampled trades to match corresponding audit-log signals and timestamps. Patches are delivered as evidence-driven change requests with patch budget $n$9 lines and hard enforcement of frozen semantics (Cao et al., 6 Apr 2026).

DriftVis addresses a different operational layer: what analysts do after a detector fires. It combines a distribution-based drift detector with a streaming scatterplot, density-difference views, feature selection, and a prediction-level interface that links drifted regions to base learners and class-specific true positive, false positive, and false negative structure. Its workflow is explicitly analyze–adapt–analyze: distributional drift is localized visually, a new base learner is built or an existing one is replaced, and post-adaptation performance is then compared against the pre-adaptation state (Yang et al., 2020).

MMC+ frames the same operational problem as scalable surveillance in a clinical environment. Reference windows are periods in which model performance is well characterized; detection windows are fixed-duration sliding windows; multimodal distances are normalized using within-reference variability; and high MMC+ values are used to trigger a performance audit rather than continual labeling. This suggests a division of labor in which unsupervised drift monitors screen for changes, while targeted labeling and model review are reserved for periods in which drift exits its normal band (Merkow et al., 2024).

A plausible implication is that plan drift diagnostics increasingly function as governance infrastructure rather than as isolated detectors: they specify contracts, preserve evidence, constrain modifications, and determine when human review is required.

6. Limitations, controversies, and research directions

Current methods expose several systematic blind spots. SysTradeBench notes that semantic equivalence is defined at the level of the strategy card rather than arbitrary natural-language descriptions; behavioral drift is checked only on a finite set of micro-scenarios; some sophisticated leakage modes may slip through; and evidence-driven patching tends to drive code convergence—$n$0 similarity by Iter2 and byte-identical code by Iter3 in one setting—reducing solution diversity even when semantic drift is prevented (Cao et al., 6 Apr 2026).

Drift-Bench shows that more interaction does not monotonically improve alignment. In white-box environments, clarification often raises success, but in black-box service environments clarification can reduce performance, partly through Clarification-Induced Syntactic Collapse and Abandonment Catalyst. The benchmark also acknowledges limits in task coverage, fault coverage, persona realism, and model diversity (Bao et al., 2 Feb 2026).

Theoretical work on adversarial drift detection highlights a deeper controversy: some blind spots are structural rather than empirical. For two-window detectors, the paper constructs window adversarials—genuinely drifting streams that remain undetected because window averages are equal. For two equal sliding windows of length $n$1, all $n$2-periodic functions constitute improper adversarial functions in the limiting analysis. By contrast, the same construction does not apply to block-based detectors (Hinder et al., 2024).

Visual analytics frameworks also delimit their own scope. DriftVis focuses on changes in $n$3, treats drift in the data distribution as the root source of concept drift, and leaves label drift as future work. The method further depends on the quality of incremental clustering and the faithfulness of a 2D projection, even though those are stabilized by explicit constraints (Yang et al., 2020).

Future work is correspondingly moving toward finer-grained invariants, stronger leakage detection, richer behavior-regression suites, environment-aware clarification policies, generalized plan DSLs and equivalence checkers, transport-aware calibration under severe drift, and diagnostics that separate within-plan repair from warranted plan evolution (Cao et al., 6 Apr 2026, Bao et al., 2 Feb 2026, Opoku et al., 14 Jun 2026).

Plan drift diagnostics has therefore become a cross-domain research area centered on one technical question: how to distinguish permissible adaptation from unauthorized or unsafe plan change. The dominant answers now combine specification anchoring, behavioral regression, statistical monitoring, and structured audit trails, but they also show that no single detector is sufficient when plans are ambiguous, multimodal, interactive, or non-stationary.