Teacher-Forced Accuracy (TFA)

• Teacher-Forced Accuracy (TFA) is a metric that quantifies a model's next-step prediction accuracy using teacher-forced (reference) states.
• It contrasts local step accuracy with closed-loop rollout measures, where errors can compound when models operate autonomously.
• Empirical studies show that high TFA does not always ensure reliable autonomous performance, especially in complex weighted graph algorithms.

Searching arXiv for the specified paper and closely related evaluation context. Teacher-Forced Accuracy (TFA) is a local, next-step prediction metric: it measures how often a model chooses the correct next action when it is always fed the correct current state, rather than the states resulting from its own past actions. In "Closed-Loop Graph Algorithm Execution with Small LLMs: Step Accuracy and Rollout Reliability" (Podstawski, 23 Jun 2026), this idea appears under the name step accuracy. Within that framework, TFA is contrasted with exact rollout accuracy and several other closed-loop, trajectory-level diagnostics, and the central conclusion is that strong next-step prediction does not necessarily translate into reliable autonomous execution.

1. Concept and terminology

In the framework of closed-loop graph algorithm execution, TFA is the teacher-forced regime for evaluating a model’s next action. The model is queried only on correct reference states, and the evaluation asks whether it can identify the correct local transition when the input state is known to be correct. The paper states: “In teacher-forced step evaluation, the model is queried only on reference states $s_t^\star$. This measures whether it can identify the correct local transition when the input state is known to be correct.” By contrast, “In autonomous rollout evaluation, execution starts from $s_0^\star$, but every subsequent state is produced by applying the model's own action” (Podstawski, 23 Jun 2026).

This establishes TFA as a metric of local competence rather than autonomous reliability. It is teacher-forced because the model is not allowed to condition on the consequences of its own earlier predictions. A plausible implication is that TFA is most informative when the primary question concerns next-step decision quality on valid states produced by a reference algorithm.

2. Formal setup

The paper considers classical graph algorithms executed step-by-step via a LLM policy. Let $G = (V, E, w)$ be a graph, with optional weights $w$. For a given algorithm $A$, the symbolic state after $t$ transitions is $s_t$, an action at step $t$ is a discrete symbol $a_t$, and state transitions are defined by the deterministic transition function

$s_{t+1} = T_A(G, s_t, a_t).$

A termination predicate $s_0^\star$0 decides when the algorithm is complete. A deterministic reference implementation defines, for each reachable state, a canonical next action $s_0^\star$1, and hence a reference trajectory

$s_0^\star$2

The LLM is treated as a policy,

$s_0^\star$3

where $s_0^\star$4 is a textual serialization of the graph and the current state, and $s_0^\star$5 are model parameters. The model outputs a one-line action, which is parsed and applied via $s_0^\star$6 (Podstawski, 23 Jun 2026).

Within this formalization, TFA is defined over the reference trajectory rather than over model-induced trajectories. This is the decisive distinction between local evaluation on oracle states and closed-loop execution over self-induced states.

3. Metric definition and relation to rollout measures

The paper defines step accuracy as follows: “Step accuracy is the fraction of reference states for which the parsed model action exactly matches the canonical next action. It is teacher-forced: every query uses a correct reference state even if the model would have failed at an earlier step.” Under teacher forcing, for each graph instance $s_0^\star$7 with reference trajectory $s_0^\star$8, let

$s_0^\star$9

A direct formalization of the verbal definition is

$G = (V, E, w)$0

The same evaluation framework defines several closed-loop metrics. Exact rollout accuracy is the fraction of test graphs for which the final answer equals the canonical reference answer after autonomous execution from the initialized state. Constraint validity is a weaker task-specific check: for traversal it verifies a source-rooted, duplicate-free sequence containing only reachable vertices; for Prim and Borůvka it verifies that selected edges form a spanning tree; for Dijkstra it verifies a complete non-negative distance labeling satisfying edge-wise inequalities; and for coloring it verifies a complete conflict-free assignment. Soft score measures partial agreement with the reference answer: it is order-position accuracy for BFS and DFS, node-color accuracy for coloring, edge F1 for Prim and Borůvka, and exact node-distance accuracy for Dijkstra. The rollout diagnostics include the step-to-trajectory gap, defined as the difference between step and exact rollout accuracy; prefix survival, the fraction of test rollouts whose first $G = (V, E, w)$1 actions match the reference trajectory; prefix AUC, the mean of this survival curve across eligible positions; the first-error index, the first position at which an erroneous rollout deviates from the reference; and intervention rollout corrections, the mean number of model predictions per graph that would require replacement when the model is queried at every reference-corrected state and the system advances using the oracle action (Podstawski, 23 Jun 2026).

These definitions make the scope of TFA precise. TFA answers whether the model selects the canonical next action on correct reference states. Rollout measures answer whether the model can preserve correctness across an entire execution.

4. Empirical relation between TFA and rollout reliability

The empirical results show that adaptation can produce reliable policies for structural procedures such as traversal and coloring, while weighted algorithms remain substantially more sensitive to error accumulation. For traversal tasks, the step-to-trajectory gap is small. For Llama-3.2-1B, BFS, Step is 99.90%, Traj. is 97.33%, Gap is 2.57, Prefix AUC is 97.78%, and Corr. is 0.03 corrections per rollout. For Qwen2.5-1.5B, BFS, Step is 96.99%, Traj. is 95.33%, Gap is 1.66, AUC is 94.30, and Corr. is 0.77. Here, high TFA does largely translate into high rollout reliability.

For coloring, the gap is moderate. For Llama, Coloring, Step is 98.77%, Traj. is 80.00%, Gap is 18.77, AUC is 92.29, Err. is 23.57, and Corr. is 0.31. For Qwen, Coloring, Step is 99.24%, Traj. is 86.00%, Gap is 13.24, AUC is 93.85, and Corr. is 0.19. Even with approximately $G = (V, E, w)$2 teacher-forced step accuracy, a nontrivial fraction of graphs fail to achieve exact rollout correctness.

The largest discrepancies appear in weighted procedures. For Borůvka (MST), Llama has Step 92.70%, Traj. 21.33%, Gap 71.37, AUC 73.10, Corr. 1.79, Valid 54.67%, and Soft 95.01; Qwen has Step 92.92%, Traj. 25.33%, Gap 67.58, AUC 70.43, and Corr. 1.73. For Dijkstra (shortest paths), Llama has Step 83.54%, Traj. 13.33%, Gap 70.21, AUC 15.47, Err. 4.46, Corr. 4.19, Valid 13.33%, and Soft 86.17%; Qwen has Step 86.68%, Traj. 17.33%, Gap 69.35, AUC 18.28, Err. 4.29, and Corr. 3.39. For Prim (MST), Llama has Step 93.90%, Traj. 41.33%, Gap 52.56, AUC 61.36, Corr. 1.49, Valid 62.67%, and Soft 95.97%; Qwen has Step 87.74%, Traj. 14.67%, Gap 73.08, AUC 42.43, and Corr. 3.00 (Podstawski, 23 Jun 2026).

These results show that high TFA can coexist with poor closed-loop reliability. This is most pronounced when locally plausible actions alter the global state in ways that affect every later decision.

5. Why teacher-forced accuracy and autonomous execution diverge

The discussion identifies several reasons why TFA and rollout performance answer different questions. First, “Step accuracy measures performance on states produced by the reference algorithm. A rollout additionally tests whether the model remains correct after repeated application and whether it can handle states induced by its own earlier decisions.” Second, teacher forcing evaluates only on-policy states from the reference algorithm, whereas rollouts expose the model to off-trajectory states after an error. Third, “Some decline is also expected without feedback effects because a trajectory succeeds only if every required action is correct.” This is the compounding-error effect. Fourth, “Step accuracy micro-averages decisions over all reference states, whereas exact rollout accuracy counts a graph as correct only when its complete execution matches the reference.” Fifth, weighted algorithms require repeated comparison of numerical edge or distance values while preserving globally coupled state: “A locally plausible edge selection or relaxation can alter the frontier, component structure, or distance map used by every later decision.” Finally, “The difference between Prim and Borůvka further indicates that ‘weighted’ is not a complete explanation. The action representation, state serialization, tie-breaking policy, and way an error changes subsequent state all influence rollout reliability” (Podstawski, 23 Jun 2026).

Together, these observations explain why a local teacher-forced metric can overstate operational reliability. They also clarify why the same nominal step accuracy can correspond to different rollout behaviors across algorithms.

6. Methodological role and scope

Within this framework, TFA is useful as a local competence metric. It measures how often the model picks the correct next action when given the right state, it is directly tied to training objectives such as token-level cross-entropy, and it is diagnostic of representation and decision quality at each step. At the same time, it is misleading as a reliability metric for autonomous execution. The methodological recommendation is explicit: “evaluating an algorithmic LLM only on isolated reference states can substantially overstate its operational reliability. When predictions change future inputs, complete closed-loop rollouts should be treated as a primary evaluation unit” (Podstawski, 23 Jun 2026).

A robust evaluation protocol therefore includes TFA or step accuracy, closed-loop rollout accuracy, structural validity checks, soft or partial metrics, and rollout diagnostics such as prefix survival, prefix AUC, first-error index, and correction counts. This suggests a broader principle: when model outputs affect future inputs, evaluation on teacher-forced states alone is insufficient. The same logic applies to agents, tool-using systems, planners, and code executors, where autonomous behavior depends not only on local next-step prediction but also on stability under repeated self-conditioned execution.