Flexible Goal Accuracy (FGA) in DST

• Flexible Goal Accuracy (FGA) is an evaluation metric for Dialogue State Tracking that measures both cumulative and turn-level performances.
• It assigns partial credit for locally correct predictions when errors are inherited from earlier turns, using an exponential decay controlled by a parameter λ.
• Empirical evaluations show that FGA correlates better with human judgment and yields more nuanced insights on model performance than Joint Goal Accuracy.

Flexible Goal Accuracy (FGA) is an evaluation metric for Dialogue State Tracking (DST) proposed as a more balanced alternative to Joint Goal Accuracy (JGA). In DST, the dialogue state or belief state for a given turn contains all the intents shown by the user till that turn, so an early misprediction can make later cumulative states appear incorrect even when the model is locally correct at subsequent turns. FGA addresses this by assigning partial credit to mispredictions that are locally correct, namely cases where the root cause of the error is an earlier turn rather than the current prediction. It is presented as a generalized version of JGA that flexibly incorporates both cumulative and turn-level performance and is intended to provide a better insight than existing metrics for DST model evaluation (Dey et al., 2022).

1. Conceptual basis and motivation

JGA is the primary evaluation metric traditionally used for DST and is defined as the fraction of turns where the ground-truth dialogue state exactly matches the prediction. Because the belief state is cumulative, a single mistake—especially early in the conversation—can lead to zero JGA for all subsequent turns unless corrected. The paper identifies this all-or-nothing behavior as a source of harshness: JGA can underestimate a model’s true ability by giving no credit to locally correct predictions if the cumulative state does not match (Dey et al., 2022).

The motivation for FGA is tied to two specific limitations of JGA. First, JGA does not capture cases where the model makes correct predictions at the current turn, but past errors have spoiled the cumulative state. Second, an improvement in JGA can sometimes decrease the performance of turn-level or non-cumulative belief state prediction due to inconsistency in annotations. The paper therefore argues that using JGA as the only metric for model selection may not be ideal for all scenarios, particularly on datasets such as MultiWOZ where annotation inconsistencies can propagate harsh penalties through later turns (Dey et al., 2022).

Within this framing, FGA occupies an intermediate position between strict cumulative evaluation and more locally sensitive evaluation. This suggests an attempt to preserve the diagnostic value of joint-state matching while reducing the distortion introduced by inherited errors.

2. Formal definition and scoring procedure

The formalization of FGA begins with cumulative and turn-level belief states. Let $B_t$ be the ground-truth cumulative belief state at turn $t$, and let $B'_t$ be the predicted belief state at turn $t$. Let $T_t = B_t \setminus B_{t-1}$ denote the turn-level belief state, that is, intents newly expressed at turn $t$, and let $T'_t = B'_t \setminus B'_{t-1}$ be the model’s predicted turn-level state (Dey et al., 2022).

For each turn $t$, the scoring rule is defined by a three-way distinction. If $B_t = B'_t$, the prediction is correct and the assigned score is 1. Otherwise, the method checks whether the mismatch is due to the current turn or inherited from an earlier turn. If $T'_t \not\subseteq B_t$ or $t$0, the mistake is local; this is designated a type 1 error, the score is 0, and the turn is recorded as the last error turn $t$1. If neither of those local-failure conditions holds, the mismatch is inherited from an earlier turn; this is a type 2 error, and the method assigns a partial score (Dey et al., 2022).

Case	Condition	Score
Exact match	$t$2	$t$3
Type 1 error	$t$4 or $t$5	$t$6
Type 2 error	Locally correct but cumulative mismatch	$t$7

For type 2 errors, the partial score is

$t$8

where $t$9 controls the penalty decay rate. The FGA for a conversation is then the mean of the turn scores. In effect, the metric does not merely ask whether the cumulative state is correct; it asks whether the current turn should continue to be penalized for a prior error (Dey et al., 2022).

3. Relation to JGA, turn-level accuracy, and metric strictness

A central property of FGA is that the hyperparameter $B'_t$0 controls strictness. At $B'_t$1, FGA reduces to JGA, which is the strictest setting and gives no partial credit. As $B'_t$2, FGA approximates turn-level accuracy, which is the most relaxed setting and gives full credit for locally correct predictions even when the cumulative state is still mismatched. Partial credit increases the farther away the current turn is from the turn where an error first happened, effectively implementing exponential forgetting of old errors (Dey et al., 2022).

This parametrization is meant to balance two failure modes in DST evaluation. JGA is characterized as too strict because inherited errors can dominate the score across many subsequent turns. Slot accuracy or average goal accuracy, by contrast, may overestimate performance because they can remain high even when critical state errors are present. FGA is positioned between these extremes: it preserves penalties for local failures while discounting penalties that arise only because an earlier cumulative state was incorrect (Dey et al., 2022).

A common misconception is to treat FGA as merely a relaxed version of JGA. The formal definition is narrower and more structured than that. Partial credit is not assigned to any incorrect turn; it is assigned specifically when the current turn is locally correct and the remaining mismatch is inherited. Another misconception is to equate FGA with turn-level accuracy. The $B'_t$3 limit approximates turn-level accuracy, but for finite $B'_t$4 the metric remains explicitly sensitive to cumulative error history (Dey et al., 2022).

4. Illustrative behavior in multi-turn dialogues

The paper’s illustrative example uses a six-turn conversation with an error at turn 2. From turn 3 onward, the model makes locally correct turn-level predictions, but the cumulative state remains wrong due to the error in turn 2. Under JGA, only turns with no error in cumulative belief state are credited, so the score is $B'_t$5, or $B'_t$6. Under FGA with $B'_t$7, partial credits are given to turns 4 and 6, producing individual turn scores $B'_t$8 and an overall FGA of $B'_t$9 (Dey et al., 2022).

The example is used to show that FGA does not eliminate penalties for inherited errors, but it attenuates them when subsequent turn-level behavior is correct. The same example is also used to contrast FGA with slot or average accuracy, which may severely overestimate performance; the summary notes that slot accuracy can remain above $t$0 even with critical errors. The intended interpretation is that FGA avoids the overestimation associated with overly local metrics while avoiding the underestimation associated with purely cumulative metrics (Dey et al., 2022).

This example also clarifies the operational meaning of “locally correct.” A turn can be locally correct even when the cumulative belief state is globally wrong. FGA formalizes that distinction and assigns a score accordingly.

5. Empirical behavior and discrimination power

The empirical analysis reported for FGA evaluates four DST models—TRADE, Hi-DST, SOM-DST, and Trippy—on MultiWOZ 2.1. The relevant comparison table includes the number of exact matches, denoted $t$1 JGA, turn-level matches, denoted $t$2, and several FGA scores with varying $t$3. The reported findings are qualitative but specific: Trippy achieves the highest JGA but not the highest turn-level matches, Hi-DST has fewer exact matches but higher turn-level accuracy than Trippy, and SOM-DST shows balanced performance. FGA is described as distinguishing among models more smoothly by rewarding sustained turn-level correctness even when earlier errors continue to affect the cumulative state (Dey et al., 2022).

The paper also reports a human evaluation in which human raters judged 100 conversations for model satisfaction, phrased as whether the model tracked user intent through the conversation. Pearson correlation was then calculated between human satisfaction and the evaluation metrics. The reported correlation is $t$4 for JGA and $t$5 for FGA with $t$6. The stated conclusion is that FGA better correlates with human judgment than JGA (Dey et al., 2022).

These results support two claims made in the paper. First, FGA is presented as a better discriminator of DST model performance. Second, it is presented as a fairer proxy for practical model satisfaction because it incorporates both cumulative correctness and turn-level recovery behavior. A plausible implication is that FGA is particularly useful when model rankings depend on whether inherited errors should dominate evaluation.

6. Terminological scope and acronym ambiguity

Within the supplied literature, the named metric “Flexible Goal Accuracy” is introduced in the DST context by “Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performances” (Dey et al., 2022). Other papers use the same acronym or closely related phrasing in different senses.

In “GUIDE-CoT: Goal-driven and User-Informed Dynamic Estimation for Pedestrian Trajectory using Chain-of-Thought,” the summary explicitly states that the paper does not define a stand-alone “Flexible Goal Accuracy (FGA)” metric by name; instead, Final Displacement Error (FDE) serves as the primary metric for goal accuracy, while flexibility is validated qualitatively through controllable trajectory generation (2503.06832). In “DB-FGA-Net: Dual Backbone Frequency Gated Attention Network for Multi-Class Classification with Grad-CAM Interpretability,” FGA denotes “Frequency-Gated Attention,” not Flexible Goal Accuracy; the summary further states that “Flexible Goal Accuracy” there refers not to the attention block itself but to the network’s ability to achieve high accuracy across different classification granularities (Shreya et al., 23 Oct 2025).

The phrase also appears descriptively in “Fast or Better? Balancing Accuracy and Cost in Retrieval-Augmented Generation with Flexible User Control,” where the summary characterizes the user-controllable RAG framework as enabling flexible goal accuracy through a control parameter $t$7 that interpolates between cost-optimized and reliability-optimized classifiers (Su et al., 17 Feb 2025). By contrast, “Generic and Universal Parallel Matrix Summation with a Flexible Compression Goal for Xilinx FPGAs” discusses a flexible compression target or goal, and the supplied summary explicitly notes that “Flexible Goal Accuracy” as a formal term is not defined in that paper (Preußer, 2018).

This suggests that, across recent arXiv literature, “FGA” is not a stable cross-domain term. In the strict technical sense documented here, Flexible Goal Accuracy refers to the DST evaluation metric proposed in 2022; elsewhere, the acronym either collides with unrelated terminology or the phrase is used informally to describe adaptability rather than a specific metric.