FIER: Foresight Interactive Experience Replay

• FIER is an interactive imitation-learning mechanism that converts teacher feedback on uncertain novice actions into validation, annotation, or relabeling demonstrations.
• It operates within the ASkDAgger framework, using SAG to trigger queries and PIER to prioritize replay based on derived reward labels.
• By exploiting the informational value of novice proposals, FIER reduces annotation costs and improves adaptation across unseen tasks and domain shifts.

{"query":"arXiv (Luijkx et al., 7 Aug 2025) ASkDAgger Foresight Interactive Experience Replay DAgger HER interactive imitation learning", "max_results": 5} Foresight Interactive Experience Replay (FIER) is an interactive imitation-learning mechanism introduced as a core component of ASkDAgger. Its function is to collect teacher feedback on the novice’s planned action during an uncertainty-triggered query and to convert that interaction into training data rather than discarding the novice proposal. In ASkDAgger, FIER recasts valid and relabeled novice action plans into demonstrations and complements S-Aware Gating (SAG), which determines when to query, and Prioritized Interactive Experience Replay (PIER), which determines what to replay more often during training (Luijkx et al., 7 Aug 2025).

1. Position within ASkDAgger

ASkDAgger consists of three named components: SAG, FIER, and PIER. Within that pipeline, the novice policy $\pi_N$ predicts an action $a_t^k$ from observation $o_t^k$ and goal $g^k$, and also computes uncertainty $u_t^k$. SAG then decides whether to query the teacher based on uncertainty and a dynamic threshold $\gamma$. If a query occurs, FIER is invoked: the teacher sees the novice’s proposed action, validates it, rejects it, and/or relabels it, and the resulting data are stored as a trajectory tuple. At the end of the episode, the collected tuples are added to the dataset, and PIER later reweights replay so that learning emphasizes useful, recent, uncertain, or failure-related demonstrations (Luijkx et al., 7 Aug 2025).

The motivation for FIER is that many active imitation learning and active DAgger-style methods query a teacher but ignore the novice’s proposed action. ASkDAgger treats that proposal as informative because it can reveal what the novice thought was appropriate, whether the plan was actually correct, how much uncertainty the novice had, and whether the novice can already solve the task in some cases. FIER is therefore a query-time mechanism for extracting supervision from the interaction itself rather than treating the query purely as a request for expert replacement.

2. Query-time operation and data conversion

The paper gives Algorithm 2 for FIER. In the notation used there, the inputs are the current observation $o$, novice action $a$, teacher policy $\pi_T$, current trajectory buffer $\boldsymbol{\tau}$, current goal $a_t^k$0, and goal set $a_t^k$1. FIER begins by querying the teacher,

$a_t^k$2

where $a_t^k$3 is teacher feedback and $a_t^k$4 is an optionally relabeled goal (Luijkx et al., 7 Aug 2025).

Three outcomes are possible. In the validation case, if the teacher judges the novice plan valid, then

$a_t^k$5

and the novice action is accepted as a demonstration under the original goal:

$a_t^k$6

In the rejection or annotation case, the teacher instead provides the correct action

$a_t^k$7

and an annotation tuple is added:

$a_t^k$8

In the relabeling case, if the teacher also determines that the novice action achieves another valid goal,

$a_t^k$9

then the same novice plan is additionally stored as a relabeled tuple:

$o_t^k$0

This organization makes FIER more than a teacher-correction interface. It is a mechanism for turning one query into one or more reusable demonstrations, depending on whether the novice plan is correct, incorrect but useful for another goal, or incorrect and requiring expert replacement.

3. Trajectory representation, labels, and relabeling rule

ASkDAgger defines trajectories as

$o_t^k$1

with reward or feedback

$o_t^k$2

These labels distinguish validated novice demonstrations, expert annotations, and relabeled novice demonstrations, and the paper states that this reward label is later used by PIER to determine replay priority (Luijkx et al., 7 Aug 2025).

Tuple type	Stored form	Label
Validation demonstration	$o_t^k$3	$o_t^k$4
Annotation demonstration	$o_t^k$5	$o_t^k$6
Relabeling demonstration	$o_t^k$7	$o_t^k$8

The relabeling condition is stated explicitly in terms of state sets. Let $o_t^k$9 be the current state, $g^k$0 the action, $g^k$1 the next state, $g^k$2 the original goal, $g^k$3 the set of states satisfying goal $g^k$4, and

$g^k$5

If a transition

$g^k$6

reaches a state that satisfies some other goal $g^k$7, then it can be relabeled as success,

$g^k$8

provided

$g^k$9

The paper describes this rule as directly inspired by Hindsight Experience Replay (HER). The critical difference in emphasis is that FIER operates through teacher-mediated validation and relabeling of a novice’s proposed action inside an interactive imitation-learning loop, rather than through autonomous hindsight relabeling of past reinforcement-learning transitions.

4. Functional relation to SAG and PIER

FIER does not determine when to ask for help. SAG performs that role by setting the gating threshold $u_t^k$0 to maintain a target level of sensitivity, specificity, or minimum system success rate. The novice is queried if

$u_t^k$1

or randomly with probability $u_t^k$2. FIER is therefore conditional on a prior decision that the state is uncertain or risky enough to justify intervention (Luijkx et al., 7 Aug 2025).

PIER operates after FIER has created a mixture of validation tuples, annotation tuples, and relabeled tuples. The paper defines replay probability as

$u_t^k$3

with priority

$u_t^k$4

where

$u_t^k$5

Here $u_t^k$6 is uncertainty, $u_t^k$7 is an age or recency signal, $u_t^k$8 is the tradeoff between uncertainty and age, $u_t^k$9 controls how sharply priorities vary, $\gamma$0 is the prioritization exponent, and $\gamma$1 is the importance-sampling correction exponent.

In this division of labor, SAG controls query frequency, FIER controls the value extracted from each query, and PIER exploits the resulting dataset composition during replay. A plausible implication is that the term “interactive experience replay” in FIER refers less to a standalone replay algorithm than to the production of replayable interactive tuples whose semantics are later used by PIER.

5. Reported effects in simulation, domain shift, and real-world evaluation

Across CLIPort tasks, ASkDAgger is reported to require significantly fewer annotation demonstrations than baselines such as SafeDAgger, ThriftyDAgger, and Active DAgger. The paper attributes this reduction largely to FIER because many queried novice actions become validation demonstrations or relabeled demonstrations rather than requiring full manual annotation (Luijkx et al., 7 Aug 2025).

The strongest evidence reported for FIER is in unseen-object and unseen-scenario CLIPort results. The paper states that ASkDAgger outperforms the active baselines on unseen tasks because failures are sometimes relabeled to valid goals and these additional demonstrations expand the dataset into novel object-goal combinations. An ablation in which relabeling was removed is reported to confirm that the performance gain on unseen tasks came from FIER’s relabeling mechanism. With relabeling, ASkDAgger achieved an average evaluation reward improvement of 62% across the four unseen tasks.

In domain-shift experiments, ASkDAgger adapted slightly faster after shifts than variants without FIER. The paper further states that ASkDAgger without FIER required more annotation demonstrations, that FIER contributed to faster initial adaptation after domain shift, and that, combined with PIER, it helped maintain higher performance and lower annotation cost.

In the real-world engine assembly task, FIER enabled validation and relabeling in practice. The dataset eventually consisted mostly of validation demonstrations, relabeling was shown to be feasible beyond simulation, and the system maintained high success while reducing annotation burden.

6. Conceptual interpretation and relation to adjacent methods

FIER is presented as a way to transform an active query from a simple request for expert correction into a richer interaction with three possible outcomes: validation of the novice plan, relabeling of that plan under a different valid goal, or expert annotation when neither validation nor relabeling is appropriate. The paper’s short conceptual summary is that if SAG asks when to interrupt the novice, FIER determines what useful data can be extracted from that interruption (Luijkx et al., 7 Aug 2025).

A common misunderstanding is to treat FIER as equivalent to teacher override. That is not how it is defined. The novice’s proposed action is central: it is explicitly shown to the teacher, may itself become the stored demonstration, and may be repurposed through goal relabeling. Another misunderstanding is to conflate FIER with foresight-style model-based relabeling in sparse-reward reinforcement learning. The closest comparison in the provided literature is the Foresight Relabeling (FR) mechanism in MRHER, which also uses future-oriented reasoning rather than purely retrospective hindsight, but does so in a model-based relay hindsight framework for sequential object manipulation, with relay subtasks, self-guided exploration, a later-state rollout start $\gamma$2, and a threshold-based goal-reset rule intended to mitigate the Identical Non-Negative Reward problem (Huang et al., 2023).

On that basis, FIER and MRHER’s FR are conceptually aligned in using future-oriented information instead of relying only on retrospective relabeling, but they are not the same method. FIER is an interactive imitation-learning mechanism centered on teacher feedback about novice plans and the conversion of those plans into demonstrations. MRHER’s FR is a model-based goal relabeling rule for sparse-reward reinforcement learning. This suggests that the shared term “foresight” denotes a family resemblance in orientation toward future consequences, not identity of algorithmic design.

7. Significance within interactive imitation learning

Within ASkDAgger, FIER addresses a specific inefficiency in active imitation learning: a teacher query is expensive, but the novice’s planned action often contains supervisory value even when it is uncertain. FIER formalizes three ways to preserve that value—validation, relabeling, and annotation—using a trajectory representation whose labels are subsequently exploitable by prioritized replay (Luijkx et al., 7 Aug 2025).

The method is especially relevant in settings where failures may still correspond to valid behavior under another goal. The paper highlights this in language-conditioned manipulation tasks such as CLIPort, where a failed action for the commanded goal may nonetheless be valid for a different object or distractor. In that regime, relabeling expands the support of the training data beyond the exact commanded goals. This is presented not merely as an efficiency device for reducing annotation burden, but also as a mechanism for improving generalization and adaptation under changing domains.

Taken together, these properties situate FIER as the data-formation component of ASkDAgger: SAG regulates intervention, FIER structures the supervision extracted from intervention, and PIER governs how that supervision is reused during learning.