---
title: 'ASkDAgger: Active Skill-Level Data Aggregation'
url: https://www.emergentmind.com/topics/active-skill-level-data-aggregation-askdagger
type: topic
---

# ASkDAgger: Active Skill-Level Data Aggregation

Searching arXiv for the specified papers and closely related work on ASkDAgger.
Active Skill-level Data Aggregation (ASkDAgger) is an interactive imitation learning framework for robotics that reduces human teaching effort by querying a teacher selectively while reusing the novice policy’s own skill-level plans as training signal. In its canonical formulation, a novice policy $\pi_\mathrm{N}$ maps observations and goals to mid-/high-level actions, an uncertainty operator $U$ estimates confidence in the proposed plan, and teacher feedback is used not only to correct errors but also to validate or relabel novice plans as demonstrations [2508.05310]. The framework is organized around three components—S-Aware Gating (SAG), Foresight Interactive Experience Replay (FIER), and Prioritized Interactive Experience Replay (PIER)—and is designed for settings where actions are parameterized skills rather than low-level torques [2508.05310]. A distinct but related use of the ASkDAgger lens appears in humanoid whole-body control, where rollout-based aggregation of skill-induced command trajectories is used to maintain a shared low-level controller under distribution shift during long-horizon skill composition [2602.13850].

## 1. Formal problem setting and core idea

ASkDAgger is proposed for interactive imitation learning in robotics, with the explicit goal of reducing human teaching effort while improving safety and generalization [2508.05310]. The novice policy is defined as
\[
\pi_\mathrm{N}: \mathcal{O} \times \mathcal{G} \rightarrow \mathcal{A},
\]
where at episode $k$ and time $t$ it receives observation $o_t^k \in \mathcal{O}$ and goal $g^k \in \mathcal{G}$ and outputs action $a_t^k \in \mathcal{A}$ [2508.05310]. The teacher, modeled as a human or simulation oracle with policy $\pi_\mathrm{T}$, can provide annotation demonstrations, validate a novice’s proposed action as acceptable, or relabel a failed novice action as successful for another goal [2508.05310].

The environment is described as a dynamical system with hidden state $s_t^k \in \mathcal{S}$ and observations given by $o_t^k = O(s_t^k)$, while goals define success sets $\mathcal{S}^g \subset \mathcal{S}$ [2508.05310]. A transition $(s_t^k, a_t^k, s_{t+1}^k, g^k)$ is successful if $s_{t+1}^k \in \mathcal{S}^{g^k}$ [2508.05310]. The novice learns from a demonstration dataset $\mathcal{D} = \{\tau^k\}$ of trajectories
\[
\tau^k = \{(o_t^k, a_t^k, g^k, r_t^k)\}_{t=0}^{T_k},
\]
where the scalar teacher reward records the outcome of a query:
\[
r_t^k =
\begin{cases}
1 & \text{if the teacher validates the novice action},\\
-1 & \text{if the teacher rejects the novice plan and provides an annotation},\\
0 & \text{otherwise}.
\end{cases}
\]
The learning objective is behavioral cloning on the aggregated dataset,
\[
\pi^* = \arg\min_{\pi \in \Pi} \mathcal{L}(\pi, \mathcal{D}),
\]
with a supervised loss between novice and teacher actions over $\mathcal{D}$ [2508.05310].

The distinctive protocol is the novice’s ability to present a plan together with uncertainty: “I plan to do this, but I am uncertain” [2508.05310]. Formally, ASkDAgger assumes an uncertainty operator
\[
U : \Pi \times \mathcal{O} \times \mathcal{G} \rightarrow [0,1], \quad
u_t^k = U(\pi_\mathrm{N}, o_t^k, g^k),
\]
where larger $u_t^k$ indicates lower confidence in the novice’s planned skill-level action [2508.05310]. This shifts the role of a query from merely requesting a corrective label to presenting a candidate action that can itself become part of the training corpus if validated or relabeled.

## 2. Skill-level abstraction and relation to DAgger

ASkDAgger operates at the level of parameterized skills rather than low-level motor commands [2508.05310]. A skill-level action is described as a vector of parameters for a primitive such as grasping, walking, pushing, door opening, or insertion [2508.05310]. In the CLIPort setting used in the paper, a pick-and-place primitive is parameterized by pick pixel location and orientation together with place pixel location and orientation [2508.05310]. Because one decision corresponds to a complete skill execution rather than a single control cycle, queries are less frequent and more interpretable to a human teacher [2508.05310].

This abstraction distinguishes ASkDAgger from step-level active imitation learning. Classical DAgger executes the novice and queries the expert for labels at all visited states, addressing covariate shift by collecting expert supervision on the novice’s own state distribution [2508.05310]. Active DAgger variants reduce the label burden by querying only in uncertain, risky, or novel states, but at queried states they typically keep only the expert action and discard the novice proposal [2508.05310]. ASkDAgger differs in three stated respects: plan-aware data usage, statistically adaptive gating, and priority-based updates [2508.05310].

The framework is therefore an active DAgger variant at skill level with performance-aware gating and replay, and with explicit use of novice plans as training signal [2508.05310]. This suggests a conceptual reinterpretation of covariate shift: the key issue is not only that the expert must label states visited by the novice, but also that the novice’s own plan distribution contains information about capability, uncertainty, and alternative goal satisfaction.

## 3. Architecture: SAG, FIER, and PIER

The ASkDAgger framework is decomposed into three components: S-Aware Gating (SAG), Foresight Interactive Experience Replay (FIER), and Prioritized Interactive Experience Replay (PIER) [2508.05310]. The main loop initializes an optional behavioral cloning dataset $\mathcal{D}_\mathrm{BC}$ and a pre-trained novice $\pi^0_\mathrm{N}$, sets $\mathcal{D} \leftarrow \mathcal{D}_\mathrm{BC}$, and maintains arrays for update index per tuple, uncertainty per tuple, and reward per tuple [2508.05310]. For each episode, the novice plans an action, computes uncertainty, obtains a threshold from SAG, and then either queries through FIER or executes autonomously; the resulting tuple is appended, PIER computes replay priorities, and the policy is updated from the aggregated dataset [2508.05310].

SAG dynamically maintains a query threshold on uncertainty so as to track one of three user-selected metrics: sensitivity, specificity, or minimum system success rate [2508.05310]. FIER governs what happens when the novice queries the teacher: the teacher may validate the plan, reject it and provide an annotation, and optionally relabel the novice plan as a success for a different goal [2508.05310]. PIER defines how tuples in the aggregated dataset are sampled during training, prioritizing them on the basis of uncertainty, novice success, and demonstration age [2508.05310].

The division of labor among the three components is central. SAG decides *when* the teacher is asked, FIER decides *how* a queried interaction becomes data, and PIER decides *how strongly* that data influences subsequent optimization [2508.05310]. This modularity also clarifies the framework’s relationship to prior active imitation learning methods: thresholding alone is not ASkDAgger; the defining feature is the joint use of adaptive gating, plan-aware reuse of novice proposals, and replay prioritization.

## 4. S-Aware Gating (SAG)

SAG is described as an adaptive gating strategy that controls whether to query the teacher, with the explicit goal of maintaining a desired sensitivity, specificity, or minimum overall system success rate [2508.05310]. It casts each time step as a binary classification problem in which “positive” means a query is made and “negative” means the novice acts autonomously, while the latent ground truth is whether the novice’s action is valid or invalid [2508.05310]. In this view, a false positive wastes teacher time by querying on a valid novice action, whereas a false negative corresponds to an unqueried invalid action and thus system failure [2508.05310].

SAG keeps rolling histories of uncertainty, reward, and episode index:
\[
U = [u_t^k], \quad R = [r_t^k], \quad C = [k].
\]
It uses a window of recent entries
\[
U_W, R_W, C_W = \text{get\_window}(U,R,C,N_\mathrm{min},mode),
\]
where the definition of relevant labels depends on the selected mode [2508.05310]. Because uncertainty tends to decrease as the policy improves, older uncertainty values are normalized using linear regression fitted on the current window:
\[
w_\mathrm{lin}, b_\mathrm{lin} = \text{LinRegres().fit}(C_W, U_W),
\]
followed by shifting each uncertainty to its expected value at the current episode $K$:
\[
u_t^k \leftarrow u_t^k + w_\mathrm{lin}(K-k).
\]
This compensates for temporal drift in the uncertainty distribution [2508.05310].

SAG then defines failure labels $f_t^k = -r_t^k$, so that teacher rejections correspond to novice failures and teacher validations correspond to novice successes [2508.05310]. A logistic regression is fitted,
\[
w_\mathrm{log}, b_\mathrm{log} = \text{LogRegres().fit}(U_W, -R_W),
\]
to estimate
\[
P(f_t^k = 1 \mid u_t^k) \approx \sigma(w_\mathrm{log}u_t^k + b_\mathrm{log}),
\]
and unlabeled points with $r_t^k = 0$ receive pseudo-labels by sampling from this inferred Bernoulli model [2508.05310]. Repeating the pseudo-labeling process $N_\mathrm{rep}$ times yields a distribution of candidate thresholds, and the final gating threshold is taken as the median [2508.05310].

The method explicitly accounts for random querying. In sensitivity-aware mode, the expected total sensitivity under active thresholding plus random queries with probability $p_\mathrm{rand}$ is
\[
\mathbb{E}_{\epsilon \sim U[0,1)}[\sigma^\mathrm{sens}]
=
\sigma^\mathrm{sens}_\gamma + p_\mathrm{rand}(1-\sigma^\mathrm{sens}_\gamma),
\]
where $\sigma^\mathrm{sens}_\gamma$ is the sensitivity under active gating alone [2508.05310]. In specificity-aware mode,
\[
\mathbb{E}[\sigma^\mathrm{spec}] = \sigma^\mathrm{spec}_\gamma(1-p_\mathrm{rand}),
\]
and in success-aware mode,
\[
\mathbb{E}[\sigma^\mathrm{succ}] = \sigma^\mathrm{succ}_\gamma + p_\mathrm{rand}(1-\sigma^\mathrm{succ}_\gamma)
\]
under the assumption that the expert is always correct [2508.05310]. By construction, increasing the desired sensitivity or desired success rate lowers the threshold and produces more queries, whereas increasing the desired specificity raises the threshold and favors autonomy [2508.05310].

## 5. Foresight Interactive Experience Replay (FIER) and Prioritized Interactive Experience Replay (PIER)

FIER governs the data transformation performed at query time [2508.05310]. Given observation $o_t^k$, novice action $a_t^k$, teacher policy $\pi_\mathrm{T}$, current trajectory $\tau$, and goal $g^k$, the teacher responds to the novice’s plan by either validating it, rejecting it and supplying an annotation, or optionally relabeling the novice plan as successful for another goal $g' \in \mathcal{G}$ [2508.05310]. When the teacher validates the plan, FIER appends
\[
(o_t^k, a_t^k, g=g^k, r=1)
\]
to the trajectory and executes the novice plan [2508.05310]. When the teacher rejects the plan, it computes $a_\mathrm{T} = \pi_\mathrm{T}(o_t^k)$ and appends the annotation tuple
\[
(o_t^k, a_\mathrm{T}, g=g^k, r=-1),
\]
and if a relabeled goal is provided it also appends
\[
(o_t^k, a_t^k, g=g', r=0)
\]
as a relabeled tuple [2508.05310].

The stated interpretation is that validated novice actions become expert-quality demonstrations, while novice failures may still become positive examples for a different goal via relabeling [2508.05310]. The paper explicitly compares the latter mechanism to Hindsight Experience Replay, because the transition can be reinterpreted as successful with respect to another goal if the resulting state belongs to the corresponding goal-success set [2508.05310]. The significant departure from standard interactive imitation learning is that many useful demonstrations arise from the novice’s own plan and from teacher foresight, not only from executed expert corrections [2508.05310].

PIER determines sampling probabilities for tuples in $\mathcal{D}$ during training [2508.05310]. It defines an age feature
\[
\frac{K-k}{K},
\]
a prioritization exponent
\[
c_t^k = \lambda u_t^k + (1-\lambda)\frac{K-k}{K}, \quad 0 \le \lambda \le 1,
\]
and a priority score
\[
p_t^k = 1 - r_t^k \frac{b^{1-c_t^k} - 1}{b-1}, \quad b>1,
\]
which corresponds to Eq. (4) in the paper [2508.05310]. The resulting sampling distribution is
\[
P(k,t) = \frac{(p_t^k)^\alpha}{\sum_{i,j}(p_j^i)^\alpha},
\]
and importance weights are
\[
w_t^k = \left(|\mathcal{D}| \cdot P(k,t)\right)^{-\beta} / \max_{i,j} w_j^i
\]
to correct bias from non-uniform replay [2508.05310].

The stated training effect is that recent failures at low uncertainty receive the highest priority, since they indicate confident mistakes that should be corrected quickly, whereas old low-uncertainty successes receive low priority because they correspond to already mastered behavior [2508.05310]. Relabeled and offline demonstrations remain at priority $1$, which prevents them from being ignored or overemphasized [2508.05310].

## 6. Uncertainty estimation, experiments, and empirical findings

ASkDAgger is agnostic to the particular uncertainty estimator, requiring only an operator $u_t^k = U(\pi_\mathrm{N}, o_t^k, g^k) \in [0,1]$ [2508.05310]. In the MNIST experiments, uncertainty is computed with Monte Carlo Dropout using $M=16$ stochastic forward passes:
\[
P_\mathcal{C}(y \mid x) = \frac{1}{16} \sum_{i=1}^{16} P_i(y \mid x), \quad
u = 1 - \max_y P_\mathcal{C}(y \mid x),
\]
with dropout rate $0.4$, batch size $128$, and $468$ time steps per run, querying when uncertainty exceeds the SAG threshold or with probability $p_\mathrm{rand}=0.1$ [2508.05310]. In CLIPort experiments, uncertainty is the prediction entropy over pixel-wise Q-maps, while in the affordance-learning comparison a least-confidence measure is used:
\[
u = \left|\mathbf{1}\{Q(o,a) > 0.5\} - Q(o,a)\right|
\]
[2508.05310].

The principal robotic simulation benchmark consists of four CLIPort language-conditioned manipulation tasks: packing-google-objects-seq, packing-google-objects-group, packing-shapes, and put-blocks-in-bowls [2508.05310]. Training uses $10$ runs per method and $300$ interactive demonstrations, with checkpoints evaluated every $100$ demonstrations [2508.05310]. In these experiments, SAG is configured with mode $=$ sensitivity, $\sigma_\mathrm{des}=0.9$, $N_\mathrm{min}=15$, and $p_\mathrm{rand}=0.2$, while PIER uses $\alpha=1.5$, $b=10$, $\beta=1$, and $\lambda=0.5$ [2508.05310]. The compared methods are ASkDAgger, an Active DAgger baseline defined as ASkDAgger without FIER and PIER, SafeDAgger, ThriftyDAgger, and several ablations [2508.05310].

The experimental findings are organized around three claims. First, SAG tracks desired operating characteristics: in MNIST, for $\sigma_\mathrm{des} \in \{0.1,0.2,\dots,0.9\}$, it tracks sensitivity and specificity accurately, and in success-aware mode the overall system success rate follows the desired minimum [2508.05310]. In CLIPort tasks and real assembly, sensitivity is maintained near $0.9$ throughout training [2508.05310]. Second, FIER reduces the need for annotation demonstrations while improving generalization: ASkDAgger collects fewer annotation demonstrations than Active DAgger and the reported baselines, and on unseen tasks it significantly outperforms them [2508.05310]. The paper reports that with relabeling, ASkDAgger achieves on unseen CLIPort tasks an average evaluation reward improvement of about $62\%$ over its no-relabeling variant [2508.05310]. Third, PIER improves adaptation under domain shift, particularly as the demonstration dataset grows [2508.05310].

The real-world experiments use a Franka Panda for engine assembly and a Boston Dynamics Spot for object sorting [2508.05310]. In the assembly setting, the task is to insert colored 3D-printed bolts into specified locations on a 3D-printed engine block using an in-hand RealSense D405 RGB-D camera processed into a top-down projection and commands of the form “Insert the [color] bolt at location number [n]” [2508.05310]. A Gradio GUI presents plans to the human, who validates, relabels, or rejects them [2508.05310]. With $150$ interactive demonstrations, the system shows the same qualitative pattern seen in simulation: a growing fraction of demonstrations become validations and relabelings while maintaining high system success rate [2508.05310]. The reported approximate user interaction times are $3.7 \pm 1.9$ s for validation, $7.0 \pm 3.8$ s for annotation, and $7.5 \pm 1.9$ s for relabeling; with environment overhead, validation and annotation demonstrations take roughly $37$ s and $40$ s respectively, while relabeling can often be provided during an annotation interaction without additional execution [2508.05310].

## 7. Extensions, limitations, and the ASkDAgger lens in shared low-level control

ASkDAgger is explicitly designed for moderate query frequencies and high-level skills, not for raw torque control or very high-frequency decisions [2508.05310]. The framework assumes that the teacher can interpret proposed skill plans, judge validity, and supply relabelings without excessive cognitive burden, and that uncertainty is sufficiently calibrated to correlate with failure [2508.05310]. In success-aware mode, the expert is assumed always correct [2508.05310]. Relabeling is straightforward in structured manipulation tasks where alternative goal satisfaction is visually apparent, but the paper notes that it becomes harder in more abstract settings [2508.05310].

Within the broader literature, ASkDAgger is positioned at the intersection of active imitation learning, affordance learning, and RLHF-like interactive feedback, though it does not learn a reward model and instead performs behavior cloning at skill level with intelligent query and replay mechanisms [2508.05310]. The authors describe possible extensions to long-horizon and non-sparse tasks, integration with foundation models for plan visualization or synthetic demonstrations, hierarchical control architectures, and settings with heterogeneous or imperfect teachers [2508.05310].

A distinct but informative extension of the ASkDAgger perspective is provided by "Humanoid Hanoi: Investigating Shared Whole-Body Control for Skill-Based Box Rearrangement" [2602.13850]. That work studies a skill-based hierarchy for long-horizon humanoid box rearrangement in which all skills share a single task-agnostic whole-body controller (WBC), and observes that naïve reuse of the same pretrained WBC causes distribution shift as new skills and their compositions generate new state and command distributions [2602.13850]. The proposed remedy is a rollout-based data aggregation procedure: execute the hierarchical system in closed loop, record the directive trajectories sent from skill policies to the WBC during successful rollouts,
\[
\mathcal{S}_i = \{\text{directive trajectories from skill } \pi_i\},
\]
expand the reference set by
\[
\mathcal{R}_i = \mathcal{R}_{i-1} \cup \mathcal{S}_i,
\]
and continue PPO training on the enlarged directive distribution using the same task-agnostic objective with domain randomization and disturbance injection [2602.13850].

That paper does not present the human-interactive ASkDAgger algorithm of SAG, FIER, and PIER; however, it explicitly frames its procedure as “DAgger for the low-level shared controller” and as a concrete instance of Active Skill-level Data Aggregation for a shared low-level controller [2602.13850]. The aggregation occurs at WBC command level rather than at the level of teacher-labeled novice plans, and there is no external expert because the corrective signal is implicit in the original tracking-and-stability objective [2602.13850]. The reported results show that this coverage-expansion approach improves long-horizon robustness in Humanoid Hanoi, with the “Extended” shared WBC achieving approximately $70\%$ final success without dynamics randomization and $49\%$ with dynamics randomization, outperforming base, residual, and per-skill finetuned alternatives in the long-horizon setting [2602.13850]. A plausible implication is that the term “ASkDAgger” now names both a specific interactive imitation learning framework centered on teacher feedback at the skill level and a broader methodological viewpoint in which skill execution actively drives data aggregation to mitigate distribution shift in hierarchical control systems.

Source: https://www.emergentmind.com/topics/active-skill-level-data-aggregation-askdagger