Query-Centric Inverse Reinforcement Learning

• Query-centric IRL is a framework that employs direct queries or conditioning signals to resolve ambiguities in reward inference from demonstrations.
• It integrates various query modalities such as subgoal strings, state-action critiques, and summary observations to tailor the learning process.
• This approach improves policy performance and learning efficiency by focusing on high-risk states and adapting the reward model to task-specific constraints.

Query-centric inverse reinforcement learning (IRL) denotes a family of IRL formulations in which reward inference is conditioned, guided, or disambiguated by an explicit query mechanism or by an additional information channel beyond passive state-action trajectories. In the strict sense, the query may be a membership question over subgoal strings, a state at which an action label is requested, or a trajectory presented for critique; in a broader sense, it may be a summary observation channel, a temporal relevance distribution, learner-side preference constraints, an optimality profile, or expert visitation side information that changes what is inferred and how policy optimization is carried out (Memarian et al., 2020, Brown et al., 2019, Kangasrääsiö et al., 2017, Jarboui et al., 2021, Haug et al., 2020, Swamy et al., 2023).

1. Scope and defining interpretations

In the narrowest usage, query-centric IRL is an active inverse-learning setting in which the learner selects an informational target and updates its reward model from the answer. "Active Task-Inference-Guided Deep Inverse Reinforcement Learning" formalizes this with membership queries and conjecture queries over symbolic subgoal strings $\omega$, then uses the inferred automaton to make non-Markovian reward learning Markovian on a product MDP (Memarian et al., 2020). "Risk-Aware Active Inverse Reinforcement Learning" instead queries either a state for its correct action or a rollout for segment-level critique, and it selects those queries by a policy-loss $\alpha$-Value-at-Risk criterion rather than generic uncertainty reduction (Brown et al., 2019). "Inverse Reinforcement Learning from Summary Data" is not active in this sense, but it is query-centric in a passive observation-channel sense: the learner conditions on $\sigma(\xi)$, a possibly stochastic summary of an unobserved latent trajectory $\xi$ (Kangasrääsiö et al., 2017).

A broader usage includes methods that do not query a human directly but still condition inverse inference on information that selects which aspects of behavior matter. In this broader sense, the conditioning signal may be a temporal weighting distribution $\eta$ over future offsets, learner-side preferences and constraints, an optimality profile $P_{\mathrm{tgt}}$, a latent behavior cluster $c_j$, or the expert visitation distribution $\rho_E^t$; these mechanisms do not all constitute explicit human queries, but they all modify the inverse problem away from passive global demonstration matching (Jarboui et al., 2021, Tschiatschek et al., 2019, Haug et al., 2020, Rajasekaran et al., 2017, Swamy et al., 2023).

2. Query modalities and information channels

The literature supports several distinct query objects. Some are explicit elicitation targets; others are conditioning signals that function as query-like side information because they determine which latent reward structure is recoverable.

Modality	Queried or conditioned object	Canonical representation
Symbolic task-structure elicitation	Subgoal string $\omega$	Membership and conjecture queries; DFA over $E^*$
Action or critique querying	State $\alpha$0 or rollout from $\alpha$1	Action label or positive/negative segment critique
Summary-observation conditioning	Latent trajectory $\alpha$2 through $\alpha$3	$\alpha$4
Quality-summary supervision	Demonstration-quality distribution	Optimality profile $\alpha$5
Context-conditioned inverse learning	Learner constraints or temporal relevance	$\alpha$6-weighted objectives; preference-constrained IRL
Broad-sense expert side information	Expert visitation or latent mode	$\alpha$7; latent cluster $\alpha$8

The first four rows correspond to direct queried or externally supplied observation channels. In the automaton-based setting, the query is a sequence of subgoals and the answer is a Boolean task-completion judgment; in ActiveVaR, the query is a state or a rollout and the answer is an action label or critique; in summary-data IRL, the query answer is the compressed observation itself; and in optimality-profile matching, the conditioning signal is a distribution over reward-derived quantities rather than pairwise preferences (Memarian et al., 2020, Brown et al., 2019, Kangasrääsiö et al., 2017, Haug et al., 2020).

The last two rows are broader conditioning mechanisms. In generalized IRL, $\alpha$9 specifies what future horizons matter and thereby changes the matched statistic from classical discounted occupancy to a future-weighted occupancy $\sigma(\xi)$0. In efficient expert-reset IRL, the expert visitation distribution $\sigma(\xi)$1 removes the need to rediscover expert-relevant states during each policy update. In nonparametric behavior clustering IRL, the latent variable $\sigma(\xi)$2 plays the role of an unobserved query or intent mode that indexes one reward function per cluster (Jarboui et al., 2021, Swamy et al., 2023, Rajasekaran et al., 2017).

3. Formal models and objective functions

A canonical direct formulation appears in active task-structure-guided IRL. The environment is a reward-free MDP $\sigma(\xi)$3, the task is a Boolean language

$\sigma(\xi)$4

and the latent task structure is represented by a DFA $\sigma(\xi)$5. Query answers constrain the hypothesized language, and the reward learner then operates on the product MDP $\sigma(\xi)$6 with state space $\sigma(\xi)$7. Reward is parameterized as $\sigma(\xi)$8, and the maximum-entropy policy is defined through the soft Bellman fixed point

$\sigma(\xi)$9

This converts non-Markovian reward dependence on subgoal history into ordinary Markovian reward learning on the augmented space (Memarian et al., 2020).

Risk-aware active IRL uses a different formal core. It assumes a Bayesian IRL posterior

$\xi$0

with a softmax-rational demonstrator likelihood. The acquisition criterion is not posterior entropy but statewise policy-loss risk. For a candidate state $\xi$1, the statewise loss random variable is

$\xi$2

and the learner queries the state maximizing $\xi$3-VaR: $\xi$4 This turns active IRL into risk-targeted intervention: query where the current policy may generalize badly under plausible rewards (Peyton et al., 2019).

Summary-data IRL replaces full trajectories by a general summary channel. Instead of observing $\xi$5 directly, the learner observes

$\xi$6

The exact likelihood therefore marginalizes over latent trajectories: $\xi$7 and the full posterior is $\xi$8. When exact likelihood is infeasible, the paper uses Monte Carlo estimators or approximate Bayesian computation with discrepancy $\xi$9, making the observation channel itself the central object of inverse inference (Kangasrääsiö et al., 2017).

Broader query-like conditioning appears in two mathematically distinct forms. Generalized IRL replaces the fixed initial-state weighting of classical IRL by a user-chosen temporal weighting $\eta$0, leading to

$\eta$1

and the generalized objective

$\eta$2

The matched object is the future-weighted occupancy $\eta$3, not the classical discounted occupancy $\eta$4 (Jarboui et al., 2021). Learner-aware teaching modifies maximum-causal-entropy IRL by adding learner preference constraints, so that inverse inference becomes

$\eta$5

subject to reward-feature matching and convex preference constraints. In the hard-constraint limit, the learner projects demonstrated reward-feature expectations onto a learner-feasible set $\eta$6, making the inverse problem explicitly context-conditioned by learner admissibility (Tschiatschek et al., 2019).

4. Algorithmic patterns

The task-inference-guided line exemplifies a tightly coupled symbolic-neural loop. ATIG-DIRL takes as input a reward-free MDP $\eta$7 and stopping threshold $\eta$8, initializes $\eta$9, $P_{\mathrm{tgt}}$0, and $P_{\mathrm{tgt}}$1, then alternates between $P_{\mathrm{tgt}}$2 TaskInferenceModule$P_{\mathrm{tgt}}$3 and $P_{\mathrm{tgt}}$4 RewardLearningModule$P_{\mathrm{tgt}}$5. The reward module repeatedly computes soft $P_{\mathrm{tgt}}$6 and $P_{\mathrm{tgt}}$7, differentiates through the soft Bellman equations, performs gradient ascent on $P_{\mathrm{tgt}}$8, evaluates the current success ratio $P_{\mathrm{tgt}}$9 by Monte Carlo rollout, and, if performance remains below threshold, returns a symbolic counterexample $c_j$0 for the next automaton-learning round (Memarian et al., 2020).

ActiveVaR uses a more conventional active-learning loop but with a risk-based acquisition function. It samples rewards from $c_j$1, computes the MAP reward $c_j$2 and policy $c_j$3, estimates a one-sided confidence upper bound on statewise $c_j$4-VaR of policy loss for each candidate state, queries the highest-risk state, augments the demonstration set, reruns Bayesian IRL, and stops when

$c_j$5

A critique-query variant first selects the highest-risk state, rolls out $c_j$6 from that state, and then asks for a critique of the rollout rather than an isolated action label (Peyton et al., 2019).

Summary-data IRL supports three inference patterns. Exact inference enumerates all plausible latent trajectories and marginalizes them through $c_j$7. Monte Carlo likelihood approximation samples latent trajectories from the model and averages summary compatibility. ABC dispenses with explicit likelihood and keeps only a discrepancy test $c_j$8. To reduce repeated RL cost, the paper builds a Gaussian-process surrogate over either log-likelihood or discrepancy and uses Bayesian optimization to choose parameter settings to evaluate; the resulting surrogate likelihood is then sampled by MCMC (Kangasrääsiö et al., 2017).

Several adjacent methods fit the same design pattern of replacing global, monolithic inverse inference with structured conditioning. GIRL/MEGAN changes replay-buffer sampling so that discrimination and occupancy matching are performed over future-weighted occupancy $c_j$9 rather than classical occupancy. "Inverse Reinforcement Learning without Reinforcement Learning" replaces repeated full RL solves by local policy-improvement subproblems on expert-reset states through MMDP, NRMM, and the practical interpolating meta-algorithm FILTER. OCR-IRL computes two null spaces—one from intra-option policy optimality and one from termination optimality—intersects them to obtain a compatible $\rho_E^t$0-feature space $\rho_E^t$1, and then converts $\rho_E^t$2 into reward features $\rho_E^t$3 by option-aware reward shaping before selecting a concrete reward via second-order criteria (Jarboui et al., 2021, Swamy et al., 2023, Hwang et al., 2019).

5. Empirical evidence and application domains

Active task-structure-guided IRL was evaluated on a custom task-oriented navigation domain with three DFA-defined tasks. The method inferred a DFA equivalent to the ground-truth DFA for all three tasks in at most three iterations of the outer loop. On 10 randomly generated test environments it achieved roughly $\rho_E^t$4 to $\rho_E^t$5 task success across tasks, while memoryless IRL collapsed near zero on all tasks; on the hardest repeated-subgoal task, IRL-IB achieved only $\rho_E^t$6 mean success versus $\rho_E^t$7 for ATIG-DIRL. The paper also states that for complex tasks such as long ordered sequences with repeated subgoals, completion probability is at least nine times higher than baselines (Memarian et al., 2020).

ActiveVaR reports two different empirical messages. In gridworld action-query experiments, it reduces policy loss faster than Random and AS, because it directly targets high-risk regions rather than high-entropy ones. In critique-query experiments, ARC can outperform ActiveVaR per number of trajectory queries, but ActiveVaR is far cheaper computationally: Random requires $\rho_E^t$8 s/iteration, ActiveVaR $\rho_E^t$9 s/iteration, and ARC $\omega$0 s/iteration. Additional appendix results report that Random may require on average $\omega$1 more demonstrations than ActiveVaR to achieve low worst-case policy loss (Peyton et al., 2019).

Summary-data IRL shows that meaningful posterior inference is possible even when the learner observes only coarse summaries. In grid world, the summary is $\omega$2. In the menu-search cognitive-science model, the observed data contain only task completion time and menu condition, yet ABC still recovers a useful posterior. At the MAP estimate, simulated summary statistics are close to the observed ones: task completion time absent is $\omega$3 ms versus $\omega$4 ms observed, and task completion time present is $\omega$5 ms versus $\omega$6 ms observed (Kangasrääsiö et al., 2017).

Several broader conditioning methods report quantitative gains that are directly relevant to query-centric design principles. In generalized IRL, increasing horizon emphasis in $\omega$7 produces a $\omega$8 to $\omega$9 reduction in MMD divergences as $E^*$0, and in Ant and Half-Cheetah this is accompanied by a factor $E^*$1 to $E^*$2 reduction in cumulative cost. In efficient expert-reset IRL, learners are given $E^*$3 demonstrations, and in $E^*$4 of $E^*$5 environments FILTER variants reach strong policies significantly faster than the standard moment-matching baseline; in locomotion $E^*$6 worked best, whereas in antmaze $E^*$7 worked best. In optimality-profile matching on LunarLander, the target profile plus about $E^*$8–$E^*$9 pairwise comparisons and fewer than $\alpha$00 fixed points can recover rewards good enough that PPO-trained policies sometimes become near-optimal and solve the environment with $\alpha$01 average episodic return, in some cases outperforming the demonstrations used for fitting. In OCR-IRL’s Four-Rooms transfer setting, combining default reward with recovered hierarchical reward by

$\alpha$02

worked best at $\alpha$03, supporting the claim that option-compatible rewards transfer temporally abstract structure (Jarboui et al., 2021, Swamy et al., 2023, Haug et al., 2020, Hwang et al., 2019).

6. Misconceptions, scope boundaries, and limitations

A common misconception is to equate query-centric IRL with pairwise preference learning. The literature considered here is much broader. Queries may be symbolic membership tests over subgoal strings, action labels at carefully selected states, critiques of rollouts initiated from high-risk states, or summary observations delivered through a known channel $\alpha$04. Even optimality-profile supervision is distributional rather than pairwise, and learner-aware teaching conditions inverse inference on the learner’s own preferences and constraints rather than on a conventional human preference comparison (Memarian et al., 2020, Peyton et al., 2019, Kangasrääsiö et al., 2017, Haug et al., 2020, Tschiatschek et al., 2019).

An equally important boundary separates direct querying from adjacent conditioning mechanisms. Generalized IRL with temporal weighting $\alpha$05, expert-reset reductions, curricular subgoals, nonparametric behavior clustering, and option-compatible reward recovery all alter the inverse problem by changing what portion of future behavior, state space, or temporal abstraction is treated as salient; this makes them close relatives of query-centric IRL, but not all of them are query-centric in the strict sense. Their assumptions are also strong: the strongest theory for generalized IRL is for geometric $\alpha$06; expert-reset methods require access to the expert visitation distribution strongly enough to reset into expert states; curricular subgoals are automatically discovered rather than user queried; nonparametric behavior clustering infers a latent mode instead of conditioning on an explicit query; and OCR-IRL assumes options are assigned for each domain rather than discovered (Jarboui et al., 2021, Swamy et al., 2023, Liu et al., 2023, Rajasekaran et al., 2017, Hwang et al., 2019).

Taken together, this literature defines query-centric IRL less as a single algorithmic family than as a design principle: replace undifferentiated demonstration fitting by an information channel that directly constrains the latent reward relevant to the downstream use case. In some works that channel is an active query; in others it is a summary, a constraint set, a temporal relevance kernel, a latent mode, or a reset distribution. The central technical question is therefore not whether a method contains a human-in-the-loop query, but which additional signal most effectively resolves the particular inverse ambiguity that passive demonstrations leave open.