MO-IRL: Minimal Observation Inverse RL

• MO-IRL is a class of inverse reinforcement learning methods that infers reward structures from minimal or summarized observations rather than complete state-action trajectories.
• It employs inference strategies such as Monte Carlo sampling, approximate Bayesian computation, and maximum entropy methods to handle sparse or state-only data.
• MO-IRL has been applied to continuous-space tasks and human-reaching experiments, achieving improved trajectory reconstruction and efficient learning with very few demonstrations.

Searching arXiv for the cited MO-IRL and related papers to ground the article in current literature. Minimal Observation Inverse Reinforcement Learning (MO-IRL) denotes a class of inverse reinforcement learning formulations in which the learner must infer an underlying reward or cost structure from observations that are weaker than the full state-action trajectories assumed in classical IRL. In the recent human-reaching literature, the term refers to a specific maximum-entropy IRL procedure designed to work with very few demonstrations while estimating time-varying, phase-specific cost weights. In a broader lineage, related formulations treat trajectories as latent variables observed only through summary statistics or through state-only trajectories, thereby extending IRL to settings with severe information loss or restricted supervision (Mehrdad et al., 30 Sep 2025, Mehrdad et al., 13 May 2025, Kangasrääsiö et al., 2017, Chang et al., 2022).

1. Scope and terminology

Classical IRL assumes that demonstrations are available as full state-action paths. MO-IRL relaxes that assumption in several distinct but closely connected ways. In one formulation, the observer sees only a summary generated by a function $\sigma$ acting on the latent trajectory, so the data may consist of path length, task completion time, or other aggregated observables rather than the trajectory itself. In another, the learner has full trajectories but only a very small number of them, so the central challenge is not missing modalities but sample scarcity. In a third, closely related state-only setting, the learner receives expert state trajectories without actions or rewards and must recover a reward surrogate that supports forward control (Kangasrääsiö et al., 2017, Mehrdad et al., 13 May 2025, Chang et al., 2022).

Across these formulations, the unifying issue is that the inverse problem is posed under weakened observability. This suggests that “minimal observation” is not a single observation model but a design principle: reward inference must proceed when demonstrations are partial, summarized, state-only, or numerically scarce. A common misconception is therefore to equate MO-IRL with one specific data type. The literature instead uses the label for several mechanisms that reduce the dependence of IRL on dense trajectory supervision, while preserving an explicit forward model, a trajectory distribution, or both (Mehrdad et al., 30 Sep 2025, Kangasrääsiö et al., 2017).

2. Latent-trajectory formulations under restricted observation

A foundational generalization is IRL from summary data. Let $\xi_i$ denote a latent state-action path generated under an optimal policy for MDP parameters $\theta$, and let the observer receive only $\xi_{i\sigma}$ produced by a deterministic or stochastic summary function $\sigma$. The summary likelihood is then

$$P(\xi_{i\sigma} \mid \theta) = \sum_{\xi_i \in \Xi_{ap}} P(\xi_{i\sigma} \mid \xi_i)\,P(\xi_i \mid \theta),$$

and the full posterior is

$$P(\theta \mid \Xi_\sigma)\propto L(\theta \mid \Xi_\sigma)\,P(\theta).$$

This formulation makes the trajectory latent and pushes the observation model into $\sigma$, so identifiability depends jointly on the MDP, the summary map, and the amount of data rather than on Markov factorization alone (Kangasrääsiö et al., 2017).

The same paper develops exact inference for finite problems and two approximate routes for larger ones. The Monte Carlo method samples trajectories from $P(\xi\mid\theta)$ and approximates the summary likelihood through empirical averaging of $P(\xi_{i\sigma}\mid\xi_n)$. The ABC route replaces explicit likelihood evaluation by a discrepancy $\xi_i$0 between simulated and observed summaries, then models the discrepancy with a Gaussian process surrogate and uses Bayesian optimization plus MCMC to obtain a posterior approximation. The examples are deliberately severe: in the grid world, the summary is $\xi_i$1, so the observer sees only the start cell and the number of steps until the goal; in the menu-search case study, the observation is task completion time only, together with whether the target item was present or absent (Kangasrääsiö et al., 2017).

These constructions establish an important baseline for MO-IRL. Minimal observation does not merely mean “few trajectories”; it can mean that the entire path is unobserved and only a low-dimensional statistic remains. In that regime, posterior uncertainty is not an accessory output but a central object, because broad or multimodal posteriors directly reflect the information lost through $\xi_i$2 (Kangasrääsiö et al., 2017).

3. Continuous-space maximum-entropy MO-IRL

A second line of work formulates MO-IRL directly in continuous state and action spaces with a linear-in-features trajectory cost

$\xi_i$3

and a finite-sample maximum-entropy likelihood over an expert trajectory $\xi_i$4 and a small set of non-expert trajectories $\xi_i$5:

$\xi_i$6

Instead of solving for $\xi_i$7 in one shot, the algorithm updates $\xi_i$8 and reweights each stored trajectory by

$\xi_i$9

The resulting step-direction problem is solved in $\theta$0, optionally with sub-trajectory augmentation and elastic-net regularization, while nonnegativity is enforced through $\theta$1 (Mehrdad et al., 13 May 2025).

The defining computational choice is that non-expert trajectories are generated by solving an optimal control problem under the current cost rather than by random sampling. This makes each trajectory in the partition function informative with respect to the current weight vector. The method further uses a moving window over recent trajectories, with experiments often taking $\theta$2, and accepts steps only when a newly generated optimal-control rollout improves either a cost-based merit or the feature mismatch

$\theta$3

The step size starts at $\theta$4, is reduced by a factor of $\theta$5 if needed, and the procedure terminates after $\theta$6 unsuccessful trials (Mehrdad et al., 13 May 2025).

Empirically, the continuous-space algorithm is explicitly designed for “minimal observations” in the sense of requiring very few trajectories in the partition function. On point-mass tasks it converges in $\theta$7–$\theta$8 iterations with only the last trajectory used at each iteration; on the KUKA IIWA task it uses $\theta$9 total trajectories, compared with $\xi_{i\sigma}$0 for IS-IRL and $\xi_{i\sigma}$1 fixed local rollouts for PI$\xi_{i\sigma}$2-IRL. Reported computation times on KUKA are $\xi_{i\sigma}$3 s for MO-IRL, $\xi_{i\sigma}$4 s for IS-IRL, and $\xi_{i\sigma}$5 s for PI$\xi_{i\sigma}$6-IRL (Mehrdad et al., 13 May 2025).

This incremental, observation-weighted scheme is also the basis for later human-reaching extensions. There, the update is lifted from a single static weight vector to a segmented weight matrix, multiple demonstrations are handled simultaneously, and the step-acceptance criterion is evaluated in state space using both positions and velocities rather than positions alone (Mehrdad et al., 30 Sep 2025, Mehrdad et al., 8 Mar 2026).

4. Time-varying cost inference for human reaching

In the human-reaching literature, MO-IRL is a specific maximum-entropy inverse optimal control framework for learning phase-dependent cost weights from a very small number of demonstrations. The forward model is a planar two-link arm with state $\xi_{i\sigma}$7, control $\xi_{i\sigma}$8, and discrete dynamics $\xi_{i\sigma}$9. The cost is a segmented linear combination of seven hand-crafted features: Cartesian velocity, energy, geodesic, joint acceleration, joint torque change, joint velocity, and joint torque. With $\sigma$0 time windows, the phase-dependent cost is

$\sigma$1

The inverse problem is therefore to estimate either a posture-specific or a more general time-varying weight structure under the assumption that observed reaches are approximately optimal for this cost (Mehrdad et al., 30 Sep 2025, Mehrdad et al., 8 Mar 2026).

The 2025 reaching study extends MO-IRL to multiple demonstrations and segmented weights by defining a multi-segment cost difference for each candidate trajectory and optimizing a regularized increment $\sigma$2. A merit function in state space,

$\sigma$3

is used in a line-search-like step-acceptance procedure. For each posture, the method initializes $\sigma$4, uses $\sigma$5 training demonstrations, keeps only the most recent generated trajectory in the approximating set, and stops if no merit improvement is obtained within $\sigma$6 line-search trials. Training on ten trials per posture yields average joint-angle RMSE of $\sigma$7 deg and $\sigma$8 deg for six- and eight-segment weight divisions, respectively, versus $\sigma$9 deg using a single static weight. Cross-validation on remaining trials and inter-subject validation on an unseen subject’s $$P(\xi_{i\sigma} \mid \theta) = \sum_{\xi_i \in \Xi_{ap}} P(\xi_{i\sigma} \mid \xi_i)\,P(\xi_i \mid \theta),$$0 trials produce comparable predictive accuracy, around $$P(\xi_{i\sigma} \mid \theta) = \sum_{\xi_i \in \Xi_{ap}} P(\xi_{i\sigma} \mid \xi_i)\,P(\xi_i \mid \theta),$$1 deg RMSE (Mehrdad et al., 30 Sep 2025).

The 2026 follow-up asks whether a single unified cost can explain reaching across subjects and postures. It evaluates Subject-Dependent Posture-Dependent, Subject-Dependent Posture-Independent, and Subject-Independent Posture-Independent settings on data from $$P(\xi_{i\sigma} \mid \theta) = \sum_{\xi_i \in \Xi_{ap}} P(\xi_{i\sigma} \mid \xi_i)\,P(\xi_i \mid \theta),$$2 right-handed naïve subjects, with $$P(\xi_{i\sigma} \mid \theta) = \sum_{\xi_i \in \Xi_{ap}} P(\xi_{i\sigma} \mid \xi_i)\,P(\xi_i \mid \theta),$$3 trials per posture over five initial postures. Time-varying weights substantially improve trajectory reconstruction, yielding an average $$P(\xi_{i\sigma} \mid \theta) = \sum_{\xi_i \in \Xi_{ap}} P(\xi_{i\sigma} \mid \xi_i)\,P(\xi_i \mid \theta),$$4 reduction in RMSE compared to the baseline. In the most specific SDPD setting, the reported average RMSE is $$P(\xi_{i\sigma} \mid \theta) = \sum_{\xi_i \in \Xi_{ap}} P(\xi_{i\sigma} \mid \xi_i)\,P(\xi_i \mid \theta),$$5 for MO-IRL, compared with $$P(\xi_{i\sigma} \mid \theta) = \sum_{\xi_i \in \Xi_{ap}} P(\xi_{i\sigma} \mid \xi_i)\,P(\xi_i \mid \theta),$$6 for the baseline. Across SDPD, SDPI, and SIPI, the inferred costs consistently assign the dominant role to joint acceleration, with joint torque change contributing notably during mid-movement (Mehrdad et al., 8 Mar 2026).

These results are significant because they show that a minimal number of demonstrations can support estimation of a relatively large number of parameters, namely $$P(\xi_{i\sigma} \mid \theta) = \sum_{\xi_i \in \Xi_{ap}} P(\xi_{i\sigma} \mid \xi_i)\,P(\xi_i \mid \theta),$$7 weights. They also show that the main explanatory gain comes from time variation rather than from merely refitting a static weight vector: the improvement from one segment to six or eight segments is large, and the learned profiles are markedly non-uniform across movement phases (Mehrdad et al., 30 Sep 2025, Mehrdad et al., 8 Mar 2026).

5. State-only reward inference and the IL-flOw connection

A closely related formulation arises in imitation learning from observation. IL-flOw considers an MDP $$P(\xi_{i\sigma} \mid \theta) = \sum_{\xi_i \in \Xi_{ap}} P(\xi_{i\sigma} \mid \xi_i)\,P(\xi_i \mid \theta),$$8 in which the learner is given only a dataset of expert state sequences, $$P(\xi_{i\sigma} \mid \theta) = \sum_{\xi_i \in \Xi_{ap}} P(\xi_{i\sigma} \mid \xi_i)\,P(\xi_i \mid \theta),$$9, without expert actions, expert rewards, or a parametric expert policy. The objective is to learn a policy $$P(\theta \mid \Xi_\sigma)\propto L(\theta \mid \Xi_\sigma)\,P(\theta).$$0 such that the induced trajectory distribution matches the expert distribution, with the key construction being a reward obtained from a learned transition density: $$P(\theta \mid \Xi_\sigma)\propto L(\theta \mid \Xi_\sigma)\,P(\theta).$$1, or more generally

$$P(\theta \mid \Xi_\sigma)\propto L(\theta \mid \Xi_\sigma)\,P(\theta).$$2

under a noise-conditioned model (Chang et al., 2022).

The density model is a normalizing flow, specifically Neural Spline Flows, trained on expert transitions and their noise-perturbed counterparts. The resulting reward is stationary because the flow is trained once by maximum likelihood and then frozen before policy optimization. Under deterministic and invertible-in-action dynamics, the paper relates the induced objective to reverse KL minimization between learner and expert trajectory distributions, so maximizing expected log-density reward plus policy entropy approximately minimizes $$P(\theta \mid \Xi_\sigma)\propto L(\theta \mid \Xi_\sigma)\,P(\theta).$$3. The practical pipeline is to train the flow on state-state transitions, define reward as transition log-probability, augment the action with a scalar noise-level component $$P(\theta \mid \Xi_\sigma)\propto L(\theta \mid \Xi_\sigma)\,P(\theta).$$4, and run finite-horizon Soft Actor-Critic on the resulting reward (Chang et al., 2022).

In a broader interpretive sense, this occupies almost exactly a minimal-observation IRL setting: the learner sees only state trajectories and must infer a reward surrogate and a control policy from them. The difference from the continuous-space MO-IRL algorithms above is methodological. IL-flOw decouples reward modeling from policy learning and uses density estimation plus forward RL, whereas the continuous MO-IRL line uses a finite-sample maximum-entropy objective, explicit linear feature costs, and repeated optimal-control solves. The commonality is that both avoid reliance on expert actions and both convert restricted observations into a reusable reward signal (Chang et al., 2022).

The empirical evidence is correspondingly state-only rather than biomechanical. IL-flOw is evaluated on Hopper-v2, Walker2d-v2, and HalfCheetah-v2 with expert datasets of $$P(\theta \mid \Xi_\sigma)\propto L(\theta \mid \Xi_\sigma)\,P(\theta).$$5, $$P(\theta \mid \Xi_\sigma)\propto L(\theta \mid \Xi_\sigma)\,P(\theta).$$6, or $$P(\theta \mid \Xi_\sigma)\propto L(\theta \mid \Xi_\sigma)\,P(\theta).$$7 trajectories, approximately $$P(\theta \mid \Xi_\sigma)\propto L(\theta \mid \Xi_\sigma)\,P(\theta).$$8, $$P(\theta \mid \Xi_\sigma)\propto L(\theta \mid \Xi_\sigma)\,P(\theta).$$9, and $\sigma$0 state transitions. It matches or surpasses expert performance on several tasks, consistently outperforms f-IRL and state-only MaxEnt IRL across all data regimes, and shows that average log-probability under the learned density tracks environment return for expert, noisy expert, and random policies (Chang et al., 2022).

6. Limitations, controversies, and open directions

The principal limitation shared by all MO-IRL variants is identifiability under restricted information. In the summary-data setting, not all summary functions $\sigma$1 are informative enough to identify $\sigma$2; some settings yield narrow likelihoods, while others leave substantial posterior uncertainty or identify only ratios between reward components. This is why the Bayesian posterior is central rather than optional in that line of work (Kangasrääsiö et al., 2017).

In continuous-space MO-IRL, the main limitations are algorithmic and modeling-related. The method depends strongly on feature design, because the cost is linear in hand-specified features. It has no formal global convergence guarantees, remains subject to local minima, and requires repeated optimal-control solves. Practical performance can also depend on small non-zero initialization, regularization parameters, sub-trajectory choices, and, in high-dimensional feature spaces, explicit bounds such as $\sigma$3 (Mehrdad et al., 13 May 2025).

The human-reaching applications add domain-specific constraints. The current implementations use a planar $\sigma$4-DOF arm model and a simple pointing task, with trajectories segmented into equal-duration windows chosen heuristically rather than learned. Inter-subject validation in the 2025 study uses only one additional subject, and the approach assumes that the biomechanical model and the DOC solver are accurate enough to support consistent inverse inference. The 2026 study further notes the limitations of the optimality assumption, the linear-in-features cost class, and the restriction to a stylized planar reaching task (Mehrdad et al., 30 Sep 2025, Mehrdad et al., 8 Mar 2026).

The state-only density-modeling line has a different theoretical caveat. Its reverse-KL derivation relies on deterministic, approximately invertible dynamics, so the connection between transition-density reward and trajectory-distribution matching is looser in highly stochastic or non-invertible environments. It also inherits the usual difficulty of density estimation in more complex observation spaces, especially when moving from low-dimensional states to richer sensory inputs (Chang et al., 2022).

Taken together, these limitations clarify what MO-IRL is and is not. It is not a single algorithm, not a guarantee of identifiability from arbitrarily weak data, and not a replacement for forward modeling. It is a family of inverse methods that preserve an explicit generative account of behavior while relaxing the standard requirement of abundant state-action demonstrations. Within that family, the main current directions are richer observation models, better uncertainty quantification, more general dynamics, learned representations for higher-dimensional observations, and broader tests of whether time-varying or state-only reward surrogates capture transferable optimality principles across tasks and agents (Kangasrääsiö et al., 2017, Mehrdad et al., 8 Mar 2026)