MA-HERP: Hierarchical Motion-Action Prediction

• The paper introduces MA-HERP, a framework that jointly estimates continuous motion trajectories and discrete actions using a coupled probabilistic model.
• It employs a hierarchy of action intervals with Allen relations and a recursive Bayesian filtering scheme to synchronize top-down predictions with bottom-up sensory evidence.
• Experimental results demonstrate high accuracy in noise-free conditions and highlight performance challenges under noise, emphasizing the need for enhanced uncertainty calibration.

MA-HERP, short for Movement-Action Hierarchical Estimation and Recursive Prediction, is a hierarchical and recursive probabilistic framework for the joint estimation and prediction of human movements and actions in human–robot collaboration (Bulanti et al., 3 Apr 2026). It is designed for online settings in which a robot must reason simultaneously about continuous trajectories in $\mathbb{R}^d$ and discrete action labels such as reach, grasp, or release, rather than treating kinematics and symbolic intent as separate inference problems. Its defining elements are a hierarchy of action intervals linked by admissible Allen interval relations, a unified factorization coupling continuous dynamics, discrete labels, and explicit durations, and a Bayesian-filtering-inspired recursive inference scheme that alternates top-down action prediction with bottom-up sensory evidence (Bulanti et al., 3 Apr 2026).

1. Conceptual scope and problem formulation

MA-HERP addresses a characteristic difficulty of human–robot collaboration: safe and timely control depends on anticipating how the human body will move, while task-level coordination depends on inferring what the human intends to do next (Bulanti et al., 3 Apr 2026). In this setting, movements supply continuous geometric information, whereas actions provide task semantics. The framework is explicitly motivated by the observation that realistic collaborative behaviour is not a single sequential stream: sub-actions can overlap across effectors, and composite actions impose temporal semantics such as “reach must occur before grasp” or “release may meet the end of a reach” (Bulanti et al., 3 Apr 2026).

The framework therefore represents actions as labeled time intervals rather than as isolated instantaneous symbols. An action interval is written

$a_k = (\ell_k, s_k, e_k),$

where $\ell_k \in L$ is a discrete label, $s_k$ and $e_k$ are integer start and end instants with $1 \le s_k < e_k \le T$, and the duration is

$d_k = e_k - s_k + 1.$

Continuous observations are denoted by $X_t \in \mathbb{R}^d$, optional context by $C_t$, and the induced instantaneous label at time $t$ by $a_k = (\ell_k, s_k, e_k),$0 whenever $a_k = (\ell_k, s_k, e_k),$1 for some interval $a_k = (\ell_k, s_k, e_k),$2 (Bulanti et al., 3 Apr 2026).

A central implication of this formulation is that intent recognition and motion forecasting become mutually informative rather than merely adjacent. Action-conditioned dynamics sharpen continuous forecasts, while the same dynamics yield data-dependent evidence for the discrete label posterior. The framework is therefore neither a purely symbolic temporal grammar nor a purely kinematic predictor; it is a coupled continuous–discrete model with explicit temporal structure.

2. Interval hierarchy and temporal admissibility

The temporal core of MA-HERP is Allen interval algebra. Given two intervals $a_k = (\ell_k, s_k, e_k),$3 and $a_k = (\ell_k, s_k, e_k),$4, their relation $a_k = (\ell_k, s_k, e_k),$5 belongs to the Allen set

$a_k = (\ell_k, s_k, e_k),$6

together with converses such as after, met-by, overlapped-by, contains, started-by, and finished-by (Bulanti et al., 3 Apr 2026). The primitive relations are defined by the interval endpoints: before if $a_k = (\ell_k, s_k, e_k),$7, meets if $a_k = (\ell_k, s_k, e_k),$8, overlaps if $a_k = (\ell_k, s_k, e_k),$9, during if $\ell_k \in L$0 and $\ell_k \in L$1, starts if $\ell_k \in L$2 and $\ell_k \in L$3, finishes if $\ell_k \in L$4 and $\ell_k \in L$5, and equals if $\ell_k \in L$6 and $\ell_k \in L$7 (Bulanti et al., 3 Apr 2026).

Not every relation is permitted for every pair of labels. MA-HERP encodes this through label-dependent admissibility sets

$\ell_k \in L$8

which specify which relations are physically or task-wise plausible for the pair $\ell_k \in L$9 (Bulanti et al., 3 Apr 2026). Hard constraints are expressed by plausibility factors

$s_k$0

with the option for soft or fuzzy versions to tolerate segmentation noise (Bulanti et al., 3 Apr 2026). The framework therefore supports partial order and controlled overlap rather than enforcing a strictly linear action chain.

Hierarchy is introduced through $s_k$1 abstraction levels $s_k$2 (Bulanti et al., 3 Apr 2026). Level $s_k$3 contains movement segments aligned with continuous samples $s_k$4, while levels $s_k$5 contain increasingly abstract actions. A level-$s_k$6 action $s_k$7 has a possibly empty child set $s_k$8, and a level-specific composition table

$s_k$9

specifies which child-label pairs may stand in which relations (Bulanti et al., 3 Apr 2026). A parent action is therefore interpreted as a temporally organized pattern of lower-level intervals, with consistency enforced by a composition factor $e_k$0.

This hierarchical representation is especially suitable for tasks in which semantic structure is carried by interval composition rather than by a flat state transition graph. The paper’s illustrative example is a handover whose children may require reach(object) before grasp(object), grasp(object) before reach(handover_pose), and reach(handover_pose) meets release(object) (Bulanti et al., 3 Apr 2026).

3. Unified probabilistic formulation

MA-HERP couples interval structure, explicit durations, and continuous dynamics in a single factorization. At the action level, it adopts a semi-Markov prior with explicit durations:

$e_k$1

where $e_k$2 can be parameterized by a duration distribution or a hazard function $e_k$3, as in HSMMs, to avoid memoryless durations (Bulanti et al., 3 Apr 2026).

Within each level, admissible Allen relations are imposed by products of $e_k$4 over selected relevant interval pairs. Across levels, products of $e_k$5 ensure that the children of each parent action satisfy the appropriate composition table $e_k$6 (Bulanti et al., 3 Apr 2026). The continuous component is action-conditioned:

$e_k$7

so that the instantaneous label $e_k$8 selects the motion regime (Bulanti et al., 3 Apr 2026). An extended state-space form is also allowed, with optional latent continuous state $e_k$9, raw sensory measurement $1 \le s_k < e_k \le T$0, and regime variable $1 \le s_k < e_k \le T$1, although the reported experiments instantiate the model directly over $1 \le s_k < e_k \le T$2 without separate $1 \le s_k < e_k \le T$3 or $1 \le s_k < e_k \le T$4 (Bulanti et al., 3 Apr 2026).

Conditioned on context $1 \le s_k < e_k \le T$5, the full joint distribution factorizes as

$1 \le s_k < e_k \le T$6

The paper states three corresponding conditional-independence assumptions: given $1 \le s_k < e_k \le T$7 and $1 \le s_k < e_k \le T$8, $1 \le s_k < e_k \le T$9 is independent of other labels and intervals; Allen factors couple only the start and end times of the affected interval pairs; and composition factors couple a parent interval with its children and the children among themselves via $d_k = e_k - s_k + 1.$0 (Bulanti et al., 3 Apr 2026).

The continuous–discrete coupling operates in both directions. Top-down, the action-conditioned likelihood

$d_k = e_k - s_k + 1.$1

makes motion prediction label-specific. Bottom-up, the same likelihood acts as a data-dependent score for label inference, so the posterior over $d_k = e_k - s_k + 1.$2 favours labels whose dynamics best explain the observed movement while satisfying duration and Allen constraints (Bulanti et al., 3 Apr 2026).

4. Recursive inference and computational profile

The online inference procedure is described as approximate message passing in filtering form rather than exact inference in the full factor graph (Bulanti et al., 3 Apr 2026). For a single chain of labels, the framework defines $d_k = e_k - s_k + 1.$3 as the predictive prior, $d_k = e_k - s_k + 1.$4 as the action-conditioned observation likelihood, and $d_k = e_k - s_k + 1.$5 as the posterior belief over the instantaneous label. The recursion is

$d_k = e_k - s_k + 1.$6

modulated in practice by the duration hazard $d_k = e_k - s_k + 1.$7 to control boundary opening and closing,

$d_k = e_k - s_k + 1.$8

and

$d_k = e_k - s_k + 1.$9

where $X_t \in \mathbb{R}^d$0 summarizes active Allen constraints and boundary or duration bookkeeping, and $X_t \in \mathbb{R}^d$1 denotes a summary of current interval hypotheses and associated statistics (Bulanti et al., 3 Apr 2026).

The per-step online algorithm is stated explicitly:

1. Predict the label prior $X_t \in \mathbb{R}^d$2 from $X_t \in \mathbb{R}^d$3.
2. Score observations through $X_t \in \mathbb{R}^d$4 from $X_t \in \mathbb{R}^d$5.
3. Enforce structure through $X_t \in \mathbb{R}^d$6.
4. Update durations and boundaries via the HSMM hazard $X_t \in \mathbb{R}^d$7 and maintain $X_t \in \mathbb{R}^d$8.
5. Predict motion via

$X_t \in \mathbb{R}^d$9

with $C_t$0 the MAP label or a mixture if needed (Bulanti et al., 3 Apr 2026).

With multiple streams or hierarchy levels, $C_t$1 becomes a tuple of labels and $C_t$2 factorizes over selected intra-stream and inter-stream pairs; composition constraints are updated when a parent hypothesis becomes sufficiently probable and its children’s partial ordering can be enforced (Bulanti et al., 3 Apr 2026).

The reported computational profile is

$C_t$3

per time step for a single stream, where a dense transition matrix yields $C_t$4 for label prediction, likelihood evaluation costs $C_t$5, Allen penalties over $C_t$6 active intervals cost $C_t$7, and duration updates cost $C_t$8 (Bulanti et al., 3 Apr 2026). Sparse transitions reduce the prediction term toward $C_t$9, and the paper notes that in online human–robot collaboration both $t$0 and $t$1 are typically modest.

5. Experimental instantiation and reported performance

The preliminary experimental evaluation instantiates MA-HERP on musculoskeletal simulations produced with Bioptim (Bulanti et al., 3 Apr 2026). The simulated body is a torso and right arm with two shoulder DoFs and one elbow DoF, so $t$2 consists of three joint angles. The task uses three target volumes $t$3, $t$4, and $t$5, each a cube of $t$6 cm side, together with a starting pose $t$7. The label set contains nine transitions: $t$8, $t$9, $a_k = (\ell_k, s_k, e_k),$00, and all pairwise transitions among $a_k = (\ell_k, s_k, e_k),$01, $a_k = (\ell_k, s_k, e_k),$02, and $a_k = (\ell_k, s_k, e_k),$03 (Bulanti et al., 3 Apr 2026). For each label, the dataset contains $a_k = (\ell_k, s_k, e_k),$04 trajectories of length $a_k = (\ell_k, s_k, e_k),$05 samples, with two noise regimes: D-0, which is noise-free, and D-0-10-30, which mixes $a_k = (\ell_k, s_k, e_k),$06, $a_k = (\ell_k, s_k, e_k),$07, and $a_k = (\ell_k, s_k, e_k),$08 uniform additive noise along the trajectory (Bulanti et al., 3 Apr 2026).

The continuous component uses label-conditioned autoregressive Transformer predictors implementing $a_k = (\ell_k, s_k, e_k),$09, trained with history window $a_k = (\ell_k, s_k, e_k),$10 and forecast horizon $a_k = (\ell_k, s_k, e_k),$11, with inference by sliding window of stride $a_k = (\ell_k, s_k, e_k),$12 (Bulanti et al., 3 Apr 2026). D-0 uses $a_k = (\ell_k, s_k, e_k),$13 train and $a_k = (\ell_k, s_k, e_k),$14 test trajectories per label, while D-0-10-30 uses $a_k = (\ell_k, s_k, e_k),$15 train and $a_k = (\ell_k, s_k, e_k),$16 test trajectories per label. The discrete component uses Transformer-based classifiers for the composite reach-to-target labels $a_k = (\ell_k, s_k, e_k),$17, $a_k = (\ell_k, s_k, e_k),$18, $a_k = (\ell_k, s_k, e_k),$19, and $a_k = (\ell_k, s_k, e_k),$20; in the joint model, these classifiers provide the instantaneous belief $a_k = (\ell_k, s_k, e_k),$21, and durations are known a priori in these experiments, so HSMM learning is not exercised (Bulanti et al., 3 Apr 2026).

For continuous motion prediction, the reported qualitative result is that D-0 forecasts are visually indistinguishable from ground truth, whereas D-0-10-30 yields more scattered forecasts under noise (Bulanti et al., 3 Apr 2026). Quantitatively, D-0 achieves near-perfect PCC of approximately $a_k = (\ell_k, s_k, e_k),$22–$a_k = (\ell_k, s_k, e_k),$23 and RMSE of approximately $a_k = (\ell_k, s_k, e_k),$24 rad for most labels, with occasional PCC drops on highly non-linear segments. Under D-0-10-30, PCC often falls to $a_k = (\ell_k, s_k, e_k),$25–$a_k = (\ell_k, s_k, e_k),$26 and RMSE rises to approximately $a_k = (\ell_k, s_k, e_k),$27–$a_k = (\ell_k, s_k, e_k),$28 rad, which the paper interprets as sensitivity to sensory perturbations and motivation for explicit uncertainty handling in human–robot collaboration (Bulanti et al., 3 Apr 2026).

Forecasting runtime for $a_k = (\ell_k, s_k, e_k),$29 steps, measured on an Intel i7-10700 CPU at $a_k = (\ell_k, s_k, e_k),$30 GHz, is reported as approximately $a_k = (\ell_k, s_k, e_k),$31–$a_k = (\ell_k, s_k, e_k),$32 s per forecast for D-0, with standard deviation $a_k = (\ell_k, s_k, e_k),$33–$a_k = (\ell_k, s_k, e_k),$34 s, and approximately $a_k = (\ell_k, s_k, e_k),$35–$a_k = (\ell_k, s_k, e_k),$36 s per forecast for D-0-10-30, with standard deviation $a_k = (\ell_k, s_k, e_k),$37–$a_k = (\ell_k, s_k, e_k),$38 s (Bulanti et al., 3 Apr 2026). The paper states that these times are well below $a_k = (\ell_k, s_k, e_k),$39 s and therefore compatible with online use.

For discrete action prediction, incremental prefix inference shows that $a_k = (\ell_k, s_k, e_k),$40 under D-0 saturates after approximately $a_k = (\ell_k, s_k, e_k),$41 samples and under D-0-10-30 after approximately $a_k = (\ell_k, s_k, e_k),$42 samples, while $a_k = (\ell_k, s_k, e_k),$43 saturates after approximately $a_k = (\ell_k, s_k, e_k),$44 samples in D-0 and approximately $a_k = (\ell_k, s_k, e_k),$45 samples in D-0-10-30 (Bulanti et al., 3 Apr 2026). $a_k = (\ell_k, s_k, e_k),$46 and $a_k = (\ell_k, s_k, e_k),$47 require longer evidence, and $a_k = (\ell_k, s_k, e_k),$48 degrades more under noise. With a sliding window of length $a_k = (\ell_k, s_k, e_k),$49 and majority vote over time, the reported accuracies are: for D-0, $a_k = (\ell_k, s_k, e_k),$50 $a_k = (\ell_k, s_k, e_k),$51, $a_k = (\ell_k, s_k, e_k),$52 $a_k = (\ell_k, s_k, e_k),$53, $a_k = (\ell_k, s_k, e_k),$54 $a_k = (\ell_k, s_k, e_k),$55, and $a_k = (\ell_k, s_k, e_k),$56 $a_k = (\ell_k, s_k, e_k),$57; for D-0-10-30, $a_k = (\ell_k, s_k, e_k),$58 $a_k = (\ell_k, s_k, e_k),$59, $a_k = (\ell_k, s_k, e_k),$60 $a_k = (\ell_k, s_k, e_k),$61, $a_k = (\ell_k, s_k, e_k),$62 $a_k = (\ell_k, s_k, e_k),$63, and $a_k = (\ell_k, s_k, e_k),$64 $a_k = (\ell_k, s_k, e_k),$65 (Bulanti et al., 3 Apr 2026). The confusions are asymmetric: class $a_k = (\ell_k, s_k, e_k),$66 recall can drop to approximately $a_k = (\ell_k, s_k, e_k),$67 under noise despite high precision, making the classifier conservative in predicting $a_k = (\ell_k, s_k, e_k),$68.

Classification runtime is approximately $a_k = (\ell_k, s_k, e_k),$69–$a_k = (\ell_k, s_k, e_k),$70 ms per prediction with negligible variance (Bulanti et al., 3 Apr 2026). An end-to-end chaining experiment tracks a sequence of $a_k = (\ell_k, s_k, e_k),$71 consecutive reaches in which the relations between consecutive reaches are before or meet; online beliefs $a_k = (\ell_k, s_k, e_k),$72 update as evidence accrues, and the system produces continuous forecasts $a_k = (\ell_k, s_k, e_k),$73 and discrete labels consistent with the chain’s temporal semantics (Bulanti et al., 3 Apr 2026).

6. Relation to adjacent models, naming ambiguity, and open issues

MA-HERP is positioned against several neighboring modeling traditions (Bulanti et al., 3 Apr 2026). Relative to separate pipelines consisting of a continuous predictor and an independent action classifier, it couples the two levels probabilistically: action-conditioned dynamics sharpen motion forecasts, motion likelihoods provide bottom-up evidence for actions, and Allen or duration constraints stabilize label inference and reduce implausible segmentations. Relative to HMM or HSMM formulations, it preserves HSMM-like durations but augments them with Allen interval constraints capable of representing meets, overlaps, starts, and finishes rather than only sequential transitions. Relative to Switching Linear Dynamical Systems, it replaces linear-Gaussian mode dynamics with non-linear label-conditioned neural dynamics and adds hierarchical interval structure with explicit Allen relations and composition tables (Bulanti et al., 3 Apr 2026).

The framework rests on several stated assumptions. It assumes a fixed or slowly varying set of admissible Allen relations $a_k = (\ell_k, s_k, e_k),$74, stationary semi-Markov durations or hazards within a task context, time-aligned observations $a_k = (\ell_k, s_k, e_k),$75, and available optional context $a_k = (\ell_k, s_k, e_k),$76 (Bulanti et al., 3 Apr 2026). The paper also identifies several limitations: the current experiments use a simplified $a_k = (\ell_k, s_k, e_k),$77-DoF reaching setup with a limited action vocabulary, do not provide end-to-end closed-loop human–robot collaboration validation under realistic sensing latency, occlusions, or multimodal fusion, leave uncertainty calibration of the continuous predictors to future work, and note that the interaction between history length $a_k = (\ell_k, s_k, e_k),$78 and horizon $a_k = (\ell_k, s_k, e_k),$79 requires systematic analysis to avoid oscillations and over-commitment (Bulanti et al., 3 Apr 2026).

The proposed extensions are correspondingly directed toward richer temporal, sensory, and interaction structure. They include multi-agent settings with cross-agent Allen relations and joint composition tables, multimodal sensing through a richer observation model $a_k = (\ell_k, s_k, e_k),$80 and learned sensor fusion, richer action grammars and symbolic task constraints through expanded composition tables and logic-based priors, and explicit latent state $a_k = (\ell_k, s_k, e_k),$81 with probabilistic emission models for uncertainty propagation (Bulanti et al., 3 Apr 2026). A plausible implication is that MA-HERP is intended less as a fixed architecture than as a template for coupled continuous–discrete online inference in temporally structured collaboration tasks.

A separate terminological issue arises because the label “MA-HERP” is not used uniformly across the broader literature represented here. In the proteomics paper “HERP: Hardware for Energy Efficient and Realtime DB Search and Cluster Expansion in Proteomics,” the term “MA-HERP” is not explicitly defined; the text instead presents “Memory-Augmented HERP” as a reasonable interpretation for a proposed extension of HERP’s compute-in-memory architecture (Nayan et al., 5 Nov 2025). By contrast, the explicit and fully specified definition of MA-HERP as Movement-Action Hierarchical Estimation and Recursive Prediction belongs to the human–robot collaboration framework summarized above (Bulanti et al., 3 Apr 2026).