IMoRe: Implicit Program-Guided Motion Reasoning

• IMoRe is a framework that leverages explicit symbolic programs to guide implicit, attention-based reasoning over multi-level 3D human motion features.
• It overcomes the scalability and brittleness of traditional neuro-symbolic pipelines by using a unified neural module for multi-step relational reasoning.
• Empirical results on Babel-QA and HuMMan-QA demonstrate state-of-the-art performance, showcasing robustness even with noisy program inputs.

Searching arXiv for the cited IMoRe-related papers to ground the article in current preprints. arXiv Search Query: all:"IMoRe: Implicit Program-Guided Reasoning for Human Motion Q&A" arXiv Search Query: all:"Motion-R1: Chain-of-Thought Reasoning and Reinforcement Learning for Human Motion Generation" arXiv Search Query: all:"MotiMotion: Motion-Controlled Video Generation with Visual Reasoning" Implicit Program-Guided Motion Reasoning (IMoRe) is a framework for human motion question answering in which a structured program guides multi-step reasoning over motion, while execution is performed by a unified neural reasoning module rather than by manually designed functional modules. In its canonical formulation, the input consists of a 3D human motion sequence $\mathcal{M} \in \mathbb{R}^{T \times J \times 3}$, a natural language question $\mathcal{Q}$, and a structured program $\mathcal{P}$, and the output is a categorical answer such as an action class, a direction, or a body part. The method was introduced to address the scalability, adaptability, and brittleness limitations of explicit neuro-symbolic motion QA pipelines, while preserving explicit symbolic programs as high-level supervision for what operations to perform and in which order (Li et al., 4 Aug 2025).

1. Conceptual foundations

IMoRe is situated between two established design choices for reasoning over motion. On one side are explicit neuro-symbolic systems, exemplified by NSPose, in which a parsed program is executed step by step through manually designed neural modules such as filter, relate, and query. On the other side are implicit reasoning models, such as MAC-style architectures, that infer operations from question words through attention. IMoRe combines elements of both: it retains explicit symbolic programs, but replaces handcrafted module execution with a single attention-based reasoning core that is directly conditioned on structured program functions (Li et al., 4 Aug 2025).

This design is motivated by three limitations of explicit module-based execution. First, scalability is constrained because every new function or query type requires a new module. Second, adaptability is limited because manually crafted modules are tuned to specific datasets and question templates. Third, explicit execution is fragile to program errors: when a predicted program step is wrong, later steps are easily corrupted. IMoRe responds by treating the program as high-level control, while allowing the neural reasoner to implement each operation in a data-driven manner (Li et al., 4 Aug 2025).

A common misconception is that IMoRe is “program-free.” It is not. The framework remains program-guided: the symbolic program is an explicit input, represented as $\mathcal{P} \in \mathbb{R}^{p \times d}$, where $p$ is the program length and $d$ is the program embedding dimension. What is implicit is the execution mechanism. Another common misconception is that IMoRe simply infers reasoning operations from question wording. The defining distinction is the opposite: unlike earlier implicit reasoning approaches that infer operations from question words, IMoRe directly conditions on structured program functions, which reduces ambiguity and aligns neural computation with the symbolic plan (Li et al., 4 Aug 2025).

The target problem is human motion Q&A over 3D skeleton sequences. Example questions include “What action does the person do before they move left?”, “What body part does the person use after they jump?”, and “Which direction does the person move between walking and sitting?” These require multi-step relational reasoning about actions over time, directions of movement, involved body parts, and temporal relations such as before, after, and between (Li et al., 4 Aug 2025).

2. Architecture and implicit program execution

The architecture has five components: a pretrained motion Vision Transformer from MotionPatches, a RoBERTa text encoder for the question, a program encoder for the structured program, a program-guided reasoning module with iterative memory refinement, and a multi-branch classifier with separate branches for query_action, query_direction, and query_body_part (Li et al., 4 Aug 2025).

The reasoning process executes for $p$ steps, where $p$ is the program length. At each step, the current program function embedding $P_{i+1}$ controls two operations. The first is attention over relevant previous memory states. If the memory states up to step $i$ are $\mathcal{Q}$0, then the pre-memory aggregation is

$\mathcal{Q}$1

where $\mathcal{Q}$2 denotes the subset of previous memory states relevant to the current step according to the program dependency structure (Li et al., 4 Aug 2025).

The second operation is a read from motion features conditioned on that aggregated memory. In the single-level form discussed in the paper, this read is

$\mathcal{Q}$3

but IMoRe replaces this with a multi-level read over projected intermediate ViT features: $\mathcal{Q}$4 The full iterative process is summarized as

$\mathcal{Q}$5

The final memory state is then routed to the classifier branch associated with the query type (Li et al., 4 Aug 2025).

Program structure is therefore not decorative metadata; it specifies dependencies between reasoning steps. For the program query_action(relate(before, filter(left))), the relate(before) step depends on the output of filter(left), so the attention over previous memory prioritizes the memory associated with that filter step. This is the operational meaning of “implicit program-guided” in IMoRe: the program defines the reasoning trace, but the trace is executed by attention and memory updates rather than by a library of hard-coded functional modules (Li et al., 4 Aug 2025).

3. Motion representation and program-guided reading

IMoRe uses a pretrained motion ViT from MotionPatches to encode raw 3D joint sequences into motion patches defined over body-part subsets such as torso, right leg, and left arm. From this encoder, the model extracts multi-level motion features

$\mathcal{Q}$6

where each $\mathcal{Q}$7 is the output of an intermediate ViT layer or the final layer. Lower layers preserve finer spatial-temporal details, whereas higher layers encode more abstract semantic information such as overall action or direction (Li et al., 4 Aug 2025).

The motion representation is first made text-aware. With $\mathcal{Q}$8 and $\mathcal{Q}$9, the paper describes a fusion step

$\mathcal{P}$0

A second fusion with question-type encoding is also applied before the reasoning module. This design makes motion features sensitive both to the language content of the question and to whether the task is an action query, direction query, or body-part query (Li et al., 4 Aug 2025).

The distinctive mechanism is program-guided reading over multi-level features. Each level is projected into a common space,

$\mathcal{P}$1

and the current reasoning state then attends over $\mathcal{P}$2. The paper reports that this allows different program functions to select different abstraction levels: filter_action(...) and query_action tend to focus on higher-level features, whereas query_body_part shifts attention to lower-level features that encode joint-level motion cues. This dynamic level selection is central to IMoRe’s claim that a single reasoning core can unify multiple query types without manually designed modules (Li et al., 4 Aug 2025).

This mechanism also clarifies why IMoRe is not reducible to generic sequence classification over skeletons. The reasoning core does not merely compress motion into a single representation and answer from that representation; it repeatedly revisits motion features under program control, with memory refinement aligned to the symbolic reasoning trace. A plausible implication is that the program functions act as control signals for selecting both temporal evidence and semantic granularity, rather than only as answer-type indicators.

4. Training, datasets, and empirical performance

IMoRe is trained end to end with cross-entropy over answer classes: $\mathcal{P}$3 The reported optimization setup uses AdamW with weight decay $\mathcal{P}$4, 100 epochs, learning rate $\mathcal{P}$5, dropout $\mathcal{P}$6, and batch size $\mathcal{P}$7. The motion encoder is a pretrained MotionPatches ViT treated mainly as a feature extractor, and the text encoder is RoBERTa (Li et al., 4 Aug 2025).

Two datasets define the empirical setting. Babel-QA, introduced in NSPose, contains 1,800 training questions, 384 validation questions, and 393 test questions derived from 771 motion sequences. It includes query action, query direction, and query body part questions with temporal relations before, after, and between. HuMMan-QA, constructed from HuMMan-MoGen, contains 2,066 training questions, 524 validation questions, and 533 test questions from 1,311 motion sequences, with 158 action classes, 18 body-part classes, and 6 direction classes. Its labels were generated with GPT-4o using in-context examples and then manually refined and verified against the actual motion sequences (Li et al., 4 Aug 2025).

On Babel-QA, the main comparison is against NSPose and motion-text fusion baselines. NSPose with ground-truth programs reaches 0.578 overall accuracy, and NSPose with predicted programs reaches 0.472. IMoRe I reaches 0.609 with ground-truth programs and 0.602 with predicted programs. IMoRe II reaches 0.640 with ground-truth programs and 0.615 with predicted programs, which is the best overall result in the reported table. The small drop from 0.640 to 0.615 under predicted programs, compared with the 0.578 to 0.472 drop of NSPose, is presented as evidence that implicit program-guided execution is substantially more robust to program noise than explicit neuro-symbolic execution (Li et al., 4 Aug 2025).

On HuMMan-QA, MotionCLIP-MLP reaches 0.686, MotionCLIP-RNN reaches 0.623, NSPose reaches 0.691, IMoRe I reaches 0.719, and IMoRe II reaches 0.730. This is reported as evidence that the framework generalizes across datasets with different motion content and larger answer vocabularies (Li et al., 4 Aug 2025).

The ablation study further isolates the source of the gains. Replacing NSPose’s original encoder with a ViT motion encoder gives 0.440. A MAC-style implicit reasoner without program guidance gives 0.582. Adding program-guided reasoning with single-level features gives 0.607. Adding multi-level feature selection, which yields the full IMoRe II, gives 0.640. The progression $\mathcal{P}$8 supports three specific conclusions stated in the paper: the gains are not due to a stronger encoder alone, implicit iterative reasoning is stronger than explicit execution in this setting, and explicit program guidance plus multi-level feature selection both make measurable contributions (Li et al., 4 Aug 2025).

The supplementary comparison with video-LLMs rendered on skeleton videos is also notable: IMoRe I reaches 0.609 overall on Babel-QA, while Qwen-2.5-VL-3B reaches 0.425, InternVL2.5-4B reaches 0.402, and MiniCPM reaches 0.384. This suggests that large generic multimodal models do not automatically recover fine-grained structured motion reasoning in the absence of explicit program guidance (Li et al., 4 Aug 2025).

5. Relation to adjacent motion-reasoning paradigms

IMoRe belongs to a broader family of motion systems that insert an intermediate structured representation between input language or observations and final motion-related output, but it is distinctive in keeping the structured program explicit while making execution implicit. The surrounding literature spans human motion Q&A, text-to-motion generation, video generation, pixel-level motion grounding, and trajectory-grounded video reasoning.

System	Program-like intermediate	Primary task
IMoRe	structured program functions $\mathcal{P}$9	human motion Q&A
Motion-R1	> and <output> action paths	text-to-3D-human-motion generation
>	MotiMotion	refined prompt plus refined and secondary trajectories
>	Motion-o	discrete <motion/> tags for direction, speed, and scale
>	LaMoGen	LabanLite conceptual sequence
>	MotionMERGE	atomic edit instructions plus motion-grounded CoT

In human motion generation, “Motion-R1: Chain-of-Thought Reasoning and Reinforcement Learning for Human Motion Generation” frames text-to-motion as generation conditioned on an explicit reasoning chain $\mathcal{P} \in \mathbb{R}^{p \times d}$0, comprising CoT text and a structured action plan, before motion tokens are emitted. The model uses a VQ-VAE motion tokenizer and a Qwen-2.5-3B-Instruct backbone, and the paper explicitly describes the CoT plus action plan as a latent program that acts as a planner while the motion token sequence acts as an executor. The authors state that Motion-R1 can be seen as a strong IMoRe prototype because it uses “program-like CoT descriptions” and group-relative RL to jointly optimize the program and its execution (Ouyang et al., 12 Jun 2025).

In motion-controlled video generation, “MotiMotion: Motion-Controlled Video Generation with Visual Reasoning” reformulates motion control as a reasoning-then-generation pipeline. A training-free VLM reasoner takes the first frame, user trajectories, and an optional prompt, then outputs a refined narrative prompt, refined primary trajectories, and secondary trajectories for reactive objects. A confidence-aware control scheme modulates the strength of trajectory guidance through Gaussian scaling $\mathcal{P} \in \mathbb{R}^{p \times d}$1, so high-confidence plans are followed strictly while low-confidence plans act as soft hints. The paper explicitly interprets this as implicit motion reasoning and as a concrete blueprint for IMoRe-style systems (Hsin-Ying et al., 21 May 2026).

At the pixel level, “Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level” defines a question-conditioned task $\mathcal{P} \in \mathbb{R}^{p \times d}$2 in which the answer is a sequence of segmentation masks rather than text. The GROUNDMORE dataset contains 1,715 video clips, 249.5K object masks, and four question types—Causal, Sequential, Counterfactual, and Descriptive—designed to require implicit spatiotemporal reasoning. Its baseline model, MoRA, combines a multimodal LLM, a SAM-based segmentation head, and a temporal localization head with a special [LOC] token. From an IMoRe perspective, the paper argues that questions operate as latent programs, and that the dataset is suitable for explicit program induction over motion events (Deng et al., 2024).

For trajectory-grounded reasoning in general video VLMs, “Motion-o: Trajectory-Grounded Video Reasoning” introduces Spatial-Temporal-Trajectory reasoning and Motion Chain of Thought (MCoT). Here the program-like intermediate is a discrete <motion/> tag summarizing per-object direction, speed, and scale change. The reward function includes a trajectory term

$\mathcal{P} \in \mathbb{R}^{p \times d}$3

and a visual grounding term based on counterfactual motion masking, so the model is rewarded for producing motion descriptors that depend on visual evidence rather than static priors. The paper explicitly presents Motion-o as a concrete instantiation of implicit program-guided motion reasoning for video (Galoaa et al., 19 Mar 2026).

In contrast, “LaMoGen: Language to Motion Generation Through LLM-Guided Symbolic Inference” uses an explicit symbolic layer, LabanLite, rather than an implicit executor over structured programs. The LLM composes conceptual descriptions for body-part groups and durations, a kinematic detail augmentor fills in detail symbols, and a Laban-Motion Decoder reconstructs continuous motion. This is a more explicit program-guided motion generation pipeline than IMoRe, but it sharpens the same principle: separate high-level symbolic planning from low-level continuous execution (Jiang et al., 12 Mar 2026).

Finally, “MotionMERGE: A Multi-granular Framework for Human Motion Editing, Reasoning, Generation, and Explanation” extends the program-guided perspective to editing and explanation. Its training objective combines cross-granularity, temporal grounding, localized alignment, motion coherence, and motion-grounded CoT losses, and MotionFineEdit provides 837K atomic plus 144K complex triplets with motion-grounded CoT annotations. The paper interprets complex edit instructions as implicit programs over body parts and temporal segments, and describes the T5-based model as a learned interpreter over discrete motion tokens (Wu et al., 18 May 2026).

Taken together, these works suggest that IMoRe is not only the name of a specific human motion QA framework but also a useful organizing concept for a broader design space in which structured intermediate representations—program functions, CoT action paths, trajectories, symbolic motion languages, or atomic edit operations—guide motion reasoning without requiring manually engineered executors.

6. Limitations, unresolved issues, and future directions

The original IMoRe paper identifies several limitations. Fine-grained body-part reasoning remains difficult, especially for “body part between” questions where multiple similar motions are temporally close. Some “direction after” queries are also misclassified, indicating that subtle directional changes are not always captured reliably by the current motion features or temporal aggregation scheme. Although the model is more robust to noisy programs than NSPose, it still depends on having reasonably accurate program annotations or predictions (Li et al., 4 Aug 2025).

These limitations point to a tension at the center of IMoRe. The model benefits from explicit program structure, but it does not explicitly parse or verify the semantics of each function during execution. This suggests that future work may need tighter coupling between parsing and reasoning, richer program vocabularies, or stronger motion representations for subtle joint-level cues. The paper itself proposes joint learning of parsing and reasoning, tighter integration with parsers, and extensions toward more complex logical constructs such as and, or, not, counting, comparatives, and causal reasoning (Li et al., 4 Aug 2025).

A broader unresolved issue is the level of explicitness that yields the best trade-off between flexibility and interpretability. IMoRe eliminates manually designed functional modules, but other systems in the same research area push in different directions. Motion-o shows that small discrete symbolic states such as direction, speed, and scale can be highly effective for evidence-based reasoning; LaMoGen shows that explicit symbolic motion languages can improve controllability and interpretability; MotionMERGE shows that large-scale motion-grounded CoT can support fine-grained editing; and Motion-R1 and MotiMotion show that planner–executor structure can be embedded within a single autoregressive or diffusion-based model. This suggests that future IMoRe systems may converge on hybrid designs in which program structure is more explicit than in current implicit executors, yet more scalable than hand-built neuro-symbolic pipelines.

In that broader sense, IMoRe marks a methodological shift: motion reasoning is treated as structured computation over temporally evolving evidence, but the computation is learned rather than hard-coded. Its significance lies less in a single architectural trick than in a specific proposition about how motion understanding should be organized—explicit symbolic supervision for the reasoning trace, implicit neural execution for the reasoning steps, and dynamic access to multi-level motion representations under program control (Li et al., 4 Aug 2025).