HieroAction: Fine-Grained Action Assessment

Updated 9 July 2026

HieroAction is a two-stage vision–language system that provides a detailed decomposition of human actions into sub-actions with expert-aligned scores and explanations.
It employs a structured Stepwise Action Reasoning framework and hierarchical policy learning to ensure format fidelity, temporal precision, and semantic accuracy.
Empirical results show that HieroAction outperforms traditional methods by delivering interpretable, phase-by-phase feedback across varied domains such as sports, healthcare, and robotics.

HieroAction is a two-stage vision–language system for fine-grained human action analysis that is designed to deliver accurate and structured assessments rather than only a final score. It is introduced as a model for settings such as sports, healthcare, and robotics, where decisions rely not only on final outcomes but also on interpretable reasoning. The system builds on Qwen-VL-2.5-7B and combines supervised fine-tuning with a structured reasoning trace, termed Stepwise Action Reasoning, and reinforcement fine-tuning with Hierarchical Policy Learning. Its stated objective is to map a video clip and textual prompt to a structured assessment consisting of sub-action labels, expert-aligned scores, and natural-language explanations (Wu et al., 23 Aug 2025).

1. Conceptual definition and problem setting

HieroAction addresses the problem of evaluating human actions with clear and detailed feedback. The motivating claim is that most existing methods provide only a final score without explanation or detailed analysis, which limits practical applicability in domains where interpretable reasoning is required (Wu et al., 23 Aug 2025). The model therefore targets not only action quality estimation but also explicit decomposition of an action into sub-actions and justification of each judgment.

The formal input is a video clip, denoted by the frame sequence $V = \{I_1,\dots,I_T\}$ , together with a textual prompt $q$ , such as “Evaluate this dive” or “Assess this synchronized swimming routine.” The output is a structured assessment

$A = \{(a_k, s_k, e_k)\}_{k=1}^K$

where $a_k$ is the $k$ -th sub-action label, $s_k$ its expert-aligned score, and $e_k$ a natural-language explanation (Wu et al., 23 Aug 2025).

This design positions HieroAction as an action-assessment model rather than a generic captioning or action-recognition system. A plausible implication is that its core contribution lies in aligning symbolic action structure, scalar scoring, and free-form rationale within one generative framework.

2. System architecture and staged workflow

At a high level, HieroAction is described as a two-stage vision–language system built upon Qwen-VL-2.5-7B. The first stage is Supervised Fine-Tuning with Stepwise Action Reasoning, and the second stage is Hierarchical Policy Learning (Wu et al., 23 Aug 2025).

In the supervised stage, each annotated video is converted into a Chain-of-Thought–style QA sequence using four tagged stages: <look>…</look>, <recognition>…</recognition>, <assessment>…</assessment>, and <answer>…</answer>. Qwen-VL is then fine-tuned to generate this structured trace. In the reinforcement stage, the supervised model is frozen as an initial policy $\pi_0$ and refined under a Group-Relative Policy Optimization constraint so as to encourage high rewards on format, temporal alignment, semantic action correctness, and score accuracy (Wu et al., 23 Aug 2025).

The overall learning pipeline is summarized as learning

$F:(V,q)\to A$

via supervised fine-tuning on SAR-formatted data followed by policy optimization that maximizes a weighted sum of rewards (Wu et al., 23 Aug 2025).

This architecture is explicitly hierarchical in two senses. First, the generated reasoning proceeds from global observation to local sub-action analysis and then to final judgment. Second, reinforcement learning is applied over rewards that separately target structural validity, temporal precision, action semantics, and score alignment. This suggests that “hierarchical” refers not merely to model depth, but to the organization of both reasoning and optimization.

3. Stepwise Action Reasoning

Stepwise Action Reasoning, abbreviated SAR, is the reasoning format used during supervised fine-tuning. Unlike traditional CoT, which is described as simply interleaving reasoning steps, SAR enforces a four-phase structure intended to mirror an expert judge (Wu et al., 23 Aug 2025).

The four phases are as follows:

Observation (<look>): focus on the main actor and context.
Recognition (<recognition>): segment the video into semantically coherent sub-actions, each with “Phase: [...], Observation: [...], Conclusion: [...]”.
Assessment (<assessment>): for each sub-action, produce a quantitative score and textual rationale.
Conclusion (<answer>): synthesize a final global judgment (Wu et al., 23 Aug 2025).

The paper gives an abbreviated example in diving. The <look> stage identifies the diver’s board and stance. The <recognition> stage identifies a “take-off” phase with an observation about bent knees and a conclusion of good extension. The <assessment> stage assigns a take-off score of 8.5 due to a solid launch angle. The <answer> stage synthesizes the overall dive type and a final score of 8.1, stated as “difficulty 2.4 × execution average 8.3” (Wu et al., 23 Aug 2025).

The stated role of SAR is direct supervision of structured decomposition and justification. The claim is that this staged structure is directly supervised in SFT, so the model learns to decompose actions and justify each score (Wu et al., 23 Aug 2025). In context, this distinguishes HieroAction from black-box quality regressors: the output is not only scalar but also phase-annotated and rationale-bearing. A common misconception would be to equate SAR with generic chain-of-thought prompting. The paper explicitly rejects that equivalence by stating that SAR “instead enforces a four-phase structure that mirrors an expert judge” (Wu et al., 23 Aug 2025).

4. Hierarchical Policy Learning and reward design

Hierarchical Policy Learning, abbreviated HPL, is the reinforcement-learning stage used after supervised fine-tuning. Its motivation is that SFT can still cause the model to latch onto spurious language patterns, even when it reproduces expert reasoning traces (Wu et al., 23 Aug 2025).

The generator is treated as a policy $\pi_\theta(y\mid x)$ over token sequences $q$ 0, given input $q$ 1. The state is

$q$ 2

and the action is the next token $q$ 3 (Wu et al., 23 Aug 2025).

The model uses four reward components for a complete generated trace $q$ 4:

Format reward:

$q$ 5

Temporal alignment reward:

$q$ 6

where $q$ 7 and $q$ 8 are ground-truth and predicted sub-action intervals, and $q$ 9 is the optimal one-to-one matching.

Hierarchical action reward:

$A = \{(a_k, s_k, e_k)\}_{k=1}^K$ 0

with

$A = \{(a_k, s_k, e_k)\}_{k=1}^K$ 1

and $A = \{(a_k, s_k, e_k)\}_{k=1}^K$ 2 in practice.

Assessment reward:

$A = \{(a_k, s_k, e_k)\}_{k=1}^K$ 3

where $A = \{(a_k, s_k, e_k)\}_{k=1}^K$ 4 and $A = \{(a_k, s_k, e_k)\}_{k=1}^K$ 5 are predicted quality and difficulty, with ground truth $A = \{(a_k, s_k, e_k)\}_{k=1}^K$ 6 and $A = \{(a_k, s_k, e_k)\}_{k=1}^K$ 7 (Wu et al., 23 Aug 2025).

These are combined as

$A = \{(a_k, s_k, e_k)\}_{k=1}^K$ 8

with $A = \{(a_k, s_k, e_k)\}_{k=1}^K$ 9 and $a_k$ 0 (Wu et al., 23 Aug 2025).

Under GRPO, described as a variant of PPO, the policy update minimizes

$a_k$ 1

where $a_k$ 2 is the pre-RL policy and $a_k$ 3 controls divergence (Wu et al., 23 Aug 2025).

The reward construction is notable because it mixes symbolic constraints, temporal localization, sequence-level action semantics, and scalar score fidelity. This suggests that HPL is intended to regularize not only output quality but also the internal structure of the generated analysis. In that sense, the policy is “hierarchical” because it is rewarded at multiple levels of granularity.

5. Training configuration and benchmark datasets

The base model is Qwen2.5-VL-Instruct-7B, and training is reported on 4×A100-40 GB GPUs (Wu et al., 23 Aug 2025). The training curriculum is explicitly “SAR first to instill structure, then HPL to refine” (Wu et al., 23 Aug 2025).

The two stages use the following optimization settings:

Stage	Configuration	Optimization
Supervised Fine-Tuning	Batch size 16 (accumulated), 25 epochs	Learning rate $a_k$ 4, greedy decoding, max length 3072 tokens
Hierarchical Policy Learning	Same batch setup, 10 epochs	Learning rate $a_k$ 5, KL coefficient $a_k$ 6; for each $a_k$ 7, sample $a_k$ 8 trajectories at temperature 1.5, take highest-reward example

Three benchmark datasets are used (Wu et al., 23 Aug 2025):

Dataset	Split and scale	Action structure
FineDive	3 000 clips, 2 251 train / 749 test, 10–20 s lengths	Three phases: take-off / flight / entry
FineFS	1 167 solos, 933 / 234 train/test, 2–4 min	7–12 elements per program: jumps, spins, step sequences
LOGO	200 team routines, 150 / 50 splits, 2–3 min	Eight athletes, multi-person synchronization and artistic criteria

The dataset selection spans single-person and multi-person evaluation, short and long durations, and both execution-focused and artistic criteria. A plausible implication is that the model is intended to generalize across diverse scoring protocols rather than only one sport-specific benchmark.

6. Empirical performance and ablation evidence

The evaluation is organized along three axes: Action Assessment, Score Assessment, and Content Quality. The reported metrics are accuracy and Sub-action Edit Distance (SED) for action assessment, Spearman $a_k$ 9 and relative $k$ 0 for score assessment, and LLM-judged alignment and clarity for content quality (Wu et al., 23 Aug 2025).

On FineDive, HieroAction reports the following excerpted results: Acc_action = 0.9344, SED = 0.9731, difficulty ρ = 0.9423, R-ℓ₂ = 0.0158, score ρ = 0.8564, R-ℓ₂ = 0.0714, and content Acc = 0.9279, Avg. LLM score = 4.12. Against the best fine-tuned baseline, InternVL-2.5-SFT, the reported differences are Acc = 0.9012 (+3.3 pts), SED = 0.9483 → ours +2.5 pts, and score ρ = 0.8148 vs. 0.8564 (+4.9 %), with R-ℓ₂ 0.0886 → 0.0714 (Wu et al., 23 Aug 2025).

On FineFS, the reported comparison is SED = 0.6761 vs. 0.6421, ρ = 0.4232 vs. 0.2653, R-ℓ₂ = 0.1136 vs. 0.3732, and Content Acc = 0.5556 vs. 0.4615. On LOGO, the reported comparison is SED = 0.5793 vs. 0.5071, ρ = 0.3441 vs. 0.1038, R-ℓ₂ = 0.2103 vs. 0.3507, and Content Acc = 0.4200 vs. 0.2400 (Wu et al., 23 Aug 2025).

The ablation study states that a “No CoT” baseline collapses on sub-action metrics, adding a generic CoT improves slightly, SAR alone lifts SED and score accuracy substantially, and SAR + HPL yields the best across all datasets, with +2–4 % absolute (Wu et al., 23 Aug 2025).

These results support two claims made in the paper. First, structured reasoning contributes materially to sub-action decomposition and score estimation, rather than merely to stylistic explanation. Second, the RL stage contributes beyond SFT. The reported pattern of gains across FineDive, FineFS, and LOGO also suggests that the model’s advantages are not confined to a single benchmark regime.

7. Qualitative behavior and relation to adjacent hierarchical action research

The qualitative analysis centers on a synchronized-swimming example. In the <look> stage, HieroAction identifies “8 swimmers” and an “initial formation circle→line.” In the <recognition> stage, it segments the routine into phases such as “Approach” with synchronized leg-lifts and “Propulsion” with a sculling pattern. In the <assessment> stage, it assigns a leg-lift score of 8.2 with the rationale “minor height jitter” and a scull score of 8.5 with the rationale “excellent tempo, small lateral drift.” In the <answer> stage, it outputs “Overall routine technical 8.3, artistic 8.0, final 8.15” (Wu et al., 23 Aug 2025).

The paper characterizes this trace as “timestamped, phase-by-phase, rationale-rich” and contrasts it with “a black-box 8.2” (Wu et al., 23 Aug 2025). That comparison captures the central interpretability claim: the model is intended to provide actionable feedback by exposing intermediate structure rather than only terminal evaluation.

Within the broader landscape of hierarchical action modeling, HieroAction belongs to a family of methods that explicitly exploit multilevel action structure, but its target differs from action anticipation and zero-shot step grounding. HiMemFormer, for example, is a “Hierarchical Memory-Aware Transformer for Multi-Agent Action Anticipation” that combines global memory and a hierarchical local memory decoder for online multi-agent action anticipation, with evaluation on LEMMA using per-frame mAP over a 2 s anticipation horizon (Wang et al., 2024). HiERO-StepG, by contrast, frames procedural activities as hierarchies of steps and sub-steps and performs zero-shot step grounding through clustering, hierarchical agreement, temporal monotonicity, and post-processing on Ego4D (Zenotto et al., 29 May 2026). Compared with those systems, HieroAction is specifically centered on fine-grained action assessment with expert-aligned scores and explanations rather than future action prediction or step localization.

The principal novelties claimed for HieroAction are therefore twofold: “a domain-tailored chain-of-thought (SAR) that forces explicit sub-action decomposition and expert-style rationalization,” and “a hierarchical RL objective that jointly optimizes format fidelity, temporal precision, semantic correctness and score alignment under a KL-regularized policy update” (Wu et al., 23 Aug 2025). The paper concludes that, together, these yield “SOTA fine-grained scoring performance, robust sub-action segmentation and fully interpretable feedback” across diving, figure skating, and artistic swimming (Wu et al., 23 Aug 2025).