Assessment Hierarchical Task Networks (A-HTNs)

• Assessment Hierarchical Task Networks (A-HTNs) are an extension of classical HTNs that integrate modular, automated assessment routines into the task decomposition structure.
• They employ both task-level and action-level metrics—such as trajectory accuracy, orientation similarity, and dynamic time warping—to measure user performance precisely.
• Empirical evaluations on serious game scenarios demonstrate A-HTNs achieve over 90% correlation with expert assessments, confirming their reliability for complex multi-agent tasks.

Assessment Hierarchical Task Networks (A-HTNs) extend classical Hierarchical Task Networks (HTNs) by embedding automated, fine-grained performance assessment logic directly within the task decomposition structure. Developed to address deficiencies in standard HTNs—specifically, the absence of quantitative evaluation, user-specific scoring, and continuous (rather than Boolean) feedback—A-HTNs provide a formalism and runtime mechanism for measuring user performance in complex, multi-agent 3D serious games. Each primitive task node in an A-HTN is augmented with modular assessment routines that compare user-generated object trajectories or manipulations against reference data from subject-matter experts (SMEs), yielding normalized scores integrated into a composite evaluation. This framework enables automated, high-fidelity assessment of both task-level and action-level behaviors for single or multiple users, with empirical results demonstrating correlations to expert human raters above 90% for representative laboratory tasks (Desai et al., 2023).

1. Motivation and Distinction from Classical HTNs

Standard HTNs provide a declarative model for organizing high-level goals into a hierarchy of subtasks via abstract and primitive nodes, enabling structured planning and sequencing in domains such as robotics, autonomous agents, and simulated environments. However, classical HTNs are inherently descriptive: they specify what to do but not how well it was done. Core limitations include:

• No assessment module: Primitives are evaluated as "done/not done" without quantitative scoring.
• No actor specificity: All actions are generically assigned, lacking explicit responsibility mapping in collaborative scenarios.
• Binary granularity: Performance is not measured on continuous or qualitative scales.

A-HTNs are designed to retain the clear pedagogical decomposition of HTNs while equipping each primitive with an assessment pipeline operating on user trace data (object positions, orientations, attachment states, text entries) and comparing them against SME demonstrations or reference values (Desai et al., 2023).

2. Formal Structure and Node Specification

An Assessment HTN is formally a rooted, directed acyclic graph, where each interior node corresponds to an abstract task and leaves to primitive tasks. The main enrichment lies in the primitive node, defined as: $$N_i = \{\, T_i,\, I_i,\, O_i,\, U_i,\, W_i,\, P_i,\, G_i,\, A_i,\, F_i \,\}$$ where:

• $T_i$: Task label/name
• $I_i,\,O_i$: Input/output specifications
• $U_i$: User(s) assessed (single, group, individual-in-group)
• $W_i$: Node weight in final score
• $P_i$: Predecessor tasks
• $G_i$: Relevant game objects/joints
• $A_i$: Assessment mode (task-level, action-level, hybrid)
• $F_i$: Feedback type (real-time, final-score)

After all leaves are executed per the topology, per-node normalized scores $\Omega_i \in [0,1]$ are computed, which aggregate into an overall score using weighted summation: $T_i$0 The normalization $T_i$1 ensures $T_i$2 (Desai et al., 2023).

3. Modes of Assessment: Task-Level and Action-Level

A-HTNs support two primary assessment modalities, selected via field $T_i$3 per primitive:

• Task-Level Assessment: Object manipulations in $T_i$4 are scored using:
  • Orientation similarity: Based on quaternion deviation over time.
  • Position accuracy: Euclidean distance between user and reference trajectories, clamped to tolerance.
  • Attachment-time ratio: Fraction of correct attachment duration.
  • Collision penalty: Decrement per collision event, clamped to $T_i$5.
  • Text input: Exponential penalty on deviation from correct entry.
• Action-Level Assessment: Compares time-series motion trajectories (e.g., hands, head, joints) using:
  • Euclidean trajectory distance:

  $T_i$6 - Dynamic Time Warping (DTW):

  $T_i$7

The similarity score is then $T_i$8 with $T_i$9. For hybrid tasks, multiple metrics are combined (Desai et al., 2023).

4. User Assignment, Feedback, and Multi-Agent Structuring

The $I_i,\,O_i$0 field enables flexible attribution of task responsibility:

• Single user: Score assigned to solo participant.
• Group: Aggregate performance (sum or min) over all participants.
• Individual-in-group: Score specifically attached to a designated subgroup member, others as supporting actors.

Feedback $I_i,\,O_i$1 is available both as real-time and final-score modes, selectable per node. Dynamic dashboards (per-user or per-role) receive the appropriate scores at runtime (Desai et al., 2023).

5. Implementation Mechanisms

A-HTN is implemented as a graph-structured data model within a Unity3D/C++ engine (with PhysX and OpenGL), driven by the following runtime pipeline:

• Node execution: Topological scheduling activates enabled primitives, triggers relevant assessment routines.
• Data capture: SME reference traces (object states, skeletons at 60 Hz) and user data (via Leap Motion for hands, Kinect V2 for body) are buffered.
• Assessment module: On task completion (or interval), scoring routines compute $I_i,\,O_i$2 for each node using the prescribed formulas, logging and updating system feedback.

A schematic assessment pseudocode for a primitive: $I_i,\,O_i$3 (Desai et al., 2023)

6. Empirical Validation and Correlation with Expert Assessment

A-HTN was empirically evaluated on two 3D VR serious games:

• Hydrometer Experiment (single-user): Tasks include picking up, lowering, and manipulating a hydrometer. Pearson correlations between A-HTN and SME scores reached 91.8%, with Spearman and Kendall above 90% and 77%, respectively for overall performance.
• Collaborative Chemistry (multi-user): Tasks involve handovers and collaborative liquid manipulations. Real-time feedback was found accurate, minimizing the need for SME intervention.

The high correlation coefficients demonstrate A-HTN’s ability to replicate SME judgments at near-expert granularity for procedural laboratory tasks (Desai et al., 2023).

7. Strengths, Limitations, and Prospective Extensions

A-HTN offers several advantages:

• Unified model combining task decomposition and granular assessment.
• Fine-grained, continuous metrics capturing subtle deviations from expert procedures.
• Support for both individual and group assessment in collaborative tasks.
• Configurable feedback for both real-time guidance and post-task analytics.

Limitations include:

• Limited applicability to open-ended or creative domains where SME-referenced matching is insufficient; "creativity assessment" modules based on generative models or variance-allowing distances are a suggested extension.
• Initial focus is on serious games; adaptation for entertainment games would require new reference corpora.
• Absence of affective or cognitive measurement layers such as gaze tracking, though these can be modularly integrated.

A-HTN constitutes an extensible framework for automated assessment of user performance in complex, interactive educational and training environments, with fidelity and scalability comparable to expert human evaluation (Desai et al., 2023).