Interface Element Knowledge Graph

• IE-KG is a directed, labeled property graph that represents human-system interface elements and their interactions in nuclear power plant control rooms.
• It encodes spatial, semantic, and structural attributes of GUIs to enable automated mapping from textual procedures to executable action sequences.
• The framework supports intelligent automation and integrates with dynamic human reliability assessments by quantifying multi-action navigation steps.

The Interface Element Knowledge Graph (IE-KG) is a directed, labeled property graph formalism developed to model human-system interface (HSI) elements and their interactions in digitalized nuclear power plant (NPP) control rooms. It encodes spatial, semantic, and structural characteristics of graphical user interfaces (GUIs) and their constituent controls, enabling automated mapping from textual operational procedures to executable sequences of interface actions. The IE-KG supports intelligent automation, enables quantification of cognitive demands via multi-action step identification, and underpins integration with dynamic human reliability assessment (HRA) and real-time decision support systems (Xiao et al., 26 May 2025).

1. Ontological Structure and Semantic Roles

The IE-KG formal ontology specifies both node classes and edge types to comprehensively represent HSI components and their operational relationships:

• Node Classes:
  • Button: Clickable actuator elements that trigger discrete actions.
  • Slider/Knob: Continuous-value selectors for parameter adjustment.
  • Display/Readout: Textual or numeric fields reflecting process variables or state.
  • Menu/Tab/Module: Hierarchical containers organizing nested controls and panels.
  • Frame/Window: Top-level GUI constructs, often corresponding to simulator panels or dialog windows.
• Edge Types and Semantic Roles:
  • hierarchicalContainment ($v_{parent} \rightarrow v_{child}$): Models containment, reflecting GUI tree structure.
  • spatialAdjacency ($v_i—v_j$): Captures proximity relations for reasoning about interface layout.
  • functionalDependency ($v_i \rightarrow v_j$): Encodes enable/disable relationships driven by operator actions.
  • navigationPath ($v_i \Rightarrow v_j$): Represents interaction sequences required for multi-step navigation.

Nodes are annotated with semantic roles such as “container” (e.g., Window, Menu), “action” (e.g., Button), “readout” (e.g., Display), and “navigation” (e.g., Tab, Module), supporting both modeling and automation.

2. Graph Data Model and Notational Formalism

The underlying data model adopts a property graph representation analogous to Neo4j, featuring arbitrary key-value pairs on both nodes and edges. RDF-style triples can also be derived for interoperability:

• Node (LaTeX notation):

$$v = (\ id \in \mathbb{N},\ type \in \{\mathrm{Button}, \mathrm{Slider}, \mathrm{Display}, \ldots\},\ x,y \in \mathbb{R},\ W,H \in \mathbb{R},\ \mathrm{properties} = \{\mathrm{label}: \mathrm{String},~ \mathrm{level}: \mathbb{N},~ \mathrm{metaTags}: \ldots \}\ )$$

• Edge (LaTeX notation):

$$e_{ij} = (\ v_i, v_j \in V,\ \mathrm{relationType} \in \{\mathrm{hierarchicalContainment},~ \mathrm{spatialAdjacency},~ \mathrm{functionalDependency},~ \mathrm{navigationPath}\},\ weight \in \mathbb{R}^+\ )$$

• Example RDF Patterns:
  • $$\langle v_{42}, \text{rdf:type}, \text{iekg:Display} \rangle$$
  • $$\langle v_{42}, \text{iekg:hasLabel}, "2LBA10CP801C" \rangle$$
  • $$\langle v_{\text{panel}3}, \text{iekg:hierarchicalContainment}, v_{42} \rangle$$

This formalism enables flexible querying and integration with both graph database and semantic web technologies.

3. Encoding of Spatial, Semantic, and Structural Attributes

The IE-KG explicitly encodes three primary categories of properties:

• Spatial:

Each node includes screen coordinates $(x, y)$ and dimensions $v_i—v_j$0 for precise representation; spatialAdjacency edges are introduced by thresholding Euclidean distances between node centers.

• Semantic:

Node labels derive from interface or OCR-extracted text (“Nuclear Island System”, “2LABDW001”), with optional keyword vectors or ontology tags for fuzzy label matching and semantic search.

• Structural:

GUI hierarchy is modeled via hierarchicalContainment edges—following the structure Window $v_i—v_j$1 Module $v_i—v_j$2 Panel $v_i—v_j$3 Control—and a “level” attribute (e.g., 0 for top-level, incrementing with depth).

The graph thus captures both logical organization and physical layout, supporting complex interface reasoning.

4. Algorithmic Construction from Interface Specifications and Operator Trace Data

Constructing the IE-KG is a multi-phase process leveraging both system metadata and dynamic operator interaction traces:

1. Initialization: Empty graph $v_i—v_j$4 is created.
2. Screen Definition Parsing: Simulator GUI definitions are traversed; for each control, nodes are added with extracted id, type, label, and spatial attributes.
3. Interaction Trace Processing: Operator click events are mapped against existing nodes; if unregistered, new nodes are created via OCR label extraction and coordinate annotation.
4. Hierarchy Resolution: Parent-child relationships are established via frame identifiers and metadata.
5. Spatial Adjacency Formation: Proximal node pairs introduce spatialAdjacency edges based on a center distance threshold.
6. Functional Dependency Inference: GUI metadata and multi-action traces serve to define enabled/disabled relationships.
7. Navigation Path Weight Assignment: Weights for navigationPath edges reflect unit click cost or may utilize Fitts’ Law for motor/action cost modeling.
8. Database Population: The final graph is committed to a backing store (e.g., Neo4j or RDF triple store).

This workflow enables both the accurate reconstruction of graphical interface layouts and population from observed use patterns.

5. Mapping Textual Procedures to Executable Interface Paths

The IE-KG supports direct automation of control room procedures through graph-driven translation from unstructured text to ordered node-action sequences:

• Text Parsing & Label Extraction: Keywords denoting target elements (e.g., “2LBA10CP801C”) are extracted from procedural text.
• Node Lookup: Label embeddings guide selection:

$v_i—v_j$5

• Path Finding: The shortest navigation path is computed:

$v_i—v_j$6

where $v_i—v_j$7 is the current top-level node.

• Multi-action Step Detection: Steps requiring path length $v_i—v_j$8 are automatically flagged as multi-action (cognitively demanding).
• Query Syntax Example:

$v_i \rightarrow v_j$0

• Result: The system produces an execution sequence of node–edge pairs suitable for robotic process automation or simulation.

6. Integration for Cognitive Load and Human Reliability Assessment

IE-KG operates as an integration nexus for automated execution, cognitive modeling, and dynamic HRA:

• HTRPM Tracker Integration: Raw click events, screen contexts, and window identifiers directly seed and validate the graph’s node structure.
• Execution Engine Support: Mapped navigation paths are replayed as operator-like click and keystroke sequences at annotated GUI coordinates.
• Cognitive Load Quantification: The presence and frequency of multi-action steps (longer navigation paths) serve as a proxy cognitive load metric; cumulative path lengths are inputs to task time modeling.
• Human Reliability Analysis (HRA): By coupling with models such as ACT-R or via Fitts’ Law ($v_i—v_j$9), IE-KG delivers empirically grounded step decomposition for error probability estimation. In frameworks such as COGMIF and DRIF, it replaces manual task analysis, enabling fully automated, dynamic HRA pipelines.

7. Illustrative Fragment: Nodes and Edges

An exemplary IE-KG fragment demonstrates typical entity and relationship structures:

id	type	label	x	y	W	H	level
v1	Module	Flowchart Module	100	20	300	600	0
v2	Panel	Nuclear Island System	150	100	250	400	1
v3	Display	2LABDW001	200	150	150	100	2
v4	Display	2LBA10CP801C	220	180	120	50	3

Edges:

• (v1→v2, hierarchicalContainment)
• (v2→v3, hierarchicalContainment)
• (v3→v4, hierarchicalContainment)
• (v4—v5, spatialAdjacency, d=30) (if v5 is a sibling Display)
• (v3⇒v4, navigationPath, weight=1)

This fragment demonstrates multi-level containment, spatial proximity, and navigation sequence relationships as encoded in the full graph structure (Xiao et al., 26 May 2025).