---
title: Interaction Initiation System Overview
url: https://www.emergentmind.com/topics/interaction-initiation-system-iis
type: topic
---

# Interaction Initiation System Overview

Interaction Initiation System (IIS) denotes a class of mechanisms that govern how an interaction begins, but the term is used differently across research areas. In adaptive learning, it refers to the design problem of **who or what is allowed to initiate the interaction and at what moment**, especially for the delivery of scaffolds such as Eye Movement Modeling Examples (EMME) during debugging [2605.04868]. In human-robot interaction (HRI), IIS denotes systems that detect or decide the opening of an exchange from embodied cues, including body language, speech direction, face orientation, and sustained visual attention [2512.03991] [2605.10087]. In software interfaces, a related interaction-initiation role appears in iDian, which lets users initiate software operations by expressing intent in natural language [1010.3177]. The acronym is also used in a distinct statistical sense—**innovated interaction screening**—for high-dimensional nonlinear classification, where it does not denote interaction initiation at all [1501.01029]. This polysemy makes IIS a cross-domain term whose meaning depends on the surrounding methodological and application context.

## 1. Conceptual scope and domain variants

Across the interaction-oriented papers, IIS is fundamentally concerned with the transition from a latent intention or need to an explicit system-level response. In the adaptive-learning study, the central claim is that **the effectiveness of EMME depends on the initiation system that governs when support appears, not just on the content of the support itself** [2605.04868]. In HRI, the same general problem appears as detection of the moment at which a human intends to begin interaction, either by conversational opening behavior in public space or by nonverbal cues in domestic environments [2512.03991] [2605.10087]. In iDian, interaction initiation is framed as the user’s ability to begin a software action by typing a natural-language instruction rather than navigating menus or memorizing command syntax [1010.3177].

| Domain | IIS meaning | Core initiation question |
|---|---|---|
| Adaptive learning | Scaffold initiation mode | Who or what launches support, and when? |
| HRI conversation opening | Timing and opener detection | Should the robot wait, speak, or listen? |
| HRI IoI detection | Initiation of interaction detection | Has the user begun interacting with the robot? |
| Natural-language software control | Intent-to-action initiation layer | How is a user’s intention converted into software operations? |
| High-dimensional statistics | Innovated interaction screening | Which variables participate in interactions? |

A plausible implication is that IIS is best understood as a control layer over the onset of interaction, rather than as a property of content alone. The interaction may be pedagogical, conversational, robotic, or operational, but the decisive issue is the rule or mechanism that converts a potentially meaningful cue into an actual intervention, greeting, service transition, or software command.

## 2. IIS as scaffold initiation in adaptive learning

In the debugging study, scaffold initiation mode is explicitly treated as an IIS design problem. The key question is **not only what scaffold is shown, but who or what is allowed to initiate the interaction and at what moment** [2605.04868]. The study compares four conditions—**Teacher-initiated EMME (T-EMME)**, **Learner-initiated EMME (L-EMME)**, **Automated EMME (Auto-EMME)**, and **No-scaffold control**—which instantiate different assumptions about regulatory ownership: guided regulation, self-regulation, system regulation, and no initiation system, respectively.

The experiment used a **between-subjects design** with **120 undergraduate computer science students**, **30 participants per condition**, at a **European university**. Participants had all completed at least one introductory programming course, and the sample included **31 female** students with **Mean age = 20.78 years, SD = 2.73**. They completed **10 Python debugging tasks** involving logical bugs rather than syntax errors. The learning environment included a main debugging panel, an EMME player, video controls, and a **Disable Help** option. The EMME materials showed an expert’s screen recording together with overlaid expert gaze as a moving dot and trail; the authors report that they ultimately used **teaching assistants** rather than experts because the experts’ debugging behavior was too efficient and their gaze patterns too compressed to model effectively [2605.04868].

The primary performance measure was the **number of correctly debugged programs** out of 10. A one-way ANOVA showed a significant effect of condition, \(F(3,116)=132.44,\ p<.05\). Mean performance was **2.50 / 10, SD = 1.13** for Control, **6.93 / 10, SD = 0.83** for L-EMME, **6.56 / 10, SD = 1.16** for T-EMME, and **4.06 / 10, SD = 0.83** for Auto-EMME. **L-EMME and T-EMME** were the best performers, with no significant difference between them, \(F(1,58)=1.97,\ p>.05\). **Auto-EMME** was significantly worse than both human-mediated conditions, though all EMME conditions outperformed the control [2605.04868].

From an IIS perspective, the central result is that the quality of support depended on initiation mode. The study reports that **human mediated initiation whether teacher or learner consistently produced higher performance than automated triggering and more integrative engagement with the EMME material**, while **automated triggering based on sustained low pupillary activity was associated with disruptive behavioral patterns suggesting mistimed delivery** [2605.04868]. This directly supports the claim that a scaffold can become effective only if the IIS launches it at a cognitively usable time and in a usable way.

The study also reports that EMME **eliminated the performance advantage of prior programming knowledge**. In the control condition, prior expertise correlated with performance at \(r = 0.58,\ p<.05\). In the EMME conditions, the correlations were \(r=-0.04\) for T-EMME, \(r=0.31\) for L-EMME (non-significant), and \(r=-0.23\) for Auto-EMME (non-significant). The ANCOVA interaction terms were **T-EMME × expertise: \(\beta=-0.40,\ p=.006\)**, **Auto-EMME × expertise: \(\beta=-0.48,\ p=.007\)**, and **L-EMME × expertise: \(\beta=-0.22,\ p=.10\)** [2605.04868]. This suggests that IIS design affects not only mean performance but also how access to support is distributed across expertise levels.

## 3. Physiological triggering, behavioral signatures, and the timing problem

The automated condition in the debugging study operationalizes IIS as a physiological trigger policy. Eye tracking served both as a source of the EMME model and as a live sensing input. The automated trigger used the **Index of Pupillary Activity (IPA)** and was defined as a participant’s IPA falling **more than two standard deviations below their individual resting-state baseline** for **longer than 60 consecutive seconds**:

$$
\text{trigger if } IPA(t) < \mu_{\text{baseline}} - 2\sigma_{\text{baseline}} \quad \text{for } t \ge 60 \text{ s}
$$

This state was interpreted as **sustained low mental effort** or disengagement [2605.04868]. The paper’s core argument, however, is that this is an ambiguous IIS signal: low effort may indicate disengagement, but it may also indicate **productive fluency**.

Behavioral log data make this ambiguity empirically visible. Seven interaction measures were logged: number of pauses, average pause duration, proportion of video played, number of backward jumps, number of forward jumps, average scanpath length, and transitions between task and EMME. Only three were meaningfully related to performance: **more pauses → worse performance** with \(r=-0.41,\ p<.00001\); **more backward jumps → worse performance** with \(r=-0.52,\ p<.00001\); and **more task-video transitions → better performance** with \(r=0.32,\ p=.002\) [2605.04868].

Condition differences were sharply patterned. **Auto-EMME** produced **more pauses**, **more backward jumps**, and **fewer task-video transitions**, whereas **L-EMME and T-EMME** produced more transitions. The study interprets the transition pattern as **integrative engagement**, meaning that learners accessed the support when they were ready to use it and then returned to the task to integrate what they had seen. A significant ANOVA was reported for behavioral differences across EMME conditions in task-video transitions, \(F(2,87)=4.95,\ p=.009\), with pairwise comparisons showing **L-EMME > Auto-EMME** and **T-EMME > Auto-EMME**. The paper also notes that backward jumps were especially harmful in T-EMME, with interaction estimate \(=-0.112,\ p=.0008\) [2605.04868].

These findings have broader significance for IIS design. The paper concludes that **a single low effort physiological threshold is insufficient as a trigger criterion for complex problem solving support** and characterizes the automated system as lacking adequate context sensitivity [2605.04868]. This suggests that in adaptive systems, initiation logic based on a single physiological metric may be structurally underdetermined when the same signal can arise from both disengagement and efficient task execution.

## 4. IIS in human-robot interaction: conversation openings and nonverbal IoI detection

In HRI, IIS addresses the socially consequential problem of how a robot determines that interaction is beginning. One paper defines IIS as a machine-learning-driven module for deciding **when a socially interactive agent should open a conversation** and, ultimately, **who opens it first** [2512.03991]. A second paper presents an **Initiation of Interaction (IoI) Detection Framework** that does **not rely on hotwords/keywords**, but instead infers intention when the user either **speaks while facing the robot** or **looks at the robot for a sustained period without speaking** [2605.10087].

The museum study used a **Furhat robot from Furhat Robotics**, named **Mira**, deployed near the entrance of the **Deutsches Museum Bonn (DMB)** from **July 23, 2024 to August 15, 2024** as a public **service and information point**. Visitors were **not given instructions** beyond a poster explaining the study and data-protection procedure. Entry into the interaction area required consent by pressing a **red buzzer**. After preprocessing, the dataset contained **\(N = 201\) suitable single-user interactions**, with **88 female** and **113 male** participants. These yielded **26,675 labeled data instances**, distributed as **12,898 wait**, **6,542 listen**, and **7,235 speak**. The split was approximately **89.1% training** and **10.9% test**, corresponding to **23,775 training instances** and **2,900 test instances**, with the test set covering **22 interactions** [2512.03991].

The domestic IoI study used the authors’ **mobile monitoring robot** equipped with an **Azure Kinect** and implemented all components in **ROS**. Its perception pipeline combined **HARK** for sound source localization, **YOLOv7** for person detection, **DeepSORT** for tracking, and **MediaPipe** head pose estimation to infer whether a user’s face was oriented toward the robot [2605.10087]. The framework defines two initiation routes. In the **audio-and-vision-based IoI** route, the robot is in the **monitoring state**, a user begins speaking, the system estimates speech direction, associates that direction with a tracked person, and infers interaction if the localized speaker’s frontal face is oriented toward the robot. The person association rule is

$$
\hat{\mathbf{X}} = \arg \max_{\mathbf{X}_i} \langle \mathbf{X}_i, \mathbf{S} \rangle
$$

with an angular rejection threshold of typically \(\delta_l = 15^\circ\). The corresponding indicator is

$$
z_a(s,f,h)=
\begin{cases}
1, & \text{for } s=1,\ f=1,\ \text{and } h=1\\
0, & \text{otherwise}
\end{cases}
$$

where \(s=1\) means sound source detected, \(f=1\) means frontal face detected, and \(h=1\) means the person is tracked [2605.10087].

The **vision-based IoI** route handles silent initiation. If the robot detects a frontal face of a tracked person, it measures the elapsed time \(t_1\) and enters the **visual attention state** when

$$
z_{v1}(t_1)=
\begin{cases}
1, & \text{for } t_1 \ge \delta_{t_1}\\
0, & \text{for } t_1 < \delta_{t_1}
\end{cases}
$$

with typical values \(\delta_{t_1} \approx 2\text{--}3 \text{ seconds}\). After that transition, it measures \(t_2\) and enters IoI when

$$
z_{v2}(t_2)=
\begin{cases}
1, & \text{for } t_2 \ge \delta_{t_2}\\
0, & \text{for } t_2 < \delta_{t_2}
\end{cases}
$$

with \(\delta_{t_2} \approx 2\text{--}3 \text{ seconds}\). The visual-only IoI indicator is

$$
z_v(z_{v1}(t_1),z_{v2}(t_2))=
\begin{cases}
1, & \text{for } z_{v1}(t_1)=1 \text{ and } z_{v2}(t_2)=1\\
0, & \text{otherwise}
\end{cases}
$$

and the overall decision rule is

$$
z_{IoI}(z_a,z_v)=
\begin{cases}
1, & \text{for } z_a=1 \text{ or } z_v=1\\
0, & \text{otherwise}
\end{cases}
$$

[2605.10087].

The two HRI lines of work differ in formulation but converge on a shared IIS problem: interaction onset is treated as a temporally extended, socially ambiguous state rather than a single keyword event. The museum IIS predicts whether the robot should **wait**, **speak**, or **listen** during the **greeting period** [2512.03991]. The domestic IoI framework detects whether the user is initiating interaction through speech-plus-facing or prolonged facing alone [2605.10087]. In both cases, timing and initiation are inferred from embodied cues rather than from explicit trigger words.

## 5. Architectures and computational realizations

The museum IIS uses **frame-level body-language features** extracted with **Google MediaPipe**. The framework provides **543 key landmarks**—**33 body landmarks**, **468 facial landmarks**, and **21 landmarks per hand**—with \(x\), \(y\), \(z\) coordinates and a visibility score in \([0,1]\). The paper states the visibility rule as \(\text{if visibility} < 0.5,\ \text{assign zero coordinates}\). In addition, the system computes **52 facial expression shape descriptors**, producing feature vectors of **1,682 values** per instance [2512.03991].

Its timing pipeline has two stages: **Human Pose Forecasting** and **Action Classification**. The forecasting model is a **BlockRNN time series model** with input from the **last 10 instances** and prediction target of the **next 5 instances**. At **10 frames per second**, this corresponds to about **0.5 seconds** of lead time. The forecasting model achieved \(\text{RMSE} = 0.0426\). The second stage is a **support vector classifier (SVC)** that predicts **wait**, **speak**, or **listen**. For the first **10 data instances**, the action classifier is applied directly; from the **11th instance onward**, the full pipeline is used. The paper writes the decision logic as

$$
\hat{a}_t =
\begin{cases}
\text{SVC}(x_t), & t \le 10 \\
\text{SVC}(\text{BlockRNN}(x_{t-9:t})), & t > 10
\end{cases}
$$

The best SVM configuration used \(C = 1\), \(\gamma = \text{scale}\), **kernel = RBF**, and **class\_weight = balanced**; **Random Forest** was also tested, but the SVM performed better and was selected as the final action classifier [2512.03991].

The domestic IoI framework uses a ROS-based sensor-fusion architecture rather than a forecasting classifier. Its state transition model includes **Monitoring state**, **Vocal attention state**, **Visual attention state**, and **IoI state**. From monitoring, the robot can follow the path **Monitoring → Vocal attention → Visual attention → IoI** for the audio-plus-vision route, or **Monitoring → Visual attention → IoI** for the silent visual route. Backward transitions occur when evidence is inconsistent, such as speech without a matching face toward the robot, speech directed elsewhere, face detection failure, or background audio from TV or radio [2605.10087].

A related but distinct initiation architecture appears in iDian, presented as **an entire interaction solution, rather than a front user interface** [1010.3177]. Its **multi-layer structure** consists of **Syntactic Layer**, **Semantic Layer**, **Explainer**, **Executor**, and **Learner**. The workflow begins when the **user inputs a natural-language sentence**. The Syntactic Layer performs segmentation, dictionary lookup, conversion into integer indices, synonym handling, quotation extraction, number recognition, and special expression handling. The Semantic Layer then applies **NUMFORMAT transformation** together with **general rule-based transformation** and **specific rule-based transformation**, using wildcard characters `*N`, `?N`, `#N`, and `!N`. The **Explainer** tags the command structure as **Action (VP)**, **Primary Object (NP)**, **Secondary Object (NP, Optional)**, and **Conditions (PP, Optional)**, then sends an **“End”** message to trigger the **Executor**, which performs the operation in the application. The **Learner** handles unfamiliar words, suggests nearest known words using a thesaurus dataset, refreshes the dictionary if a suggestion is accepted, and supports online sharing of application-specific **special suits** [1010.3177].

These architectures differ substantially in substrate—forecasting over pose sequences, multimodal state transitions, and rule-based natural-language pipelines—but all instantiate IIS as a mediating layer between a cue of intent and executable system behavior.

## 6. Evaluation, limitations, controversies, and acronym ambiguity

The HRI systems report measurable but uneven performance. In the museum study, the **SVC action classifier** achieved **75.3% accuracy** on the test data, and the full timing classifier combining forecasting and classification achieved **73.6% accuracy**, **74% weighted precision**, **74% weighted recall**, **74% weighted F1**, **69% macro precision**, **69% macro recall**, and **69% macro F1**. The paper notes that **wait** performed best and **listen** performed worst, with the confusion behavior indicating that the **listen** class is the weakest part of the system [2512.03991]. In the domestic IoI framework, **AV-IoI** achieved **Precision: 82.35%**, **Recall: 70%**, and **F-measure: 75.68%**, while **Full-IoI** achieved **Precision: 86.36%**, **Recall: 95%**, and **F-measure: 90.48%**. The addition of the vision-only route improved performance substantially, especially recall, because it captured interaction attempts in which the user did **not speak** [2605.10087].

The major limitations are also explicit. The museum study emphasizes that **listen** is difficult because the distinction between listen and the other classes can be very subtle, involving minimal mouth movement, short and uncertain greetings, and noisy pose readings. It also notes that MediaPipe outputs can be noisy, that recording angle and movement type affect detection accuracy, that class imbalance is structurally hard to avoid, and that the **Type Classifier** for greeting-style selection was developed only theoretically and **not yet evaluated** [2512.03991]. The domestic IoI framework identifies **small voice / distant speech** and **face detection failure** as key limitations [2605.10087]. The debugging study identifies a different controversy: whether physiological adaptation can replace human judgment. Its answer is negative for the tested trigger policy, concluding that **low pupillary activity is ambiguous** and that **IPA < baseline − 2 SD for >60 seconds** was too simplistic for complex debugging [2605.04868].

A recurrent misconception across these literatures is that interaction initiation can be reduced to a simple trigger. The available evidence argues against that view. In adaptive learning, support timing based on a single physiological criterion can become disruptive [2605.04868]. In HRI, keyword-free initiation requires multimodal and temporally persistent evidence rather than a single verbal event [2605.10087]. In public-space social robotics, deciding whether the robot should greet or listen depends on a short but behaviorally rich **greeting period** rather than on a fixed script [2512.03991]. In software control, iDian is described not as a menu shortcut or speech recognizer but as a layered mechanism for mapping natural-language intention into executable operations [1010.3177].

Finally, the acronym itself is ambiguous. In high-dimensional statistics, **IIS** refers to **innovated interaction screening**, the first stage of **IIS-SQDA**, a two-step procedure for nonlinear classification [1501.01029]. There, IIS screens important interactions by transforming the original \(p\)-dimensional feature vector with class-specific precision matrices and examining variance differences in only \(p\) transformed coordinates rather than all two-way interactions of order \(O(p^2)\). The method comes with a **sure screening property** under stated conditions and is followed by **sparse quadratic discriminant analysis (SQDA)** for further selection and classification [1501.01029]. This usage is unrelated to interaction initiation in learning systems, HRI, or user interfaces. Any technical discussion of IIS therefore requires immediate domain disambiguation.

Taken together, the interaction-oriented IIS literature treats initiation as a design variable with substantive consequences for performance, coordination, and usability. Whether the system is deciding when to display EMME, when a robot should say “hi,” whether a silent user is initiating contact, or how a natural-language command should launch a software action, the decisive issue is the same: initiation is not merely the start of interaction, but a structured inference and control problem whose quality can determine whether the resulting interaction is supportive, natural, or disruptive.

Source: https://www.emergentmind.com/topics/interaction-initiation-system-iis