Engage: Multimodal Perspectives on Interaction

• Engage is a multidisciplinary construct defined by dynamic human-machine and social interactions with behavioral, cognitive, and affective dimensions.
• It operationalizes engagement through diverse metrics such as session time, facial cues, sensor readings, and modeled reciprocity.
• Empirical methods span probe-based, observer-coded, log-based, and ambient strategies, informing design principles and ethical applications.

Engagement is a multidisciplinary construct that, in human–machine interaction, is commonly defined as the process by which interaction partners establish, maintain, and end their perceived connection, while in behavior-centric modeling it is also operationalized as the intensity of observable interactions between an individual and an object (Salam et al., 2022, Bonometti et al., 2020). Across the literature, it is treated as a multi-faceted phenomenon spanning behavioral, cognitive, and affective components, but its concrete operationalization varies substantially with setting: it may denote reciprocal focus in a dyad, attention difficulty during self-guided video learning, persistence and quality of use in AI tutoring, social presence in immersive meetings, reflective participation in accessibility advocacy, or even structured participation in academic policy processes (Strazdas et al., 3 Dec 2025, Leng et al., 2 May 2026, Chen et al., 31 Jan 2026, Lee et al., 20 Mar 2025, Brogle et al., 2024, Wincott et al., 2 Apr 2026).

1. Conceptual foundations

In the HMI literature, engagement is usually framed as a dynamic relational process rather than a single scalar variable. The survey on automatic context-driven inference characterizes it through initiation, maintenance, and disengagement, and relates it to attention, involvement, rapport, turn-taking, proxemics, and social bonding (Salam et al., 2022). That survey also shows that engagement can be modeled as binary, multi-phase, ordinal, or continuous, depending on whether the target system must detect intention-to-engage, sustained participation, or disengagement trajectories.

A more explicitly mechanistic formulation appears in work that links engagement to behavior through incentive salience. There, engagement is operationalized via session time, play time per session, inter-session distances, number of sessions, activity index, and activity diversity, with environmental constraints such as hour-of-day, day-of-week, day-of-year, and geography treated as modulators of those observables (Bonometti et al., 2020). This formulation makes engagement a latent dynamical construct inferred from future behavior proxies such as churn probability, remaining survival time, remaining survival sessions, and absence duration.

A different but equally formal account is provided by IM HERE, which models engagement as reciprocal focus between two engageable entities (Strazdas et al., 3 Dec 2025). Its core formulation defines effort interpretation for a single channel as

$EI = T \cdot M \cdot C \cdot A \cdot P,$

where $T$ is contrast, $M$ magnitude, $C$ contribution, $A$ alignment, and $P$ receiver preference. Focus is then assigned to the entity with the highest total interpreted effort, and engagement between $A$ and $B$ holds if and only if both focus on each other. This framework foregrounds bilateral effort, subjective interpretation, and social-norm compliance rather than only observable activity.

These formulations are complementary rather than mutually exclusive. The survey literature emphasizes context and multimodality; behavior-theoretic work emphasizes latent salience and temporal dynamics; effort-based models emphasize reciprocity and miscommunication. Taken together, they indicate that engagement research is unified less by a single ontology than by a recurring concern with how attention, effort, persistence, and mutual orientation become observable and actionable in computational systems.

2. Operationalization and measurement

The measurement of engagement spans observer labels, self-reports, interaction logs, and ambient or physiological proxies. In self-guided video learning, EduGage operationalizes momentary engagement through a probe asking, “How difficult was it to pay attention during the last part of the lecture?”, using a 5-point Likert scale where 1 denotes effortless, automatic attention and 5 denotes heavy, deliberate effort to keep up; windows marked with “X” for explicit external distraction are excluded from modeling (Leng et al., 2 May 2026). Probes appear approximately every minute at natural stopping points, and a 44-second trailing sensor window is aligned to each response.

In facial-video engagement prediction, EngageNet uses a four-class taxonomy—Not Engaged, Barely Engaged, Engaged, Highly Engaged—over 10-second clips annotated by three expert annotators, with Weighted Cohen’s $\kappa$ values of $0.73$, $T$0, and $T$1 across annotator pairs (Singh et al., 2023). The dataset contains $T$2 clips totaling $T$3 hours from 127 participants, and explicitly combines educational stimuli with follow-up questions so that behavioral engagement labels can be related to correctness and response time.

In GenAI tutoring, engagement is measured from logs rather than direct labels. A two-stage pipeline segments 188,463 turns into 113,255 conversation sessions using a combined time-based and topic-based algorithm, with the segmentation validated at $T$4 on topic-bounded LMS logs (Chen et al., 31 Jan 2026). Ten session-level features are then used to derive four engagement types—Deep Engagement, Shallow Engagement, Routine-Learning Engagement, and Exam-Driven Engagement—with 10.4% of sessions classified as “shallow engagement” where copy-pasting behavior is prevalent. This operationalization shifts the focus from instantaneous attentional state to persistent usage pattern and transition structure.

Ambient interaction can also serve as an engagement signal. In the Aegis Canopy museum installation, engagement is defined as the sum of normalized infrared proximity readings across 24 sensorized nodes, thereby combining occupancy and active interaction in a single continuous reward used by a reinforcement learning agent (Meng et al., 2019). The per-minute engagement estimate is

$T$5

and a separate active-interaction count thresholds proximity at $T$6.

These measurement strategies imply distinct epistemic commitments. Probe-based systems privilege subjective immediacy; observer-coded datasets privilege externally legible behavior; log-based typologies privilege longitudinal usage patterns; ambient systems privilege low-intrusion group-level activity. The literature repeatedly treats no single measure as exhaustive, which is why multimodal and multi-level operationalizations recur across domains (Salam et al., 2022).

3. Computational modeling and inference

Computational engagement inference is dominated by multimodal pipelines that combine visual, auditory, physiological, temporal, and contextual signals. The general survey identifies common modalities such as gaze, head pose, facial Action Units, prosody, turn-taking, proxemics, physiology, and interaction logs, and organizes models into conventional classifiers, probabilistic temporal models, deep temporal architectures, and multimodal fusion schemes (Salam et al., 2022).

In EngageNet, OpenFace 2.0 is used to extract head pose, eye gaze, and facial Action Units, while MARLIN provides a 1024-dimensional self-supervised facial-video representation (Singh et al., 2023). The strongest reported single-model result on the test set comes from a Transformer using early fusion of gaze, head pose, and Action Units, reaching 67.61% accuracy. A late-fusion Transformer combining OpenFace-derived features with MARLIN yields 66.50% test accuracy, and training on Train+Validation raises that result to 69.29% on that split.

Sensor-rich inference appears in EduGage, which combines PPG, ECG, EDA, EEG, IMU, heart rate, temperature, eye tracking, and head pose in a context-informed gated fusion architecture (Leng et al., 2 May 2026). Each modality is encoded separately, gated with video-progress context, and fused through a normalized weighted average: $T$7 Across participant-based cross-validation, the resulting model achieves MAE $T$8, 83.75% within-1 accuracy, 73.93% binary accuracy, and 68.45% binary Macro-F1, outperforming sensor-free, statistical, deep temporal, foundation-model, and LLM-based baselines.

Real-time HRI systems often prioritize social-signal detection as an intermediate layer. Work with the android ERICA detects nodding, laughter, verbal backchannels, and eye gaze during the robot’s speaking turn, then fuses those signals with a hierarchical Bayesian engagement recognizer (Lala et al., 2017). The central fusion equation is

$T$9

with $M$0 annotator-character types found optimal. Using automatic detectors rather than manual social-signal labels reduces AUC only modestly, from $M$1–$M$2 to $M$3–$M$4.

Meeting analytics use simpler but still real-time architectures. “Engagement Detection in Meetings” defines a six-state spectrum—Disengagement, Relaxed engagement, Involved engagement, Intention to act, Action, Involved action—but evaluates a three-state subset using upper-body 3D posture and motion with a linear-kernel SVM (Frank et al., 2016). On 2,321 frames, with 500 used for training, the system reports 83.36% accuracy for frame-level discrimination of Disengagement versus Intention to Act, while Action is inferred when Intention to Act co-occurs with nonzero hand speed.

A distinct use of the term appears in graph representation learning. “ENGAGE: Explanation Guided Data Augmentation” uses a label-free explanation method called Smoothed Activation Map to preserve important nodes and edges during contrastive view construction (Shi et al., 2023). Here ENGAGE is not a human-state estimator but an augmentation scheme whose objective is still recognizably engagement-like in the information-theoretic sense of preserving key shared structure while discarding superfluous information.

4. Interactive design patterns for eliciting and sustaining engagement

A major strand of the literature does not merely infer engagement but designs for it. In adolescent e-learning, a pre-practice slider exposes a difficulty hyperparameter $M$5 in steps of $M$6, allowing learners to steer the difficulty of an AI-compiled exercise series before practice (Ooge et al., 2024). The linked “what-if” visualization updates live to show predicted mastery change if the learner solves the series correctly, whereas earlier “why” explanations proved too detailed for many adolescents and were more useful to teachers. The qualitative studies reported there suggest that what-if explanations and learner control can engage adolescents at the moment of decision-making rather than only before or after recommendation.

In accessibility reflection, a different pattern is used. A VR simulation of Durlacher Tor in Karlsruhe, Germany, models a $M$7 metre transportation hub with tactile paving, traffic, station platforms, an underground station, 107 pedestrian models, 19 different car types, and tram arrivals approximately every $M$8 minutes (Brogle et al., 2024). Rather than simulating disability, the system uses exclamation marks, particle effects, path visualization, information boards, and barrier animations to direct attention to three barriers: an abruptly ending tactile guiding strip, scooters cluttering a sidewalk, and a broken elevator. The design explicitly aims to focus attention on barriers in situ and to stimulate reflection and conversation about collective responsibility for accessible environments.

Turn-level encouragement is central in TalkTive, a conversational agent for older-adult neurocognitive screening (Ding et al., 2022). It distinguishes reactive backchannels such as “hmm” and “oh” from proactive backchannels such as “please keep going,” “anything else?”, and “no rush.” Timing is modeled with a Speaking Interval Detection module, an SVM-based reactive backchannel classifier, and a proactive score

$M$9

Expert ratings judged roughly 89% of generated backchannels appropriate, and qualitative results showed that proactive backchanneling was more appreciated than reactive backchanneling by older adults in this screening context.

Immersive meetings introduce re-engagement as a specific design problem. EngageSync uses gaze direction from Meta Movement SDK, speech activity, inactivity thresholds of $C$0 seconds, and pinch gestures to switch between live avatar-fixed transcriptions and per-avatar summaries of missed utterances (Lee et al., 20 Mar 2025). Panels are marked as read when gaze remains on them for $C$1 seconds, and the interface returns automatically to live mode after missed summaries are read. In user studies with small and mid-sized groups, EngageSync improved social presence and gaze allocation relative to table-fixed panels, and reduced re-engagement time while increasing information recall relative to non-adaptive avatar-fixed interfaces.

At the dialogue-policy level, engagement can be optimized against future user reactions. In socially-driven dialogue, interactive LLMs are aligned by treating the user’s post-interaction reaction related to the dialogue intention as reward, exploring conversations with a user simulator and interaction-adapted Monte Carlo Tree Search, then applying Direct Preference Optimization to chosen-versus-rejected utterance pairs (Wang et al., 26 Jun 2025). In emotional support, this raises engaged rate from 64.06% to 80.47%; in persuasion for good, it raises donation amount from $C$2. The central methodological claim is that engagement is often a cumulative, long-horizon property of an interaction rather than a turn-local attribute.

5. Domains, consequences, and broader uses

The research literature also treats engagement as a domain-specific societal process rather than only an interactional state. Overton Engage, for example, is a structured database of publicly documented academic policy engagement opportunities paired with a semantic matching system that links opportunities to researchers based on similarity between opportunity descriptions and publication records (Wincott et al., 2 Apr 2026). The database is classified by COFOG category and opportunity type, and in matching analyses across 5,059 UK opportunities, 97.3% have at least one high-confidence match with a paper from a UK institution. Here engagement denotes structured contribution to public decision-making through consultations, advisory roles, fellowships, and related mechanisms.

Educational use at scale introduces heterogeneity rather than a single engagement mode. In the multi-institution study of GenAI tutor usage, 60.8% of 11,406 students used the tutor at least once, but engagement quality varied sharply across session types, disciplines, and institution selectivity (Chen et al., 31 Jan 2026). Deep engagement and flexible transitions tended to co-occur, whereas shallow engagement showed a stronger self-loop, with $C$3. Students from highly selective institutions were more likely to exhibit deep and routine-learning engagement, while STEM courses showed more shallow and routine-learning sessions and less exam-driven usage. This literature treats engagement as a patterned educational behavior with implications for inequality in learning experience.

Engagement can also describe harmful or undesirable commitment. In Reddit research on early QAnon participants, 13,182 users identified through banned QAnon-focused subreddits produced 2,099,875 submissions and 10,831,922 comments across 63,697 subreddits from October 2016 to January 2021 (Engel et al., 2022). Their participation was highly concentrated in sympathetic communities, especially pro-Trump and anti-establishment or pro–free speech subreddits, and low-quality source links dominated in sympathetic and related communities. The paper’s interpretation is that these users were dedicated and committed rather than casual participants, which places platform moderation and cross-community spillover at the center of engagement analysis.

Public interactive systems provide another societal framing. In the museum deployment of the Aegis Canopy, engagement serves as the reward signal for learning behaviors that increase occupancy and active interaction in a public space (Meng et al., 2019). In VR accessibility work, engagement is explicitly not reduced to entertainment or immersion but tied to sensitization, transfer to lived urban context, and conversation about equitable participation (Brogle et al., 2024). These examples indicate that the term can designate participation worth encouraging, participation worth constraining, or participation whose ethical valence depends on the surrounding social objective.

6. Limitations, ethics, and open problems

A recurring limitation is that engagement is inherently noisy, partial, and context-sensitive. EduGage explicitly concludes that fine-grained engagement estimation is feasible but inherently noisy, and recommends lightweight combinations of behavioral and physiological signals rather than full multimodal instrumentation for practical systems (Leng et al., 2 May 2026). The HMI survey similarly emphasizes that context, role, embodiment, culture, and clinical variation all modulate cue interpretation, making general-purpose inference difficult (Salam et al., 2022).

Another limitation is the gap between designed engagement and validated impact. The VR accessibility simulation is a demo paper with no formal study, no participant demographics, no standardized measures, and no statistical tests (Brogle et al., 2024). Its contribution is the simulation and engagement concept rather than empirical findings. Similar caution applies to dialogue alignment with user simulators, where simulator bias, reward misspecification, and the difference between simulated and real users remain explicit concerns (Wang et al., 26 Jun 2025).

Ethical issues differ by domain but are persistent. Sensor-based systems raise privacy and consent issues around face, gaze, physiology, and background environments (Leng et al., 2 May 2026), while engagement-adaptive HRI requires careful handling of safe failover, transparency, and subgroup bias (Salam et al., 2022). Accessibility-oriented VR work explicitly rejects disability simulation because of risks of harm, misconception, and false equivalence between brief exposure and lived experience (Brogle et al., 2024). Policy-engagement infrastructures face collection bias and visibility bias because they only observe publicly documented opportunities, with UK coverage substantially more complete than elsewhere (Wincott et al., 2 Apr 2026).

Open problems therefore include richer context modeling, longitudinal and cross-session engagement, causal rather than correlational intervention design, standardization of annotation windows and ground truth, personalization without unfairness, and evaluation that separates mere activity from productive, respectful, or equitable engagement (Salam et al., 2022, Bonometti et al., 2020). The literature also suggests a broader conceptual challenge: engagement is not uniformly desirable. In some settings it is the condition for learning, accessibility reflection, social presence, or policy participation; in others it is the mechanism by which misinformation, manipulation, or exclusion persist. A comprehensive account of ENGAGE must therefore treat it simultaneously as a measurable state, a design target, a behavioral trajectory, and a normative problem.