Socially Interactive Agents (SIAs)
- Socially Interactive Agents are embodied systems combining multimodal processing, real-time responsiveness, and social-context sensitivity for natural human-like interactions.
- They integrate speech, gesture, facial expression, and biosignals to dynamically adapt behavior and enhance user engagement.
- SIAs have diverse applications from neurorehabilitation to knowledge transfer, emphasizing trust, ethical design, and effective social presence.
Socially Interactive Agents (SIAs) are defined in the literature as virtually or physically embodied agents that autonomously communicate with people and with each other in a socially intelligent manner using multimodal behaviors—verbal, paraverbal, and nonverbal—known from human-human interaction; more broadly, they are computer systems whose interaction mechanisms resemble human social communication rather than purely transactional interfaces (Benderoth et al., 27 Aug 2025, McDonnell et al., 2021). Contemporary SIA research spans virtual humans, social robots, embodied conversational agents, rehabilitation coaches, and agentic analytical systems for socially meaningful domains, but the recurring technical commitments are multimodality, real-time responsiveness, social-context sensitivity, and coordination between perception, decision making, and expressive behavior (Lin et al., 20 Apr 2025, Chen et al., 30 Oct 2025).
1. Conceptual scope and theoretical foundations
A minimal account of SIAs is insufficiently captured by chatbot-style language generation. The stronger formulation in recent systems work treats SIAs as embodied, interactive systems situated in a perceptual and behavioral loop with users, operating through multiple modalities such as speech, language, graphics, lip-sync, motion, affect, environmental understanding, gesture, biosignals, and vision, while maintaining low latency so that interaction remains socially legible rather than merely functionally correct (Lin et al., 20 Apr 2025). This framing aligns with earlier handbook treatments in which appearance, gaze, turn-taking, body orientation, facial expression, and visible attention cues are treated as core resources for social interaction rather than decorative surface properties (McDonnell et al., 2021).
Several strands of the literature push the concept beyond interface design toward socio-cognitive competence. Developmentally informed work argues that AI should study not only how conventions emerge in populations, but also the socio-cognitive abilities that allow an individual agent to enter an existing culture: joint attention, referential communication, imitation, role reversal, formats, and scaffolding (Kovač et al., 2023). A complementary social-identity perspective argues that once agents become socially situated, users classify them into groups, infer identities from names, accents, roles, and embodiments, and reorganize trust, cohesion, bias, and conformity around them; at the same time, current artificial agents remain outside the superordinate category “human,” so not all human social-identity mechanisms transfer symmetrically (Seaborn, 12 Aug 2025).
2. Embodiment, appearance, and social presence
Appearance is treated in the SIA literature as a foundational design dimension rather than a cosmetic one. It shapes locus of attention, makes mutual gaze and turn-taking legible, and establishes expectations about role, capability, personality, and interaction style through metaphor, modality, realism, and abstraction (McDonnell et al., 2021). The design space includes graphical, virtual, video-mediated, physical, and hybrid embodiments, with the recurrent finding that coherence among appearance, behavior, task, and context matters more than realism alone; stylized or cartoon-like agents can be preferable to photorealistic ones, while mismatches in stylization or capability can produce discomfort and reduce co-presence (McDonnell et al., 2021).
For physically embodied SIAs, the strongest survey evidence comes from a systematic review of 65 studies on socially interactive robots. Across the reviewed experiments, 63.1% were classified as solely positive, 15.4% as mixed positive, 15.4% as neutral, 1.5% as mixed negative, and 4.6% as solely negative with respect to the embodiment hypothesis; among 57 studies measuring agent perception, 43 favored physical embodiment, and among 57 measuring task performance, 37 favored physical embodiment (Deng et al., 2019). The review’s broader interpretation is that physical embodiment is especially useful when effectiveness depends on social presence, compliance, motivation, shared-space interaction, or relationship maintenance, and less decisive when the task is primarily informational or privacy-sensitive (Deng et al., 2019).
Embodiment alone, however, does not solve multi-party interaction. An in-the-wild museum study with Furhat and MetaHuman, involving participants in dyads, triads, and larger groups, found no significant effect of a group-sensitive conversation design on perceived conversational satisfaction, despite explicit adaptation between singular and plural address; the authors interpret this as evidence that multimodal strategies beyond linguistic pluralization are needed for group-aware SIAs (Müller et al., 12 Jun 2025).
3. Architectures and infrastructure
Recent SIA infrastructure work treats multimodal orchestration as a primary systems problem. Estuary is presented as an open-source multimodal framework for low-latency, real-time SIAs built around five explicit design principles: flexible interoperable modules, platform agnosticism, off-cloud capability, multimodal processing support, and open-source availability (Lin et al., 20 Apr 2025). Its client-server architecture uses a Python server, SocketIO, JSON, and modular Stage and Pipeline abstractions over typed DataPackets such as AudioPacket, TextPacket, and ImagePacket, with SDK support for Unity and example clients for Apple Vision Pro, Meta Quest 3, and desktop deployment (Lin et al., 20 Apr 2025). The associated user-centered study further argues that framework design for SIAs is itself an HCI problem, with ease of integration, latency, privacy, reproducibility, sustainability, and multimodal extensibility as first-class evaluation criteria (Lin et al., 20 Apr 2025).
A second architectural lineage extends classic embodied conversational-agent pipelines. GRETA is a modular, real-time, multiplatform platform built on SAIBA, with FML for communicative intentions and BML for multimodal behavior specifications, and with explicit support for adaptation to external features and incremental dialogue processing (Grimaldi et al., 23 Jan 2025). Its architecture integrates user input, signal processing, signal interpretation, user-state interpretation, intention planning, behavior planning, realization, and rendering; it can combine symbolic planning, appraisal-based models such as FAtiMA, reinforcement-learning dialogue selection, and frame-level adaptive control, while handling modalities including speech, gaze, facial expression, gesture, proxemics, and social touch (Grimaldi et al., 23 Jan 2025).
A third line focuses on real-time learned nonverbal behavior. ReNeLiB integrates multimodal feature extraction from webcam and microphone, neural listening-behavior generation, and visualization for both FLAME-based and ARKit-based agents (Don et al., 2024). The pipeline is streaming-oriented, built around timestamped audio and visual features, an autoregressive listener-behavior generator based on “Learning to Listen,” and rendering backends including Blender and web-based ARKit-compatible agents; on the reported i9/RTX 4090 setup, the complete system loop is about $0.04$ s and can continuously deliver output at 30 fps or 24 fps after an initial delay of about 1 s (Don et al., 2024).
4. Adaptive behavior generation and alignment
A major technical trend is the shift from scripted multimodal output toward adaptive, dyadic behavior generation. AMII formulates adaptive facial-gesture synthesis as joint modeling of intra-personal and inter-personal dynamics for both speaker and listener roles, using 100-frame histories of speech and facial behavior, modality-specific memory encoders, cross-attention within each person, and cross-attention across interlocutors (Woo et al., 2023). On the French subset of NoXi, AMII achieves MAE $0.156$ and RMSE $0.197$, improving over the best baseline by MAE and RMSE, while also yielding the strongest reciprocal-adaptation resemblance on the paper’s emphasized DTW, Synchrony, and Entrainment Loop measures (Woo et al., 2023).
Another strand focuses on constraining socially problematic behavior before it is emitted. GALAD casts norm-sensitive action generation in interactive narratives as a POMDP , where is a value-alignment scoring function distinct from task reward, and uses action distillation from Delphi to suppress socially misaligned actions at the candidate-generation stage rather than through reward shaping at runtime (Ammanabrolu et al., 2022). On Jiminy Cricket, GALAD reaches completion $3.76$ and harmfulness $1.26$, amounting to a 4% task-performance increase over CMPS+ and a 25% reduction in socially harmful behaviors relative to CMPS, while also improving relative harmfulness from $0.04$0 or $0.04$1 in baselines to $0.04$2 (Ammanabrolu et al., 2022).
Interaction-level alignment has also been extended to user engagement. In socially driven dialogue, i$0.04$3MCTS and DPO are used to collect preference pairs from simulated future interactions, with engagement defined by downstream user reaction—full emotional disclosure in emotional support or donation amount in persuasion (Wang et al., 26 Jun 2025). The aligned models increase engaged rate in emotional support from $0.04$4 to $0.04$5 and average donation in persuasion from $0.04$6 to $0.04$7 without materially increasing turn count, indicating that long-horizon social outcomes can serve as a more direct alignment target than dialogue-act or knowledge-selection proxies (Wang et al., 26 Jun 2025).
Trust calibration has recently been framed as a multimodal generation problem. GPT-5.4 was used to generate augmented transcripts tagged for text, intonation, gesture, and facial expression at specified levels of ability and benevolence, yielding Random Forest classification accuracies of $0.04$8 for ability and $0.04$9 for benevolence on the generated datasets (Galland et al., 19 May 2026). The same work also shows that explicit gender prompting induces stereotyped mappings—male agents toward ability, female agents toward benevolence—and that human participants reliably distinguish low from medium/high intended trustworthiness levels, although medium and high remain harder to separate perceptually (Galland et al., 19 May 2026).
5. Evaluation and measurement
Evaluation methodology has become a major research topic in its own right because many socially relevant competencies are contingent and process-dependent. Online Agent-as-a-Judge formalizes evaluation around a target policy $0.156$0, a judge policy $0.156$1, a world $0.156$2, a criterion set $0.156$3, and criterion-level outputs $0.156$4 (Ryu et al., 6 Jun 2026). Rather than scoring only whatever behavior happens to appear in a passive trajectory, the judge is instantiated as an in-world agent that uses the environment’s native protocol to create criterion-relevant social situations and then score the resulting target-produced evidence (Ryu et al., 6 Jun 2026). In a life-simulation benchmark with 32 designer-authored criteria, this raises criteria coverage to $0.156$5 from $0.156$6 for offline LLM-as-a-judge and $0.156$7 for offline agent-as-a-judge, and raises agreement with human labels to $0.156$8 from $0.156$9 and $0.197$0, with especially large gains on conflict/norm-violation criteria where passive methods rarely observe the trigger conditions (Ryu et al., 6 Jun 2026).
A complementary evaluation line targets interaction quality from social signals rather than only explicit questionnaires. In a museum field trial with a Furhat Robotics head acting as a service and information hub, 46 single-user interactions were retained for modeling, with 23 time series derived from body pose, facial expressions, and physical distance (Schiffmann et al., 3 Dec 2025). After feature extraction, SelectKBest feature selection, and leave-one-out cross-validation, tsfresh plus Logistic Regression achieved $0.197$1 accuracy and $0.197$2 ROC-AUC for classifying low versus medium/high user satisfaction, outperforming catch22 and handcrafted features (Schiffmann et al., 3 Dec 2025). This suggests that automatically extracted nonverbal signals can support post-hoc or future online monitoring of SIA interaction quality without manual behavior annotation (Schiffmann et al., 3 Dec 2025).
6. Applications, governance, and open problems
One of the clearest application domains is robotic neurorehabilitation. A concept paper proposes a home-capable upper-limb neurorehabilitation system around the PlanArm2 device in which multimodal sensing, signal interpretation, therapist-informed supervised learning, a Therapy Manager, and a socially interactive virtual coach form a closed adaptive loop; the SIA is the visible and audible interface through which the system motivates, explains, and socially regulates training, and it can adopt supportive or demanding styles based on inferred attentiveness, stress, pain, amusement, and performance (Arora et al., 2022). A subsequent proof-of-concept study realizes this architecture with the virtual coach Lydia, a Unity3D interface, VGG16-based attention and pain classifiers, and an ECG-based stress classifier; reported accuracies are $0.197$3 for attention/distraction, $0.197$4 for pain/no-pain, and $0.197$5 for stress/not stressed, and a pilot with 18 healthy participants found a downward trend in performance deviation across sessions together with questionnaire evidence that the agent did not distract and positively impacted engagement (Arora et al., 2024).
A different application line positions SIAs as organizational knowledge-transfer facilitators. In that formulation, SIAs are not passive repositories but empathic, socially intelligent agents that help employees externalize tacit knowledge through natural-language dialogue, structured reflection, and links to organizational memory via LLMs, retrieval-augmented generation, and Chain-of-Thought prompting (Benderoth et al., 27 Aug 2025). The proposed workflow distinguishes co-creation, knowledge preservation, and knowledge application, with onboarding and expert-retirement interviews as flagship scenarios; the decisive enabling factor is not retrieval alone but trust, supported by transparency, personalization, feedback, and the perception that contributions are both valorized and safeguarded (Benderoth et al., 27 Aug 2025).
Across domains, the literature remains marked by unresolved human-factors and governance issues. Group-aware dialogue cannot be reduced to pluralized language alone, gendered multimodal generation can reproduce stereotyped trust cues, and social identity processes mean that names, accents, embodiments, and roles can trigger in-group/out-group dynamics with consequences for trust, bias, and compliance (Müller et al., 12 Jun 2025, Galland et al., 19 May 2026, Seaborn, 12 Aug 2025). At the same time, several influential SIA systems remain conceptual, architecturally clear but under-specified in their adaptive policies or only lightly validated, which leaves open questions about failure modes, long-term adherence, privacy, bias, transparency of adaptation, and the conditions under which socially fluent behavior becomes manipulative rather than supportive (Arora et al., 2022, Benderoth et al., 27 Aug 2025).