Hybrid User Interfaces (HUIs)
- Hybrid User Interfaces (HUIs) are cross-device systems that combine conventional 2D and mixed reality components to enable complementary multimodal interactions.
- They leverage each device’s strengths—such as precise 2D control and immersive 3D spatial awareness—to optimize user experience in dynamic environments.
- Research in HUIs spans architectural models, optimization frameworks, and human-AI alignment, addressing challenges like content transfer, perceptual synchronization, and adaptive interaction.
Hybrid User Interfaces (HUIs) are an area of cross-device computing that leverages distinct benefits of heterogeneous interaction components (Hubenschmid et al., 5 Sep 2025). In the most systematically surveyed subset, conventional 2D interaction components are combined with mixed reality environments so that 2D devices contribute familiar and precise control while mixed reality contributes spatial awareness, direct 3D interaction, and immersion (Hubenschmid et al., 5 Sep 2025). Adjacent literatures broaden the notion further: some treat the interface as a mediator between the physical universe of the user and the cyber universe of the computer system (Florio et al., 2015), some as a multimodal fusion layer for speech and gesture (Grazioso et al., 2021), some as a browser-side distribution layer spanning devices and applications (Firmenich et al., 2019), and some as an active component in human-AI alignment and LLM-mediated assistance (Shi, 12 Feb 2026). The field is therefore terminologically fragmented, but a recurring principle is complementarity rather than mere multiplicity of devices or modalities.
1. Definition, scope, and delimitation
A recent synthesis deliberately modernizes the term and defines HUIs as an area of cross-device computing that leverages distinct benefits of heterogeneous interaction components (Hubenschmid et al., 5 Sep 2025). In that formulation, an interaction component is a standalone device set with the input and output capabilities needed for an application, such as a desktop with mouse and keyboard or a VR head-worn display with controllers. The same synthesis also treats the concept as fuzzy rather than sharply bounded, emphasizing attributes such as complementarity, heterogeneity, and codependence instead of a rigid membership test (Hubenschmid et al., 5 Sep 2025).
That framing clarifies a central distinction. Not every system that adapts to multiple devices is a full HUI in the stronger cross-device sense. The model based on Abstract Interaction Units (AIUs), for example, is explicitly characterized as multi-device adaptive UI modeling and a precursor to later HUI ideas, but not as a full hybrid interaction framework with simultaneous use of multiple devices, real-time distribution of tasks, or shared state across heterogeneous devices (Bertini et al., 2017). Conversely, some systems are called hybrid because they combine modalities within one interaction, as in speech-plus-pointing control, or because they combine 2D and 3D interaction regimes within one visualization workflow (Grazioso et al., 2021, Cho et al., 11 Dec 2025). This suggests that HUI denotes a family of related design strategies centered on complementary integration.
A common misconception is that HUI is synonymous with any distributed, multimodal, or intelligent interface. The surveyed literature argues against that equivalence by treating complementarity as the key criterion: devices or modalities are combined because each plays a distinct role that another component performs less effectively (Hubenschmid et al., 5 Sep 2025). Under that reading, a mirrored duplication of an interface is only one possible configuration, not the defining pattern.
2. Historical development and research lineage
The historical origin of the term is traced to Feiner and Shamash’s 1991 work on “hybrid user interfaces,” summarized as a pragmatic combination of heterogeneous display and interaction technologies so that low-resolution, wide-coverage devices and high-resolution, limited-range devices could work together (Hubenschmid et al., 5 Sep 2025). The same survey describes a roughly 30-year evolution from early feasibility demonstrations toward increasingly empirical, application-specific, and conceptually differentiated systems, and reports a PRISMA-based screening process that started from 579 records and yielded 83 relevant publications (Hubenschmid et al., 5 Sep 2025).
Several later lines of work anticipated or expanded this trajectory. The AIU-based approach to internet applications proposed abstract interaction units such as BrowseImage, BrowseText, InteractTable, FillList, and SelectChoice, composed through UML Activity Diagrams and rendered differently according to device properties such as RN(dev), CN(dev), scrolling capabilities, WE(dev), JE(dev), AA(dev), CD(dev), and TSA(dev) (Bertini et al., 2017). Its importance lies in the separation of interaction logic from presentation and in structural adaptation, including reorganizing a table into a summary-plus-details interaction when a device with 14 visible rows and 30 columns must display data organized as about 40 rows and 105 columns without horizontal scrolling (Bertini et al., 2017).
Another lineage shifted distribution to the client. The browser-based platform for distributed web browsing treats DOM elements as portable UI components harvested into UIObjects within a synchronized personal information management space, enabling both opportunistic and rule-based distribution of presentation and interaction across sessions, devices, and even applications (Firmenich et al., 2019). In that model, a browser becomes the HUI runtime rather than a passive renderer.
More recent work extends the lineage into mixed reality authoring and operation. ARTHUR distributes a single authoring workflow across desktop, tablet, and head-mounted display, using the desktop for configuration, the tablet for portable touch-based control, and the HMD for in-situ inspection and refinement (Lunding et al., 4 Jan 2025). The same design logic—task allocation according to device strength—is also the logic that recent surveys identify as central to HUIs more broadly (Hubenschmid et al., 5 Sep 2025).
3. Architectural patterns and formal models
The most explicit HUI taxonomy identifies eight continuous dimensions for systems that combine conventional 2D interaction components with mixed reality environments (Hubenschmid et al., 5 Sep 2025).
| Dimension | Values | Brief meaning |
|---|---|---|
| Configuration | mirrored, logical distribution, spatial distribution, migratory, remote control, dynamic lens, augmented displays, VESADs | How content and control are split |
| Temporal | parallel, serial, exclusive | Whether components are used simultaneously or sequentially |
| Relationship | single-user, multi-user individual, multi-user shared | How users relate to components |
| Range | near, personal, social, public | Spatial and social distance |
| Device Dependency | flexible, semi-fixed, fixed | Whether components remain meaningful alone |
| Space | co-located, remote | Whether components share the same place |
| Interaction Dynamics | 2D→MR, MR→2D, bidirectional |
Direction of interaction flow |
| Anchoring | component-coupled, free, dynamic | How MR content is spatially placed |
This taxonomy is descriptive rather than prescriptive, but it provides a formal vocabulary for comparing systems that would otherwise be grouped only loosely. It also shows that HUIs are not reducible to one device pairing or one usage pattern.
The most detailed optimization model in the corpus is AdaM, which casts UI distribution as a many-to-many assignment problem solved as a mixed integer linear program (Park et al., 2018). Its central decision variable is
and its objective optimizes a weighted sum of quality and completeness,
with empirical setting (Park et al., 2018). Compatibility is computed as , where device characteristics and element requirements are represented as 4-element vectors over visual quality, text input, touch pointing, and mouse pointing (Park et al., 2018). Completeness is modeled per user so that a globally efficient allocation does not leave a participant without access to required interface elements, and the formulation explicitly encodes permissions, accessibility, size bounds, and zero-value rules (Park et al., 2018). Reported optimization times are about 0.1 seconds for toy examples and about 1 second for a large scenario with 100 users and 220 devices, which is presented as sufficient for real-time reconfiguration in dynamic collaborative settings (Park et al., 2018).
A different formal tradition treats abstract interaction itself as the stable layer. In the AIU model, browsing and inputting are specified independently of presentation, while the final interface is generated from a comparison between what an AIU requires and what the target device can support (Bertini et al., 2017). The practical implication is that adaptation may require restructuring interaction rather than merely resizing content. That principle remains recognizable in later HUI work, even when the device set becomes more heterogeneous and the interaction spans multiple synchronized components.
4. Modalities, coordinated interaction, and device complementarity
A major strand of HUI research combines input modalities within a single command. The traffic-control meta interface based on the Put That There paradigm fuses speech and pointing so that an operator can organize camera views, control video, and route audio in a surveillance room (Grazioso et al., 2021). The architecture combines Microsoft Kinect 2 for gesture and pointing capture, LUIS for speech intent and entity recognition, OpenDial for multimodal fusion using rule-based and Bayesian-network reasoning, Unreal Engine 4 for the 3D environment, and FANTASIA for rapid development (Grazioso et al., 2021). Gesture processing distinguishes pointing from non-pointing motion, computes a line from shoulder through hand to infer a semantic target, and prunes gestures that ended more than 4 seconds before the start of speech (Grazioso et al., 2021). In an evaluation with 12 participants and 6 tasks, the reported weighted task completion rate was 0.83, with 52 successes, 16 partial successes, and 4 failures; the NLU success rate was 0.76, gesture recognition accuracy 79%, multimodal sentence error rate around 15%, and intent classification accuracy about 89–92% (Grazioso et al., 2021). The significance is not merely multimodality, but coordinated interpretation of asynchronous channels into a single intended action.
HyFinBall addresses a different complementarity problem: coordinated 2D and 3D interaction in semi-immersive VR (Cho et al., 11 Dec 2025). It supports four modes—dual 6DOF input devices, dual planar constrained 3DOF input devices, dual 2-finger multi-touch, and 3D hand and finger gestures—using small spherical “buttonballs” that can be held, rested on a desk, or palmed while the fingers remain available for touch (Cho et al., 11 Dec 2025). The system automatically switches cursor representation and available techniques when the device changes between 3D and 2D regimes. In the first formal study, with 12 participants, HyFinBall had the fastest overall completion times for cross-dimensional work, incurred no significant 3D penalty, reduced fatigue relative to 6DOF-only interaction, and was preferred by 10 of 12 participants for combined 2D/3D tasks (Cho et al., 11 Dec 2025). In the second study, with 16 participants, palming reduced 3D and touch response times significantly in workflows that alternated between 3D buttonball interaction and multi-touch (Cho et al., 11 Dec 2025). The reported benefit therefore lies not only in performance within each task space but in reducing switching overhead between task spaces.
ARTHUR extends complementarity across authoring phases and devices in human-robot collaboration (Lunding et al., 4 Jan 2025). Its workflow is divided into configuration on a PC, refinement in the physical workspace with HMD plus tablet, and operation on the HMD. The system supports 20 feedback types, 10 actions, and 18 conditions for authoring interaction spaces, controls, and information visualizations in AR (Lunding et al., 4 Jan 2025). A web interface built in Vue, an AR interface built in Unity for HoloLens 2, MQTT-based communication, robot adapters, a preview service, and an assembly service together create an extensible authoring architecture (Lunding et al., 4 Jan 2025). In a qualitative evaluation with five participants, all completed the replication task and were able to add and configure their own design components (Lunding et al., 4 Jan 2025). Here the hybrid character lies less in multimodal fusion than in distributing one creative loop across desktop, tablet, and situated AR.
5. Context-awareness, adaptation, and machine understanding
One influential formulation treats the interface itself as a safety-relevant mediator between the physical universe of the user and the cyber universe where computer services execute (Florio et al., 2015). In that model, the UI logs atomic action/time pairs into iCode, dynamically profiles the user, and places context collection, context analysis, situation analysis and identification, planning of corrective responses, and execution of selected software evolution into an autonomic-computing loop (Florio et al., 2015). The paper explicitly calls context collection the perception layer and uses the term apperception for context and situation analysis (Florio et al., 2015). It also gives concrete discomfort-detection rules, including the threshold #define FASTEST_AVG_ENTRY 330.0 /* more than three character per second means distress */, and illustrates situations such as S1 = "User is likely to have changed" and S2 = "User is likely to have been taken over by a computer" (Florio et al., 2015). Proposed responses include alerts, re-authentication, shutting down critical functionality, changing widget layout, and requesting a CAPTCHA (Florio et al., 2015). The implementation is explicitly prototypic and simplistic, but it establishes a strong HUI interpretation of the interface as an active sensing, reasoning, and adaptation layer.
The contextually proximate approach (CPA) proposes a different adaptive logic: shaping the interface through local, nearby, and situation-specific resources such as location, time zone or time of day, nearby events, user behavior patterns, and recency of information so that the interface feels familiar and keeps the user “in the flow” (Joshi, 2015). The paper argues that CPA can support both novice and expert users, but it also cautions that CPA should not be applied everywhere, must not clutter the UI, and must preserve user control (Joshi, 2015). Its contribution is conceptual rather than experimental; the paper does not present a formal quantitative evaluation (Joshi, 2015).
At the representation-learning level, ActionBert treats a UI as a multimodal, structured, action-conditioned object rather than just a screenshot (He et al., 2020). It combines vision, linguistic features, View Hierarchy or DOM information, and user interaction traces in a uni-stream transformer with pre-training tasks for Link Component Prediction, Consecutive UI Prediction, and Masked VH Text Prediction (He et al., 2020). The reported pre-training corpus comprises 60,328 user action sequences, 2.69M UIs, and 5.4M UI pairs, and the model outperforms multimodal baselines across downstream tasks by up to 15.5% (He et al., 2020). The broader significance is that HUI semantics can be inferred from the interaction among appearance, text, structure, and action history.
Generative-AI-oriented work pushes the same adaptive logic into multimodal LLM interfaces. One proposed hybrid interface model is a pipeline of User Input (Text / Voice / Image), Multimodal LLM Processing, Context Adaptation & Retention, and Response Generation (Text, Voice, Image) (Bieniek et al., 2024). A mobile-oriented architecture splits Input Collection, Local Processing, Cloud Processing, Context Storage & Retrieval, and Output Generation, while discussing KV cache compression, chunk-level memory optimization, model quantization to 8-bit or 4-bit, and the use of NPUs with up to 22× speedup over CPUs (Bieniek et al., 2024). The paper’s “interface dilemma” is whether interaction should be chat-based, GUI-based, voice-based, or immersive, and it positions a hybrid approach combining the simplicity of GUIs with the versatility of multimodal inputs as a practical solution (Bieniek et al., 2024). This suggests that context retention and modality switching have become central HUI concerns in the era of multimodal foundation models.
6. Human-AI alignment, agent separation, and future directions
In human-AI alignment research, the interface is explicitly treated as part of the alignment pipeline rather than an implementation detail (Shi, 12 Feb 2026). A reference model decomposes the process into a forward path, AI system → UI → human, and a feedback path, human → UI → AI system, with design stages for data sampling, visual transformation, interactive display, analysis and interaction, and feedback formation (Shi, 12 Feb 2026). The paper focuses on evaluation alignment, where the interface helps humans understand, verify, compare, and judge outputs, and uses case studies on Interactive Groupwise Comparison for RL-agent alignment and DxHF for LLM alignment to show how richer visual and interaction structures can improve annotation efficiency, reduce cognitive load, and support more informed feedback (Shi, 12 Feb 2026). The preliminary comparison of six interfaces, spanning pairwise preferences, Visually Cluster Ranking, IGC, HH-RLHF, InstructGPT, and DxHF, is presented as evidence that richer interfaces reshape the kind of training signal collected (Shi, 12 Feb 2026).
A related line embeds LLM guidance into existing workflows rather than replacing them. In the scholarly-knowledge setting, “Smart Suggestions” is a React-based frontend component backed by a Python microservice and GPT-3.5-Turbo, returning machine-readable JSON through function calling (Oelen et al., 21 Jan 2025). The design is explicitly non-intrusive: suggestions are hidden by default, shown on demand, and may be collapsed or disabled (Oelen et al., 21 Jan 2025). The paper distills 22 guidelines grouped into six pillars—Transparency, Control, Usability, Error management, Feedback and statistics, and System performance—and reports a small-scale evaluation with 6 ORKG curators in which use cases were rated highly useful, with perceived usefulness higher than perceived correctness (Oelen et al., 21 Jan 2025). Participants did not believe Smart Suggestions could replace human assistance (Oelen et al., 21 Jan 2025). The interface is therefore hybrid in the sense of shared authority: the AI suggests, the human validates and curates.
HIDAgent externalizes the agent even further by separating it physically from the device it controls (Bigham, 31 Jan 2026). The toolkit combines a RP2040 microcontroller, an HDMI to USB converter, and a CH340 USB-to-Serial bridge, costing less than $30, and exposes a Python library that operates HID-compatible devices by visual observation plus emulated keyboard and mouse input (Bigham, 31 Jan 2026). Five prototypes—Extensible UI Agent, Universal UI Data Collection, Screen Reader Anywhere, Cross-Device Interaction, and The Helpful Observer—are used to argue that “bring your own agent” enables new scenarios in which an agent maintains its own context and memory while moving across devices (Bigham, 31 Jan 2026). The paper simultaneously flags a governance problem: decoupling the agent from the target system may also make it easier to automate locked-down systems that still accept external keyboard and mouse input (Bigham, 31 Jan 2026).
A broader synthesis of HUIs raises future-oriented questions rather than offering a single end state (Hubenschmid et al., 5 Sep 2025). It asks what a HUI is now, whether there will still be HUIs as devices become more capable, and whether the field should move from device-centric thinking toward device-agnostic complementary interfaces (Hubenschmid et al., 5 Sep 2025). The same survey highlights persistent challenges—visual attention switching, content transfer, perceptual synchronization, latency, design complexity, evaluation complexity, and the relative underexploration of collaborative and remote setups (Hubenschmid et al., 5 Sep 2025). A plausible implication is that the long-term trajectory of HUIs is not toward one canonical device combination, but toward a more general theory of complementary interaction in which devices, modalities, agents, and adaptive logic are composed according to role, context, and task.
