Perceptive Shared Autonomy

• Perceptive Shared Autonomy is a framework that combines human intent and sensor-driven perceptions to regulate robot control with adaptive, interpretable, and safety-aware assistance.
• It employs diverse architectures—including continuous blending, discrete switching, and layered control—to enable bidirectional communication between operator goals and robotic execution.
• By leveraging perceptual variables such as uncertainty, trust, and agency, PSA dynamically adjusts autonomy levels to preserve human oversight and improve task outcomes.

Perceptive Shared Autonomy (PSA) denotes a family of shared-control paradigms in which autonomy is regulated by what the system perceives about the external world, the human, and itself. In this literature, perception is not limited to scene sensing: it also includes inferred human goals, task familiarity, trust, sense of agency, social context, kinematic risk, and epistemic uncertainty. Autonomy is then adjusted, blended, switched, or shaped so that human intent remains consequential while robotic assistance remains adaptive, interpretable, and safety-aware. PSA therefore appears as autonomy that is intent-aware and perception-informed in maritime inspection, uncertainty-aware in deep-learning-based manipulation, trust-aware in human-robot teams, agency-aware in assistive manipulation, and communicative through physically grounded robot dynamics (Caissutti et al., 5 Sep 2025, Lee et al., 5 Mar 2026, Li et al., 2023, Collier et al., 13 Jan 2025, Chen et al., 4 May 2026).

1. Definition and conceptual scope

The term does not denote a single algorithm. In the maritime formulation, PSA is defined as autonomy that is both “intent-aware (human-guided) and perception-informed (sensor-grounded),” with operator intent flowing top-down in language and perception flowing bottom-up through the autonomy stack (Caissutti et al., 5 Sep 2025). In SPIRIT, PSA is a system-level paradigm in which “uncertainty estimates from deep learning (DL) perception directly modulate the robot’s level of autonomy,” enabling semi-autonomous manipulation when perception is confident and haptic teleoperation when uncertainty rises (Lee et al., 5 Mar 2026). In trust-preserved shared autonomy, autonomy is “perceptive” because it infers a latent human trust state from time-stamped relational events and adapts both autonomy level and trust-management actions accordingly (Li et al., 2023). In assistive manipulation, the same term extends to monitoring sense of agency, where disagreement between executed and user command is treated as a proxy for a component of agency that could be monitored online (Collier et al., 13 Jan 2025). In IAGF-SA, PSA explicitly includes the inverse direction of perception: the human must perceive the robot’s intent and confidence through the robot’s anisotropic dynamic response (Chen et al., 4 May 2026).

Early shared-autonomy formalisms already contained the ingredients from which PSA later emerged. Shared autonomy via hindsight optimization modeled the user’s latent goal as a partially observable variable and selected actions that minimized expected cost-to-go under a belief over goals rather than waiting for certainty (Javdani et al., 2017). Human-robot mutual adaptation extended this logic to a mixed-observability setting in which the robot also maintained beliefs over human adaptability and mode, guiding adaptable users and complying with stubborn users to preserve trust (Nikolaidis et al., 2017). Taken together, this literature suggests that PSA is best understood as a system-level generalization of shared autonomy in which perception of latent human and system variables becomes a first-class control signal rather than a post hoc diagnostic.

A common misconception is that PSA is merely better intent inference. The surveyed systems contradict that reduction. Some regulate autonomy by uncertainty rather than intent certainty, some by trust, some by agency, some by social-space constraints, and some by embodied communication of robot intent itself (Lee et al., 5 Mar 2026, Li et al., 2023, Collier et al., 13 Jan 2025, Xu et al., 2024, Chen et al., 4 May 2026).

2. Architectural patterns and information flow

A recurrent PSA pattern is layered architecture. The maritime hull-inspection system explicitly uses three layers: a LLAMA-based natural language interface for high-level goals, a Behavior-Tree mission manager for interpretable task logic, and a multi-agent execution layer in which a human teleoperates a leader AUV while a follower AUV uses DRL to maintain formation, preserve visibility of a Point of Interest on the hull, and avoid collisions (Caissutti et al., 5 Sep 2025). The resulting data flow is bidirectional: operator goals descend through language and task structure, while sensor observations and execution status ascend back to the operator through BT conditions and LLM summaries.

SPIRIT instantiates a different but similarly stratified stack: a perception pipeline over point clouds and digital-twin partitions, an NTK-based uncertainty estimator realized as a mixture-of-GP experts, and a control layer that switches between semi-autonomous virtual-fixture guidance and haptic teleoperation (Lee et al., 5 Mar 2026). SARI and its predecessor frame PSA as a “Recognize–Replicate–Return” pipeline: an encoder infers a latent task from the history of human teleoperation inputs, a decoder generates assistive actions that replicate prior demonstrations for similar tasks, and a discriminator returns control when the current behavior is insufficiently similar to previously observed interaction patterns (Jonnavittula et al., 2022, Jonnavittula et al., 2021). IAGF-SA retains the classic modules of Goal Inference, Robot Decision, and Action Blending, but augments them with a physically grounded guidance field that continuously modulates apparent stiffness and damping to expose robot intent to the operator (Chen et al., 4 May 2026).

Across these systems, one architectural distinction is especially important: PSA does not always implement assistance by direct signal blending. Some systems do use the canonical shared-control law

$$u = \alpha u_{\text{auto}} + (1 - \alpha) u_{\text{human}},$$

or equivalent variants in Cartesian wrench space (Lee et al., 5 Mar 2026). Others rely on discrete mode switching, policy shaping, interpretable mission graphs, or effect-level composition in impedance control rather than continuous blending (Caissutti et al., 5 Sep 2025, Yousefi et al., 2023, Chen et al., 4 May 2026). This diversity reflects a central PSA design choice: whether perception should regulate autonomy continuously, discretely, hierarchically, or through dynamic embodiment.

3. Perceptual variables that regulate autonomy

PSA systems differ chiefly in what they treat as the relevant perceptual variable. One major class perceives intent or task identity. Hindsight-optimization shared autonomy maintains a belief over latent user goals and acts on the expected fully observed cost-to-go, which permits assistance before goal certainty is high (Javdani et al., 2017). Learned latent-action systems similarly maintain a belief over goals and condition the decoder on both robot state and goal belief so that the meaning of a low-dimensional human input changes from coarse goal-reaching to fine preference-level manipulation as confidence increases (Jeon et al., 2020). Repeated-interaction systems replace predefined goal sets with latent task recognition from the history of user inputs, allowing discrete goals and continuous skills to be learned from repeated teleoperation rather than specified a priori (Jonnavittula et al., 2021, Jonnavittula et al., 2022). Maritime PSA extends intent to natural language: operators issue high-level goals such as “inspect the port side” or “report anomalies near the stern,” which are mapped to structured mission logic and then executed by BT-managed and RL-based control (Caissutti et al., 5 Sep 2025).

A second class perceives uncertainty. SPIRIT formalizes this most explicitly: NTK-GP predictive covariance over the Lie algebra coordinates of a rigid transform is reduced to the scalar confidence proxy

$c = \mathrm{tr}(\Sigma),$

and autonomy is regulated by a robust binary policy: if $c \ge \beta$ then teleoperation is selected and virtual fixtures are disabled, otherwise semi-autonomy is enabled with virtual fixtures active (Lee et al., 5 Mar 2026). IDA generalizes the same principle to goal-agnostic intervention: the copilot intervenes only when its action has higher expert value than the human’s action across all candidate goals, thereby preventing entry into universally bad states without constant assistance (McMahan et al., 2024). DiSCo pushes uncertainty-aware PSA into sequence modeling: recent user actions seed and inpaint the reverse diffusion process, while the diffusion ratio $\gamma$, inpainting ratio $\rho$, and execution blending ratio $\beta$ tune conformity to the expert action manifold, alignment with user intent, and perceived responsiveness (Wang et al., 24 Mar 2026).

A third class perceives human relational state. Bayesian REM-based shared autonomy infers trust from time-stamped dyadic events—queries, instructions, refusals, explanations, and uncertainty signals—and updates a posterior over a latent trust variable that then drives autonomy level and trust-repair actions such as apology, explanation, or critical-state display (Li et al., 2023). Mutual-adaptation shared autonomy likewise infers human adaptability and mode under partial observability, guiding users who are likely to adapt and complying with users who are not, with disagreement penalties serving as a trust-preserving mechanism (Nikolaidis et al., 2017). Human-centered autonomous vehicle systems extend this relational perspective to driver state by combining external perception with glance region, cognitive load, activity, and system uncertainty to form a fused risk estimate that gates authority sharing and communication (Fridman, 2018).

A fourth class perceives agency, workload, and social context. In assistive manipulation, increasing robot autonomy improved task performance but reduced sense of agency, and the disagreement angle

$$\theta_d = \arctan2\!\left(\lvert a_{\text{output}} \times a_{\text{user}} \rvert,\; a_{\text{output}} \cdot a_{\text{user}}\right)$$

was proposed as a proxy for a component of implicit agency (Collier et al., 13 Jan 2025). AI-enhanced shared control for assistive arms instead emphasized legibility and explicit acceptance of adaptive DoF mappings, precisely because opaque or automatic mode changes were found to degrade user experience (Pascher et al., 2023). Socially-aware navigation treated pedestrians, group behavior, and user preferences as perceptual quantities, integrating proxemic fields and a User Preference Field into global and local planning (Xu et al., 2024). Collaborative perception in autonomous driving broadened the same idea to V2V, V2I, and V2X settings, where shared perceptual representations are used to overcome line-of-sight and field-of-view limits (Ren et al., 2022).

4. Control and learning formalisms

PSA is methodologically heterogeneous, but several formalisms recur. The most classical is partially observable decision making. Shared autonomy via hindsight optimization defines a latent-goal POMDP and approximates assistance with

$$Q_{HO}(b_t, s_t, a) = \sum_{g \in G} b_t(g)\, Q_g^*(s_t, a),$$

so that the robot acts with respect to a belief over goals rather than a single predicted goal (Javdani et al., 2017). Mutual adaptation uses a MOMDP in which observable robot state is separated from latent human adaptability and mode, enabling policy computation over beliefs about willingness to adapt (Nikolaidis et al., 2017).

A second recurring pattern is autonomy arbitration by blending, switching, or intervention. SPIRIT uses a binary authority factor within a blended wrench controller and maps that wrench to joint torques through $\tau = J^\top u$ (Lee et al., 5 Mar 2026). IDA replaces constant copilot intervention with a strict goal-agnostic intervention rule; only universally beneficial interventions are executed, yielding lower bounds relative to both pilot-only and copilot-only performance (McMahan et al., 2024). SARI learns a confidence-driven arbitration factor $\beta_t$ and proves uniformly ultimately bounded error under repeated interaction, with the crucial behavior that control is returned when the discriminator judges the current task unfamiliar (Jonnavittula et al., 2022). Hierarchical planning and policy shaping for articulated robots uses reward scalarization $c = \mathrm{tr}(\Sigma),$0 and conditions the autonomous policy on human actions and cVAE-based human embeddings rather than directly blending control signals (Yousefi et al., 2023). IAGF-SA composes conventional SA with a Cartesian impedance controller whose force term is modulated by an anisotropic guidance field, thereby turning assistance into a simultaneously physical and communicative phenomenon (Chen et al., 4 May 2026).

Learning architectures are equally diverse. SPIRIT couples sparse 3D U-Net-like perception with NTK-GP uncertainty estimation over digital-twin partitions (Lee et al., 5 Mar 2026). Maritime PSA uses DRL for leader-follower coordination, though the exact algorithm is not specified (Caissutti et al., 5 Sep 2025). SARI uses an encoder-decoder-discriminator structure over repeated teleoperation trajectories (Jonnavittula et al., 2022). DiSCo and IDA import diffusion models into shared autonomy, but in different ways: IDA uses a diffusion copilot trained on goal-masked expert demonstrations and intervenes selectively by value comparison (McMahan et al., 2024), whereas DiSCo plans action sequences with a conditional diffusion policy and aligns them to user intent through seeding and inpainting (Wang et al., 24 Mar 2026). Z-COACH uses CompILE with weak language supervision to discover interpretable task segments and then computes a skill-wise ZPD estimator from assisted versus unassisted trajectories (Srivastava et al., 27 Feb 2025).

5. Representative domains and systems

PSA has been instantiated in maritime robotics, aerial and ground manipulation, assistive arms, human-robot teams, socially-aware mobility, driving, and proximal teaching.

System	Perceptive variable	Autonomy mechanism
Maritime hull inspection	Natural-language goals, sonar-grounded PoI visibility, leader motion	LLAMA interface, BT mission manager, DRL leader–follower control
SPIRIT	NTK-GP perception uncertainty, $c = \mathrm{tr}(\Sigma),$1	Binary switching between semi-autonomy and haptic teleoperation
SARI	Repeated-interaction task familiarity	Recognize–Replicate–Return arbitration
Socially-aware navigation	User Preference Field, pedestrian proxemics, manipulability constraints	UPF-based A*, SS-MPC-DCBF local planning
Z-COACH	Skill-wise improvement under assistance	Skill-specific shared control for coaching
IAGF-SA	Goal confidence, singularity risk, human–robot directional agreement	Anisotropic impedance modulation layered onto SA

The maritime hull-inspection architecture is notable because PSA is explicitly positioned for “ports with dense traffic, clutter, human presence, and hull-proximal operations where the cost of mistakes is high” (Caissutti et al., 5 Sep 2025). SPIRIT targets aerial manipulation in mock-up industrial scenarios and reports that, in a user study of 15 participants, SPIRITv2 achieved a 100% success rate with a mean completion time of 61.86 s, whereas vanilla teleoperation achieved 86.66% and 160.46 s, respectively; under injected DL failures, uncertainty-aware switching achieved 100% success while vanilla virtual fixtures achieved 40% (Lee et al., 5 Mar 2026). Repeated-interaction PSA was validated on assistive robotic arms, where learning from repeated interactions matched existing methods on known tasks and outperformed baselines on new tasks such as drawer opening (Jonnavittula et al., 2022). DiSCo reported, in robotic block choice, success 0.80 with zero wrong-goal reaches, compared with success 0.45 for no copilot and 0.09 for a state-based diffusion copilot (Wang et al., 24 Mar 2026). Z-COACH, in a CARLA simulation of Thunderhill Raceway Park with $c = \mathrm{tr}(\Sigma),$2, reported significant improvements in lap time, jerk, and lane invasions when compared with self-practice (Srivastava et al., 27 Feb 2025). IAGF-SA, evaluated with 12 participants across three scenarios and two interfaces, reported lower disagreement, shorter task completion and alignment times, higher minimum manipulability, and the highest Communicative Assistance Scale and System Usability Scale among NA, SA, and IAGF-SA (Chen et al., 4 May 2026).

These examples also clarify that PSA is not confined to assistive manipulation. In autonomous driving, collaborative perception has been explicitly framed as a route from shared perception to shared autonomy, allowing planning and control beyond single-vehicle sensor range (Ren et al., 2022). In human-centered autonomous vehicles, the vehicle senses driver glance region, cognitive load, activities, and its own uncertainty, and uses fused risk estimates to regulate authority sharing and communication (Fridman, 2018).

6. Evaluation, limitations, and open questions

Evaluation practice is strongly domain-specific. Manipulation papers report success rate, completion time, end-effector forces and torques, NASA TLX, SUS, geodesic pose error, and disagreement measures (Lee et al., 5 Mar 2026, Collier et al., 13 Jan 2025). Assistive-arm and repeated-interaction systems report human effort, operating time, opposing time, and subjective ratings of recognize–replicate–return behavior (Jonnavittula et al., 2022, Jonnavittula et al., 2021). Navigation papers report minimum distance to pedestrians, trajectory length, linear and angular variance, lane invasions, jerk, expert distance, and manipulability (Xu et al., 2024, Srivastava et al., 27 Feb 2025, Chen et al., 4 May 2026). Maritime PSA proposes percentage of time the PoI is in field of view, average deviation from formation, and safety violations or near-collisions as future quantitative metrics, but its current evidence remains qualitative (Caissutti et al., 5 Sep 2025).

Several recurring limitations are explicit. Maritime PSA currently lacks full BT integration, realistic currents, communication noise, sensor uncertainty, and formal safety verification (Caissutti et al., 5 Sep 2025). SPIRIT assumes that teleoperation remains feasible when autonomy is disabled and notes that NTK uncertainty quality depends on features and partitioning (Lee et al., 5 Mar 2026). Mutual-adaptation models often assume static adaptability within a task and represent trust indirectly or through event models rather than as fully observed state (Nikolaidis et al., 2017, Li et al., 2023). Assistive shared-control work repeatedly reports that high task performance can coexist with reduced agency or reduced user satisfaction, so more autonomy is not automatically better (Collier et al., 13 Jan 2025, Pascher et al., 2023). DiSCo highlights inference latency and hyperparameter sensitivity, and the diffusion-policy results depend on careful tuning of $c = \mathrm{tr}(\Sigma),$3, $c = \mathrm{tr}(\Sigma),$4, and $c = \mathrm{tr}(\Sigma),$5 (Wang et al., 24 Mar 2026). Socially-aware navigation still faces sim-to-real transfer, dense-crowd occlusion, and the difficulty of reconciling user preferences with social norms (Xu et al., 2024).

Taken together, these limitations suggest four open research directions. First, PSA needs stronger grounding of high-level intent interfaces—whether LLMs, task embeddings, or trust estimators—into real-time perceptual state and verifiable control logic. Second, richer human modeling remains central: trust, sense of agency, workload, adaptability, and skill all influence whether assistance is effective or even acceptable. Third, safety mechanisms are unevenly developed; some systems use DCBFs or passivity-based teleoperation, while others rely on learned policies plus heuristics, indicating a gap between PSA functionality and formal runtime assurance (Xu et al., 2024, Lee et al., 5 Mar 2026). Fourth, PSA increasingly requires bidirectional intelligibility. The strongest recent shift is from autonomy that merely perceives the human to autonomy that also makes itself perceptible—through interpretable mission logic, XR feedforward, confidence displays, or embodied impedance modulation (Caissutti et al., 5 Sep 2025, Pascher et al., 2023, Fridman, 2018, Chen et al., 4 May 2026).

In that sense, PSA marks a conceptual broadening of shared autonomy. It is no longer only about inferring what the human wants; it is about regulating assistance with perceptual evidence, exposing autonomy’s own internal state in usable form, and preserving the human’s role as supervisor, collaborator, teacher, or learner within a technically heterogeneous but increasingly unified human-robot control paradigm.