Action-Modification Interaction Design Space

• Action-modification is a design space that formalizes the mapping between user actions and system modifications, enabling systematic analysis and engineering.
• It defines a tuple-based framework quantifying interactions by linking actions, targets, modifications, and operational parameters.
• Applications span human-AI collaboration, XR guidance, and automotive UIs, offering measurable insights for interface innovation.

An action-modification interaction design space systematically organizes how user actions trigger modifications in a target system, such that the relationship between human input (“action”) and system response (“modification”) can be rigorously formalized, compared, and methodically engineered. Across domains as diverse as human-AI collaboration, data visualization, extended reality (XR) guidance, robotic control, and in-vehicle interfaces, the action-modification paradigm is employed to abstract interaction patterns, guide implementation strategies, and facilitate comparative analysis between systems (Tsiakas et al., 2024, Liu et al., 25 Jan 2026, Yu et al., 2024, Zhang et al., 9 Sep 2025, Jansen et al., 2022).

1. Formal Foundations and Schema Definition

The foundational abstraction in action-modification interaction design specifies each user interaction as a tuple or message that binds user action to system modification, parameterized by deterministic or data-driven rules. For visualization, Athanor formalizes the interaction specification as:

$I = (a, T_a, m, T_m, P)$

where:

• $a \in A$: user action,
• $T_a \in T$: action’s target,
• $m \in M$: corresponding modification,
• $T_m \in T$: modification’s target,
• $P$: optional parameter set.

The complete design space is defined as a Cartesian product: $$S = A \times T \times M \times T \times \mathrm{Param}$$, such that an action $a$ executed on $T_a$ triggers modification $m$ on $T_m$ under parameters $P$ (Liu et al., 25 Jan 2026).

In human-AI interaction, message passing is formalized as:

$$\mathit{msg} = \langle \mathit{sender} \rightarrow \mathit{receiver}, \mathit{action}, \{ \mathrm{modifiers} \} \rangle$$

with actions defined in terms of “provide” and “request” primitives, further parameterized by operation (create, select, map, modify) and data-types (input, output, feedback) (Tsiakas et al., 2024).

2. Taxonomies of Actions and Modifications

Visualization Domain

• User Actions ($A$): hover, click, double-click, context-click, zoom, brush, drag-and-drop, keyboard events.
• Targets ($T$): visual marks, reference components, extra widgets.
• Modifications ($M$):
  • Emphatic (highlight/select)
  • Reductive (filter/remove)
  • Annotative (tooltip, reference line)
  • Navigational (rescale_axes, panning)
  • Organizational (sorting, stacking)
  • Representational (visual_channel, change representation type) (Liu et al., 25 Jan 2026).

Human-AI Interaction

• Interaction Primitives:
  • $$\mathrm{P:provide}\{ X:\mathrm{type}^+, Y:\mathrm{type}^* \}$$
  • $$\mathrm{R:request}\{ X:\mathrm{type}^+, Y:\mathrm{type}^* \}$$
• Operation Types: create, select, map, modify.

Modification operations (e.g., modify-sample, modify-prediction) are formally distinct from selection/mapping actions by their explicit “modify” operation in the action definition (Tsiakas et al., 2024).

XR Motion Guidance

• Feedforward (Action) Dimensions: level of spatial indirection, update strategy (discrete, continuous, autonomous), perspective (1PP/Mirror/3PP), contextual cues.
• Corrective Feedback (Modification) Dimensions: information level (detection, magnitude, rectification), temporality (real-time, post-hoc), spatial placement, encoding modality (color, arrow, size, text, graph) (Yu et al., 2024).

Automotive UI

• Input Modalities (Actions): visual, auditory, kinesthetic, cutaneous, vestibular, olfactory, gustatory, cerebral, cardiac.
• Output Modalities (Modifications): same as above, mapped to human receiver side.
• Spatial Classes: nomadic (with user) and anchored (in cabin), with each location supporting subsets of modalities (Jansen et al., 2022).

3. Design Space Dimensions and Morphologies

Most action-modification design spaces are characterized by four or more principal axes:

Axis	Typical Range/Values
Intent	Provide ↔ Request
Content	Input ↔ Output ↔ Feedback
Operation	Create, Select, Map, Modify
Initiation	User-driven ↔ Model-driven ↔ Mixed
Modality	Visual, Auditory, Haptic, etc.
Location	Anchored (dashboard, seat) / Nomadic (AR)

For instance, in Athanor’s action-modification framework, state transformation is defined over visualization states $V = (P, C)$ where modifications alter control points or constraints. In vehicle HMI, a 3D morphological box (Zwicky box) captures all location, input, and output modality combinations (Jansen et al., 2022).

4. Representative Interaction Patterns and Implementation

Interaction design spaces enumerate and classify canonical interaction patterns as named sequences of action-modification pairings. In human-AI settings:

• Modify-prediction: user overrides model output via provide{Z:output.label, X:input.raw_data, Y:output.label} ← modify(Y,Z), map(X,Z).
• Sample-modification pattern: model requests modified sample, user provides via modify(X, M).
• Interactive reinforcement learning: model requests human override, user modifies action Y to A, shaping model policy accordingly (Tsiakas et al., 2024).

In visualization, actions such as brush+filter or click+change_representation correspond to elementary DOM/SVG updates and state constraint changes (Liu et al., 25 Jan 2026).

XR systems implement the full space as modular mappings from feedforward type (action cue) to corrective feedback (modification), tuned to user expertise, task complexity, and environmental constraints (e.g., continuous explicit ghost-arm cue with real-time color-change feedback) (Yu et al., 2024).

5. Comparative Models and Cross-Domain Extensions

Robotic and VLA systems apply the action-modification abstraction to integrate feedback and adjust action based on internal states. In TA-VLA, the “modification” is torque feedback, whose optimal integration—via decoder-side adapter and auxiliary prediction—enables causal alignment between proprioceptive state and physical outcome (Zhang et al., 9 Sep 2025).

Automotive UIs formalize the entire cabin as a combinatorial design space, with user sensory/actuator modalities both as action (input) and modification (output). Classification schemes, metrics (e.g., sensory bandwidth allocation), and mapping across space/modalities allow practitioners to identify novel opportunities, gaps, and multimodal synergies (Jansen et al., 2022).

6. Application Guidelines, Limitations, and Open Problems

A unified action-modification design space enables practitioners to:

• Systematically enumerate and select from existing interaction motifs.
• Browse, compose, or chain action-modification patterns to match task-specific or user-centric objectives.
• Anticipate implementation implications (e.g., widget choice, learning routine, data-flow hooks).
• Identify underexplored modalities or spatial locations for R&D (e.g., seat-embedded EDA feedback in vehicles).
• Quantitatively analyze trade-offs via formal metrics (e.g., state transformation mappings, bandwidth allocations).

Recognized limitations include a paucity of formal models capturing the cost–benefit trade-offs of pattern richness versus user cognitive load, and little empirical data on longitudinal effects of different action-modification deployments in real-world scenarios. Unexplored spaces such as adaptive or multimodal “modification” cues (e.g., visual+thermal feedback) and the mapping of action-modification design spaces to purely auditory or olfactory domains are highlighted as open areas for future work (Yu et al., 2024, Jansen et al., 2022).

7. Illustrative Examples Across Domains

Visualization

• “Hover over bar to show exact value in a tooltip” => (hover, bar, tooltip, bar, {format}).
• “Brush region to remove non-selected points” => (brush, point, filter, point, {predicate}) (Liu et al., 25 Jan 2026).

Human-AI

• “Model mis-predicts; user issues modify-prediction for post-hoc correction” (Tsiakas et al., 2024).

XR Motion Guidance

• “LightGuide” system: feedforward = abstract, continuous, 1PP; feedback = rectification arrows, magnitude-coded in area, real-time on body (Yu et al., 2024).

Robotic VLA

• “Button-pushing with torque feedback in π₀+obs+obj” (inject torque via MLP token, optimize composite MSE loss, closed-loop for failure recovery) (Zhang et al., 9 Sep 2025).

Automotive UI

• “Driver uses finger gesture on wheel (kinesthetic action), triggers seat vibration (cutaneous modification) for lane-keeping assist” (Jansen et al., 2022).

---

Action-modification interaction design spaces thus underpin a rigorous, operational vocabulary and methodology for engineering, comparing, and extending interactive systems across domains. The abstraction supports both theoretical analysis and practical implementation, and continues to inform emerging areas by its capacity to structure, taxonomize, and generalize the mapping from user actions to system modifications within complex, multimodal environments.