InSituTale: Augmented Physical Data Storytelling

• InSituTale is an augmented physical data storytelling system that maps real-world object interactions, such as lifting and aligning, to visualization commands.
• It combines RGB-D object tracking with a Vision-LLM for rapid, semantically-driven event detection, enabling near-real-time interaction.
• User studies show that the system enhances improvisation and engagement by unifying physical props with digital charts in a coordinated mixed-reality frame.

Searching arXiv for the primary paper and closely related work on in-situ storytelling and augmented presentation. InSituTale is a research prototype for augmented physical data storytelling: a presentation paradigm in which a presenter manipulates visualizations through interactions with real physical objects rather than relying only on speech or free-hand gesture. In the reported system, picking up, moving, grouping, aligning, or changing the state of an object can trigger canonical visualization commands such as show/hide, scale, compose/decompose, select data point, select data series, change chart type, change data source, and hierarchical navigation. The prototype combines RGB-D object tracking with a Vision-LLM for semantic event detection, and was evaluated as a live presentation system with 12 participants (Takahira et al., 29 Jul 2025).

1. Definition, scope, and problem framing

InSituTale addresses a specific gap in augmented presentation systems. Prior systems have demonstrated the usefulness of gestural and speech control for live, improvisational data storytelling, but they have largely underexplored interaction with physical objects. This omission is consequential in settings such as product demonstrations, education with physical specimens, science demonstrations, and other presentations where objects are the natural referents of the data being discussed. InSituTale therefore treats objects not as passive stage props but as first-class control surfaces for visualization behavior (Takahira et al., 29 Jul 2025).

The paper defines this orientation as augmented physical data storytelling. Its central premise is that ordinary object manipulations already used in presentation rhetoric—placing, lifting, bringing closer, arranging, combining, opening, or revealing—can be mapped to visualization commands in a way that reduces cognitive load and increases intuitiveness. The intended result is a unified mixed-reality frame in which the presenter, physical props, and digital charts remain visually and semantically coordinated (Takahira et al., 29 Jul 2025).

The contribution is not limited to a single implementation. The work includes a survey of 31 data-driven presentation videos to identify recurrent visualization commands, workshops with nine HCI/VIS researchers to generate mappings between physical manipulations and those commands, a working prototype, and a user study with 12 participants. The workshops yielded 143 unique object-manipulation ideas, organized into 15 types across six categories (Takahira et al., 29 Jul 2025).

2. Command vocabulary and interaction taxonomy

The survey analysis distilled a concise command vocabulary that anchors the system’s interaction design. The identified commands were Show/Hide, Scale, Compose/Decompose, Select/Deselect Data Point, Select/Deselect Data Series, Change Chart Type, Change Data Source, and Hierarchical Navigation. The workshops then explored how physical manipulations could instantiate these commands during live storytelling (Takahira et al., 29 Jul 2025).

The resulting manipulation taxonomy spans six categories. Each category is associated with characteristic command mappings rather than a single fixed function.

Manipulation category	Typical forms	Commonly associated commands
Appearance-based	reveal contents, transform appearance, open/close	show/hide, change data source, change data type
Movement-based	lifting, relocating, rotating, changing distance to camera	scale, show/hide
Arrangement-based	adjust distance and alignment among objects, isolate one object	compose/decompose, select series
Gesture-based	tapping, shaking, pinching, drawing	scaling, highlighting
Affordance-based	opening a bottle, pulling straps, pressing a laptop key	object-specific transformations
Visualization-based	direct pointing to a visual	selecting small data points

Several mappings became especially prominent. Arrangement-based manipulations dominated compose/decompose; movement-based and gesture-based manipulations dominated scale; and visualization-based pointing was favored for selecting small data points. The paper’s design guidance emphasizes semantic coherence between the physical action and the visualization command, arguing that mappings are more memorable when the manipulation metaphorically matches the intended effect—for example, an opening action corresponding to a drill-down or reveal (Takahira et al., 29 Jul 2025).

This interaction taxonomy also functions as a constraint mechanism. Rather than enabling all possible commands simultaneously, the system supports per-scene command selection, thereby reducing ambiguity among similar motions such as lifting, pointing, and arrangement (Takahira et al., 29 Jul 2025).

3. Sensing architecture and event interpretation

The prototype uses a ZED Mini stereo RGB-D camera mounted on a tripod approximately 50 cm in front of the presenter and aimed at a tabletop workspace. A laptop with Intel Core i7 3.6 GHz, 32 GB RAM, NVIDIA RTX 3070 runs Unity and the primary perception modules, while a separate server with an RTX 3090 hosts the Vision-LLM. Reported latency is approximately 0.1 s from capture to visual change (SD 0.02), while Vision-LLM responses average 1.08 s (SD 0.067) (Takahira et al., 29 Jul 2025).

The software stack is heterogeneous but tightly integrated. Unity handles capture, augmentation, scene management, and visualization. YOLOv4, trained on COCO 80 classes, performs class-level object detection from RGB frames; depth is fused to estimate 3D positions; and a heuristic nearest-neighbor tracker assigns persistent IDs frame-to-frame. A tabletop plane baseline is estimated so that vertical displacement above the plane can be used to infer lifting. LightBuzz Body Tracking SDK tracks hands and the index finger in 2D screen space, with the presenter choosing a preferred pointing hand to reduce ambiguity. Haar Cascade-based face detection supports occlusion-aware layout (Takahira et al., 29 Jul 2025).

Semantic event detection is handled by Qwen-VL-Chat, described as an instruction-tuned open-source Vision-LLM. It processes one compressed frame per second through socket I/O using scene-specific prompts such as “Is the glass filled with red wine?” and returns a binary response. The system therefore distinguishes between two trigger classes: “hard” spatial triggers, derived from geometry and object motion, and “soft” semantic triggers, derived from Vision-LLM judgments of state changes such as banana peeled, glass filled with red wine, or presenter eating an orange (Takahira et al., 29 Jul 2025).

The interaction logic is rule-based over a sensed feature space comprising object bounding boxes, 3D positions, velocities, orientations, inter-object distances, plane-relative height, hand position, face bounding box, and Vision-LLM event flags. The paper describes this conceptually as a mapping from a feature space $X$ to a command set $C$, where presenter-configurable settings—such as active commands and preferred pointing hand—serve as additional constraints. Typical rules include:

• Show when an object appears in view and its depth is below a threshold.
• Hide when it leaves view or passes a depth threshold.
• Scale as an inverse function of object depth.
• Select series when an object is lifted above the tabletop plane.
• Compose when inter-object distance falls below a threshold.
• Change type or change data source when the Vision-LLM event flag becomes positive (Takahira et al., 29 Jul 2025).

A separate greedy view-management algorithm governs visualization placement. For each object, the system evaluates eight candidate positions and scores them by penalizing overlaps with the presenter’s face, other visualizations, and objects, while rewarding vertical alignment and temporal stability. Placement is smoothed by linear interpolation, and composite visualizations are centered above the contributing objects (Takahira et al., 29 Jul 2025).

4. Authoring model and presentation workflow

InSituTale is organized around two modes: authoring mode and presentation mode. In authoring mode, a presenter configures a sequence of scenes. For each scene, the author selects tracked object classes, maps them to specific chart prefabs, enables or disables commands, selects the pointing hand, and defines Vision-LLM prompts for semantic triggers. Templates and saved settings can be imported and edited. In presentation mode, scene transitions are advanced using a ring mouse or clicker, while interactions inside a scene remain free-form and improvisational. A private on-screen panel lists active commands, mappings, and transform rules to reduce memory load (Takahira et al., 29 Jul 2025).

The visualization repertoire includes pie, donut, bar (clustered/stacked), line, and radar charts, as well as annotations containing uploaded text or images. Concrete mappings implemented in the prototype include the following:

• A chart can be shown by placing a tracked object in view, and hidden by removing it from view or moving it beyond a distance threshold.
• Scale and detail-on-demand are driven by distance to the camera; moving an object closer can trigger a detail view.
• A data series can be selected by lifting the corresponding object above the tabletop plane.
• A data point can be selected by pointing at the visual mark with the registered index finger for a dwell duration.
• Charts can be composed by bringing objects together; horizontal alignment yields a clustered bar chart, while vertical alignment yields a stacked bar chart.
• A chart can be transformed when an authored Vision-LLM condition is satisfied, such as switching to a tasting-notes radar chart after wine is poured into a glass (Takahira et al., 29 Jul 2025).

The paper presents several example story kits. In a wine comparison scenario, two bottles and a glass are associated with pie, bar, radar, and line charts. In a fruit consumption scenario, bananas and oranges are linked to donut, bar, radar, and line charts, and peeling a banana can trigger a chart transformation. These examples are intended to demonstrate that object interactions can support both planned structure and live improvisation within the same presentation frame (Takahira et al., 29 Jul 2025).

The reported authoring effort for a short story was 15–25 minutes for 4–6 scenes. The system was designed for tabletop presentations, especially remote or studio-style settings, and requires neither markers nor instrumented objects (Takahira et al., 29 Jul 2025).

5. Evaluation and observed effects

The evaluation involved 12 participants (6 female, 6 male), aged 19–30, with varied AR familiarity; most had prior experience presenting data. The study proceeded in three phases: training for approximately 10 minutes, authoring for approximately 15–25 minutes, and presentation for approximately 10–15 minutes. Participants built their own narratives using physical props and then completed a 7-point Likert usability questionnaire followed by a semi-structured interview. The paper reports descriptive results rather than SUS or NASA-TLX statistics (Takahira et al., 29 Jul 2025).

The principal quantitative finding is that ratings clustered at the high end—typically 6–7—for statements concerning ease of learning, ease of use, naturalness of interaction, helpfulness of the private presenter panel, engagement, and storytelling value. The paper does not report means, standard deviations, or hypothesis tests for these questionnaire items, but it characterizes the response pattern as strongly favorable (Takahira et al., 29 Jul 2025).

The qualitative analysis identifies three recurring benefits. First, participants reported intuitive control: routine presentation motions such as picking up, moving closer, and pointing were experienced as naturally coupled to chart behavior. Second, they reported high utility for improvisation: scene structure handled coarse sequencing, while object interactions supported unplanned emphasis, comparison, and reveal operations. Third, they described the experience as more performative and engaging than clicking through slides, because the presenter, props, and visualizations occupied a single coordinated frame (Takahira et al., 29 Jul 2025).

The evaluation also surfaced limitations. Participants noted unintended triggers, particularly confusion between lifting and pointing; grip sensitivity, where the manner of holding an object could obscure features; and class-based ID ordering, which constrained improvisation when multiple instances of the same class were present. Vision-LLM triggers were considered expressive but slower, and their reliability depended on prompt phrasing; the paper specifically notes that third-person phrasing improved performance compared with first-person prompts. Lighting robustness was not formally tested (Takahira et al., 29 Jul 2025).

6. Limitations, failure modes, and practical constraints

InSituTale’s sensing strategy deliberately avoids markers and specialized electronics, which lowers setup cost and preserves the everyday character of the objects. That same design choice, however, makes the system sensitive to occlusion, grip, and instance ambiguity. Because object identity is maintained by heuristic nearest-neighbor tracking and detections are class-based, presenters may need to remove and re-place objects when IDs flip or tracking drops. The private presenter panel is included partly to mitigate this problem by exposing current mappings (Takahira et al., 29 Jul 2025).

The division between geometric and semantic triggers also creates a latency asymmetry. Hard spatial triggers based on depth, view state, proximity, or plane-relative height can operate in real time on the laptop GPU, whereas soft semantic triggers depend on Vision-LLM inference at 1 frame per second per scene. The paper therefore recommends using Vision-LLM triggers for semantically rich but relatively slow events, while reserving geometric triggers for instantaneous interactions (Takahira et al., 29 Jul 2025).

Several practical constraints follow from the reported setup. The prototype is designed around a tabletop workspace and a camera placed roughly 50 cm from the presenter. It is therefore best understood as a studio-scale or desktop-scale system rather than a room-scale AR environment. The authors identify future directions including multimodal input combining voice with object manipulation, audience-side evaluation, broader object diversity and hybrid tracking for more reliable multi-instance differentiation, co-located projection AR or HMD variants, and more direct authoring support such as prompt-writing assistance and teach-by-demonstration workflows (Takahira et al., 29 Jul 2025).

The paper also notes an ethical and operational consideration: the system captures video of both the presenter and the objects, and the Vision-LLM receives periodic frames and textual prompts. In sensitive settings, consent and data-management practices become relevant (Takahira et al., 29 Jul 2025).

7. Position within in-situ storytelling research

InSituTale is situated most directly within augmented presentation and visualization systems such as Augmented Chironomia, RealityTalk, VisConductor, and Body-Driven Graphics, but its distinguishing claim is that physical object interactions should be treated as central rather than peripheral input for live data storytelling (Takahira et al., 29 Jul 2025).

A broader reading of recent research suggests that the prototype also belongs to a wider family of in-situ narrative systems, although those systems target different tasks and media. Heteroglossia embeds crowd-powered story ideation directly inside Google Docs and returns role-played plot ideas as inline comments, defining in-situ ideation as idea generation initiated and consumed within the writer’s active drafting environment (Huang et al., 2020). ImaginateAR extends in-situ authoring to outdoor AR by combining precomputed scene graphs, open-vocabulary 3D asset generation, and LLM-driven speech interaction, thereby supporting site-grounded creation and revision in the physical world (Lee et al., 30 Apr 2025). Promisedland applies a related logic to mixed-reality attractions through a Diorama-to-Virtual workflow, Stewart Platform synchronization, and a site-specific spatial layout, binding narrative progression to venue geometry and embodied motion (Wang et al., 12 Aug 2025). In a different domain, the In-Situ mode of EyeSee lets objects or figures inside paintings narrate in the first person, demonstrating that thing-centered narration can improve focused attention, aesthetic appeal, reward, relatability, and believability in art engagement (Li et al., 2024).

This broader landscape suggests a common research trajectory: moving narrative control, interpretation, or ideation into the environment where the activity already unfolds, whether that environment is a text editor, an AR site, a mixed-reality attraction, a painting, or a live data presentation. Within that trajectory, InSituTale’s specific contribution is to show that physical props can function as a precise and expressive control substrate for visualization commands, allowing presenters to synchronize rhetorical action and visual analytics without abandoning live improvisation (Takahira et al., 29 Jul 2025).