reCAPit: Visual Analysis for Workshops

• reCAPit is an interactive system that integrates video, audio, gaze, and note data to analyze collaborative design workshops.
• It employs AI-based artifact extraction and temporal segmentation to align and summarize multimodal data streams effectively.
• The framework offers a transparent visual analytics pipeline linking high-level summaries to raw evidence for robust design evaluation.

Searching arXiv for the reCAPit paper and closely related material. reCAPit is an interactive visual analysis system and a modular framework for understanding collaborative design processes in workshops through multimodal data acquisition, AI-based artifact extraction, and interactive visual analysis. It is designed to support the identification of design outcomes and the examination of how collaboration formed results and led to decision-making, addressing the difficulty of integrating heterogeneous observations such as video, audio, notes, and gaze into a single analytic environment (Koch et al., 8 Aug 2025).

1. Conceptual Scope and Research Context

reCAPit was introduced for the analysis of collaborative design workshops, where findings are often reduced to a consolidated textual form based on notes from interviews or observations. The system addresses the challenge that different observation sources generate large amounts and heterogeneity of collected data. Its stated contribution is a practical, modular, and adaptable framework spanning workshop setup, multimodal data acquisition, AI-based artifact extraction, and visual analysis, with the goal of supporting both analysis and communication of important workshop findings (Koch et al., 8 Aug 2025).

The framework is explicitly post-hoc rather than a live facilitation system. Its emphasis is on data-rich acquisition of multimodal observations, combined AI-based extraction and interactive visual analysis, and transparent dissemination of results. This positioning matters methodologically: reCAPit is not limited to transcript inspection or note review, but instead aligns multiple observational channels on a common temporal basis so that activity, attention, discussion, and annotations can be interpreted together.

A common misconception is that such systems are necessarily tied to a single modality or a fixed workshop genre. reCAPit is instead presented as modular and adaptable, and the reported workshops range across different themes, including social science research on urban planning and a design study on band-practice visualization. This suggests a framework intended for methodological transfer across workshop settings rather than a tool specialized to one domain.

2. Instrumentation and Multimodal Data Acquisition

The reCAPit pipeline begins with a workshop setup centered on a shared working area consisting of two paper-covered tables instrumented with printed fiducial markers (Koch et al., 8 Aug 2025). The acquisition design combines top-down and side-view sensing, audio capture, wearable gaze tracking, and versioned note-taking.

Modality or setup element	Role in the pipeline
Top-down camera	Unobstructed view of on-table activities
Side cameras	Gestures and body language
Ceiling-mounted microphone	Group audio for transcription
Eye-tracking glasses	Point-of-regard samples up to 100 Hz
Versioned digital notes	Time-stamped textual observations

The top-down camera records the shared surface, while one or more side cameras record gestures and body language. A ceiling-mounted microphone captures group audio, synchronized to video via clapperboard triggers. Participants wear lightweight eye-tracking glasses with high-frequency sampling up to 100 Hz. A second moderator records time-stamped digital notes in a versioned document, exemplified by an instrumented Word file (Koch et al., 8 Aug 2025).

These channels are formalized as multimodal data streams. Video is acquired from two perspectives; audio is recorded as a single-channel group recording for transcription; gaze consists of point-of-regard samples in screen coordinates mapped into top-down camera space by homography; and notes are represented as text edits, specifically insertions and deletions, in a versioned electronic document. The system’s analytical power depends on the synchronization of these streams, because later visualizations and summaries assume temporal alignment across modalities.

This acquisition design reflects an important methodological trade-off stated in the source: richer modalities increase analytic coverage but also introduce calibration overhead, privacy concerns, and potential participant discomfort. reCAPit therefore frames modality choice as contingent on research questions, particularly whether the emphasis is on behavioral analysis or outcome analysis.

3. AI-Based Artifact Extraction and Temporal Segmentation

The artifact extraction stage converts raw multimodal streams into structured analytical objects. The table is partitioned into $M$ static Areas of Interest (AOIs), each keyed to a semantic zone such as one participant’s sketch area (Koch et al., 8 Aug 2025). This AOI structure is the reference frame for activity and attention analysis.

Hand activity is derived through background subtraction combined with MediaPipe hand tracking to compute pixel-level motion per AOI. Eye-tracking fixations are mapped onto AOIs to yield per-frame attention assignments. Automatic speech recognition is performed with Whisper, producing a timestamped transcript, and manual diarization assigns speaker IDs and roles. Notes contribute a parallel event stream defined by document edits.

All modalities are aligned on a shared time axis $t \in [0,T]$, using the initial clapperboard signal. For each frame $t$, reCAPit computes an $M$-dimensional vector $x_t$. In activity mode,

$$x_{m,t}=\frac{\text{activePixels}_{m,t}}{\text{areaPixels}_m},$$

and in attention mode,

$$x_{m,t}=\frac{\#\text{of fixations in AOI }m \text{ at }t}{\text{\#participants}}.$$

Temporal segmentation proceeds in two steps (Koch et al., 8 Aug 2025). First, multivariate change-point detection is applied to $\{x_t\}$ using the PELT algorithm with penalty $\beta=10$: $$\underset{C}{\mathrm{argmin}}\;\sum_{i=0}^{|C|} \mathrm{Cost}\bigl(x_{c_i:c_{i+1}}\bigr) + \beta\,|C|.$$ Second, each initial interval is refined by text-embedding similarity. Transcript utterances are chunked whenever two consecutive utterance embeddings $t \in [0,T]$0, obtained from a multilingual SBERT model, satisfy

$t \in [0,T]$1

This two-step design is significant because it combines behavioral signals from activity and attention with semantic shifts in discourse. A plausible implication is that reCAPit treats topic boundaries not as purely linguistic phenomena, but as joint changes in embodied interaction and spoken content.

4. Visual Analytics Architecture

reCAPit organizes analysis through a hierarchy of visual components: a multimodal streamgraph for global overview, a timeline view for mid-level temporal detail, topic cards for semantic summary and user annotations, and drill-down panels for raw data inspection (Koch et al., 8 Aug 2025). This layered structure is central to the system’s analytic model.

The multimodal streamgraph is described as a ThemeRiver-style streamgraph that visualizes either hand activity or visual attention across AOIs. At each time $t \in [0,T]$2, the total height is $t \in [0,T]$3. To minimize vertical “wiggle,” reCAPit uses the zero-sum baseline: $t \in [0,T]$4 Each AOI stream is rendered between $t \in [0,T]$5 and $t \in [0,T]$6. Hovering highlights the AOI across all visualizations, and clicking a time region selects the corresponding segment in the timeline and loads associated topic cards.

The timeline view complements the streamgraph with a per-speaker Gantt chart of utterances or a scarf-plot of gaze, overlaid with topic-segment bands and note-change icons. Heatmaps provide spatial aggregation of activity or gaze over arbitrary intervals. Together, these views connect temporal, spatial, and conversational evidence.

The visual encoding choices are explicit. AOIs receive distinct hues reused consistently across streamgraphs, heatmaps, and donut charts. Speaker roles are assigned perceptually separated hues to reduce visual clutter when $t \in [0,T]$7. Note edits are encoded as insertions with a green plus icon, deletions with a red minus, and modifications with a yellow bakelite icon. Heatmaps are computed with Gaussian kernels with $t \in [0,T]$8 px. Topic cards use a rectangular “index card” layout with a title banner, left-aligned quote list, right-aligned donut charts, and a bottom row of screenshots. These choices reveal a design preference for cross-view consistency and for preserving the visual identity of analytic entities such as AOIs and roles.

5. Topic Cards, Summarization, and Traceability to Raw Data

Topic cards are the main semantic artifact generated by reCAPit. After automatic segmentation, the system instantiates a topic card for each coherent discussion interval (Koch et al., 8 Aug 2025). Each card is time-aligned and editable, functioning as a structured summary rather than a detached annotation.

A topic card contains a title, selected quotes, screenshots, notes, and summary statistics. The title is auto-generated by GPT-4 from the transcript subset of the segment, then editable by the analyst. Quotes are imported by clicking utterance blocks in the timeline; each quote retains its original timestamp span $t \in [0,T]$9. Screenshots are created by selecting a frame time $t$0 and a crop rectangle in any video view. Notes can be freeform or imported from moderator notes with exact insert and delete times.

The summary statistics are formally defined. For a segment $t$1 spanning frames $t$2, speaker $t$3’s proportion of talk time is

$t$4

and AOI $t$5’s average activity or attention is

$t$6

These statistics are visualized as small donut charts inside each card.

A defining property of reCAPit is drill-down transparency. Clicking any timestamp in the streamgraph, timeline, or card opens a synchronized video player at the exact frame $t$7, with audio waveform and transcript. Selected quotes are hyperlinks into the full transcript. Clicking a note icon reveals the live editing view of the note document at that version. Heatmap overlays can be toggled in the video player for the current segment. This linking of high-level artifacts back to raw evidence is central to the system’s epistemic stance: summaries are not meant to replace source material, but to remain inspectable against it.

A common misconception is that LLM-generated summaries necessarily reduce analytic transparency. reCAPit explicitly counters this by making titles editable and by binding each high-level artifact to timestamps, transcript spans, screenshots, notes, and synchronized playback.

6. Case Studies, Methodological Implications, and Practical Constraints

The reported study comprised six workshops across different themes, with two detailed case studies: a band-practice visualization workshop and an urban planning workshop (Koch et al., 8 Aug 2025). These cases illustrate what kinds of findings the system can surface.

In the band-practice visualization case, the research question concerned how musicians segment sessions and use annotation. Analysts filtered on the keyword “segmentation,” identifying five segments labeled A–E in which participants debated calendar-style session markers versus real-time annotation. Streamgraphs showed that Randy’s area dominated activity during the “Live Annotation!” discussion in segments B–D. Topic cards preserved quotes such as “I don’t even look at the first ten minutes,” linked to sketches of calendar metaphors. The analysts concluded that manual live annotation both informs and replaces rigid segmentation, shaping the requirement “Segmentation” in the final design specification.

In the urban planning workshop, the analytic focus was how “Co-Production” emerged as a priority in a diverse stakeholder group. Analysts skimmed topic cards to locate the relevant segments. One card included Lisa’s explanation, “...would be really useful...treat that as suggestions,” which motivated drill-down into video and heatmap views to resolve indexicality, specifically that “this” referred to citizen feedback posters. Another card showed overlapping attention spikes across multiple AOIs, signaling a negotiation phase that transcript-only analysis had masked. An expert specifically praised the transparency of the transcript-to-video-to-notes linkage for enabling the capture of caveats in situ.

These case studies support the claim that reCAPit can reveal interactional structures that are difficult to reconstruct from transcript text alone. This suggests a broader methodological implication: temporally aligned multimodal summaries can function as an intermediate representation between raw workshop recordings and final qualitative interpretation.

The framework also foregrounds operational constraints. Multi-camera and eye-tracking data are highly sensitive and require informed consent and robust anonymization, with Stable Diffusion cited as a means to blur faces. Physiological sensors such as eye trackers and wearables add richness but also calibration time and possible participant discomfort. Clean audio transcription may require moderators to discourage simultaneous talking or to equip each participant with a lavalier microphone, which introduces new synchronization challenges. Static AOIs simplify tracking but may fail for mobile or wall-based interactions. Eye-tracker drift and hand-detection background models require repeated quality control, especially under changing light or new crafting materials. Versioned note-taking is necessary if text edits are to be precisely time-aligned.

Taken together, these constraints define the limits of the current framework as clearly as its capabilities. reCAPit is strongest when workshop activities are sufficiently spatially structured to support static AOIs, when multimodal synchronization is carefully engineered, and when transparency to raw evidence is preserved throughout the analytic workflow.