PerTouch Framework: Gesture & Image Retouching

• PerTouch Framework is a dual-purpose system integrating real-time ensemble gesture tracking for musical interaction and diffusion-based semantic image retouching for personalized edits.
• The musical system employs synchronized iPad-to-server communication with Random Forest classification to achieve low-latency gesture recognition and adaptive musical feedback.
• The image retouching pipeline harnesses a diffusion model coupled with a VLM agent to map natural language instructions to fine-grained semantic parameter adjustments validated by quantitative metrics.

The PerTouch Framework defines two distinct technological paradigms sharing the same name, each occupying a separate domain: the original PerTouch system for ensemble gesture recognition and real-time musical interaction (Martin et al., 2020), and a later diffusion-based personalized image retouching pipeline featuring a vision-LLM (VLM)–driven agent (Chang et al., 17 Nov 2025). Both frameworks are technically advanced, architected for low-latency human–computer collaboration, and evaluated with rigorous empirical protocols.

1. System Overviews and Architectural Fundamentals

A. Musical Ensemble Gesture Tracking (Martin et al., 2020)

The original PerTouch Framework, also referenced as the “Metatone Classifier,” consists of a central server and a scalable number $K$ of iPad clients running one of three custom apps (BirdsNest, Snow Music, PhaseRings). Communication is handled by OSC over Wi-Fi. The pipeline operates in real time, with iPad clients transmitting touch events to the server, which classifies gestures, computes statistical sequence and transition metrics, and streams per-user feedback to adapt the musical interface within each app.

Pipeline Flow:

1. Client-to-Server: Each touch event $(x, y, t, \text{velocity}, \text{type})$ is transmitted.
2. Server (Metatone Classifier):
   • Logs touch events per performer.
   • Maintains a 5 s sliding window for feature extraction.
   • Classifies gestures via Random Forest.
   • Updates user-specific gesture-state sequences.
   • Computes state transition matrices.
   • Evaluates ensemble-level “flux” for detecting performance novelty.
   • Broadcasts gesture labels and “new idea” triggers back to clients at 1 Hz.
3. Client Apps: Adapt interface/sound logic responsively, leveraging this feedback.

B. Semantic Image Retouching with Diffusion Models (Chang et al., 17 Nov 2025)

The newer PerTouch system is a unified, diffusion-based retouching framework that integrates a VLM-driven agent. This architecture parses detailed or weak natural language user instructions, generates or updates semantic parameter maps mapping attributes (colorfulness, contrast, temperature, brightness) to specified image regions, and employs a diffusion model (built on Stable Diffusion + ControlNet) to perform fine-grained edits.

Core Pipeline:

1. Input: Image $X$ and user instruction $I$.
2. Agent: Parses $I$, segments the image (using SAM and region detection), and constructs a multi-channel semantic parameter map $C \in \mathbb{R}^{K\times H\times W}$.
3. Retouching Engine: Injects $C$ into all ControlNet layers during diffusion inversion, generating the edited image $\hat{X}$.
4. Feedback Loop: If the result misaligns with instruction semantics (per a VLM-based classifier), the agent iterates to refine $C$.
5. Memory System: Stores scene–parameter pairs, enabling preference recall on weak instructions.

2. Technical Methods and Mathematical Foundations

A. Ensemble Gesture Classification and Sequence Modeling

Feature Extraction: For each 5 s window, the system aggregates features: touch-start rate, movement frequency, mean and standard deviation of location, mean velocity, and additional touch kinetics.

Random Forest Classification:

• Trained with $T\approx100$ decision trees for 9 gesture classes $G$.
• Majority voting yields the classification:

$h(f) = \mathrm{majority\ vote}\{h_t(f)\}_{t=1}^T$

• Assignment is at 1 Hz; high accuracy validated by extensive cross-validation.

State-Transition Matrix:

• Each performer's gesture sequence $X_1, ..., X_N$ is modeled as a Markov chain.
• Transition probabilities:

$p_{ij} = \frac{N_{ij}}{\sum_{j=1}^m N_{ij}}$

where $N_{ij}$ is the count of transitions $i \to j$.

• Ensemble matrix $\bar{P}$ is the mean across all $K$ performers.

Flux Metric:

• Quantifies behavioral novelty:

$$\mathrm{flux}(P) = \frac{\|P\|_1 - \|\mathrm{diag}(P)\|_1}{\|P\|_1}$$

• “New idea” events are triggered when recent flux increases exceed a threshold ($\Delta\mathrm{flux} > \theta$).

B. Semantic Control in Diffusion-Based Image Retouching

Latent Diffusion Model: Operates in $Z$-space; forward process $q(Z_t|Z_{t-1})$, reverse process $p_\theta(Z_{t-1}|Z_t, C)$. At each denoising step, semantic parameter map $C$ directs the transformation; training loss:

$$L_{\text{denoise}} = \mathbb{E}_{t, Z_0, \epsilon}[\,\|\epsilon - \epsilon_\theta(Z_t, t, C)\|_2^2\,]$$

Semantic Parameter Maps:

• Soft/hard regional masks $M_i$ from panoptic segmentation.
• Each region $i$ has $K=4$ scalar attribute scores $s_i \in [-1, 1]^K$; these are aggregated to form maps $C^k(x, y) = \sum_{i} M_i(x, y) s_i^k$.

Training Augmentations:

• Semantic Replacement (probability $p_{\text{rep}}\approx 0.2$): Replaces region-wise attribute vectors with sampled ones from the batch.
• Parameter Perturbation: Adds uniform noise $(\epsilon \approx 0.02)$ and Gaussian blurring to enforce robustness to precise boundaries.

3. Intelligent Agents and Real-Time Feedback

A. Client App Feedback in Musical Performance

• BirdsNest: Disruptively toggles looping/autoplay on repeated gestures, nudging performers toward variety.
• Snow Music: Adds instrumental layers aligned with gesture style; on “new idea”, shuffles instrument mapping.
• PhaseRings: Expands harmonic options on “new idea” signals, rewarding gestural exploration.

Pseudocode (client side):

onReceive(label, newIdea): currentLabel ← label if runLengthOf(label) > τ_repeat: BirdsNest.toggleLooping() SnowMusic.enableSupport(label) if newIdea == TRUE: BirdsNest.advanceScene() SnowMusic.shuffleSamples() PhaseRings.expandRings()

B. VLM-Driven Image Retouching Agent

• Weak instructions (“Optimize this image”): Computes scene embedding, queries scene-aware memory bank using cosine similarity, and retrieves mean control map.
• Strong instructions (“Make the eagle a bit brighter”): Uses object detection and segmentation to localize regions, text embedding to infer desired adjustment, and progresses via an MLP to update region controls.

Feedback-driven rethinking (Algorithm 2): Iterates inference (max $T_{\text{rethink}}=3$), each time checking semantic alignment via a VLM-based classifier before terminating.

4. Experimental Methodologies and Empirical Results

A. Ensemble Gesture Tracking

• Classifier Validation: 10-fold cross-validation (repeated 10 times), one-way ANOVA, and Bonferroni post-hoc tests.
• Production Setting: Reported mean accuracy of $0.973$ (std. dev $0.022$) over 532 test vectors, using 1 s rolling windows.
• Real-Time Profiling: Server cycle time $\approx 0.158$s with $K=4$ (scales linearly with $K$ with per-iPad increment $0.038$s), supporting up to $\sim25$ performers at 1 Hz update rate.
• Qualitative Feedback: Musicians responded positively to agent-driven variation, leveraging interface adaptivity in both individual and ensemble contexts.

B. Image Retouching Evaluation

• Dataset: MIT-Adobe FiveK, five expert retouching styles.
• Quantitative Metrics: PSNR ($25.14$ for PerTouch vs. $24.51$ for DiffRetouch) and LPIPS ($0.0798$ for PerTouch).
• Ablation: Semantic replacement removal decreases PSNR by $0.38$ dB (and increases LPIPS by $0.010$); similar, if slightly less, degradation for perturbation removal.
• User Study: PerTouch preferred in $48\%$ of cases by human raters.

Method	PSNR↑	LPIPS↓
DiffRetouch	24.51	0.0812
PerTouch (ours)	25.14	0.0798

5. Limitations and Future Research Directions

A. Musical Performance Framework

• The Markov (order-1) sequence assumption does not capture longer-term gestural dependencies; extension to higher-order or variable-length Markov models is proposed.
• The update interval is statically set to 1s; longer cycles could suit very large ensembles.
• Only a single metric (flux) is used for behavioral change; incorporating entropy, spectral, or community-structural analyses may yield richer ensemble state representations.
• Distributed, low-latency robust operation over Internet (beyond single-LAN Wi-Fi) remains an open engineering challenge.
• The fixed gesture taxonomy constrains expressivity; adaptive/interactive classifiers could allow gestural vocabularies to emerge organically.

B. Diffusion Retouching System

• The use of semantic replacement and parameter perturbation highlights a reliance on expert-tuned supervision, and the potential for further exploration into unsupervised attribute discovery.
• Memory module currently models user preference through nearest-neighbor averaging in embedding space; alternative models for long-term user preference dynamics (e.g., meta-learning approaches) remain an open direction.
• Feedback-driven rethinking iteration is capped ($T_{\text{rethink}}=3$); a plausible extension involves adaptive stopping criteria or more sophisticated user satisfaction modeling.

6. Relationships and Disambiguation

The two distinct PerTouch frameworks are unrelated apart from their shared focus on interactive, user-adaptive computational pipelines. The original music-focused system (Martin et al., 2020) is the canonical reference for multi-user gesture recognition and adaptive UI in digital musical ensembles, while the later image retouching pipeline (Chang et al., 17 Nov 2025) establishes a technical benchmark for unified semantic editing using VLM-augmented diffusion models.

Both highlight trends toward:

• Real-time, agent-driven adaptation in response to ambiguous, high-level human inputs.
• Explicit, mathematically formalized pipelines for mapping raw input (touch or image) and user intent (gestural or linguistic) to structurally meaningful output.
• Designers leveraging region-level semantic information (gesture-state sequences; panoptic masks and attribute maps) as a locus for adaptive control and interpretability.