PerTouch: Unified Framework for Semantic Retouching

• PerTouch is a unified framework for personalized, semantic image retouching that fuses a Stable Diffusion + ControlNet backbone with region-wise parameter maps.
• It leverages SAM-based segmentation to convert regional scores into explicit control signals for attributes like brightness, contrast, and color temperature.
• The system features a VLM-driven agent for natural language instruction parsing and long-term personalization via scene-aware memory, outperforming existing SOTA methods.

PerTouch most explicitly denotes a unified framework for personalized, semantic image retouching that combines a Stable Diffusion + ControlNet backbone with a Vision-LLM-driven agent, using region-wise parameter maps to couple semantic control with globally coherent photographic enhancement (Chang et al., 17 Nov 2025). In the surrounding literature, the same name also appears more loosely in descriptions of touch-centered sensing and interaction paradigms, including cross-sensor tactile representation learning, full-hand tactile egocentric perception, pre-touch pose refinement, around-device interaction, and wrist-worn ubiquitous touch interfaces. This suggests that “PerTouch” functions both as the title of a specific image-retouching system and as an informal label for research programs centered on semantics, touch, locality, and reusable control across heterogeneous embodiments.

1. Named framework: personalized and semantic image retouching

PerTouch, in its explicit named form, is a diffusion-based, agent-driven system for image retouching. It addresses two coupled problems: controllability and subjectivity. Controllability concerns specifying what to change and where, at semantic-region granularity, over attributes such as colorfulness, contrast, color temperature, and brightness. Subjectivity concerns adapting retouching to long-term user preferences rather than producing a single deterministic enhancement (Chang et al., 17 Nov 2025).

The system is positioned between conventional deep retouching models and text-guided image editing. Standard retouching pipelines often predict a single enhancement, offer only global control, or depend on rigid local masks. Text-guided editing methods provide broader semantic manipulation but may alter content and composition, making them ill-suited to faithful photographic retouching. PerTouch instead combines explicit parametric control with a latent diffusion prior so that local instructions remain coupled to global aesthetics.

Its architecture is organized around two layers. The first is a controllable diffusion retouching backbone. The second is a VLM-driven agent that interprets natural-language instructions, differentiates strong and weak instructions, applies object- or region-specific edits, iteratively refines results through feedback-driven rethinking, and updates a scene-aware memory of accepted parameter settings.

2. Diffusion backbone and explicit parameter-to-image mapping

The retouching backbone uses Stable Diffusion as a latent diffusion model and attaches ControlNet to inject spatial control through a multi-channel parameter map. During inference, the model receives the original image $X$ and a parameter tensor

$C = \{C^1, C^2, \dots, C^K\},$

where $K=4$ in the reported system and the channels correspond to colorfulness, contrast, color temperature, and brightness (Chang et al., 17 Nov 2025).

Each channel $C^k \in \mathbb{R}^{H \times W}$ stores the spatial distribution of one attribute. The values are normalized to the control range

$C^k_{ij} \in [-1,1].$

Semantic regions are obtained via SAM panoptic segmentation with NMS and filtering. For each region and each attribute, a scalar score is computed and broadcast to all pixels in that region, yielding a region-wise parameter map. In this way, the conditioning is explicit rather than implicit: changing parameter values in a semantic region changes the control signal presented to ControlNet.

The diffusion formulation follows standard latent DDPM training. Given latent $Z_0$,

$$Z_t = \sqrt{\alpha_t} Z_0 + \sqrt{1-\alpha_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,I),$$

and the denoising network is trained with

$$\mathcal{L}_{\text{diff}}=\mathbb{E}_{Z_0,t,\epsilon}\left[\left\|\epsilon-\varepsilon_\theta(Z_t,t,m)\right\|_2^2\right],$$

where $m$ includes the original-image conditioning and the parameter map. The role of ControlNet is summarized schematically as

$$F_C=\text{ControlNet}(C),\qquad h_\ell' = h_\ell + g_\ell(F_C),$$

so that parameter-map features are injected into the denoising U-Net at multiple layers. The training objective is therefore the standard diffusion noise-prediction loss, while the supervision signal is provided by expert-retouched target images and their associated parameter maps.

Because the source image is part of the conditioning, the goal is retouching rather than content synthesis. The system is trained so that

$C = \{C^1, C^2, \dots, C^K\},$0

with $C = \{C^1, C^2, \dots, C^K\},$1 an expert-retouched target. Global quality is not imposed by a separate explicit aesthetic loss in the main text; instead it is induced by the diffusion prior and the expert retouching data.

3. Semantic parameter maps, semantic replacement, and perturbation

The central representational device in PerTouch is the semantic parameter map. SAM is used to segment an image into semantic regions; for each region $C = \{C^1, C^2, \dots, C^K\},$2 and attribute $C = \{C^1, C^2, \dots, C^K\},$3, a regional scoring function produces a scalar $C = \{C^1, C^2, \dots, C^K\},$4, and all pixels $C = \{C^1, C^2, \dots, C^K\},$5 in $C = \{C^1, C^2, \dots, C^K\},$6 receive

$C = \{C^1, C^2, \dots, C^K\},$7

This converts region-level semantics into pixel-aligned control.

Two training mechanisms are added to force the model to learn localized semantic editing rather than global style transfer. The first is semantic replacement. A semantic region is selected with probability proportional to region area, then replaced in both the image and the parameter map with a region from another image whose attribute values are maximally different in parameter space. This creates a deliberately mismatched local region that the model must resolve using the control map rather than diffuse global statistics (Chang et al., 17 Nov 2025).

The second is parameter perturbation. Noise or transformations are applied to parameter maps, including channel shifts and boundary blurring. The stated purpose is to prevent hard dependence on segmentation edges and to make the model interpret $C = \{C^1, C^2, \dots, C^K\},$8 as a soft signal. The sequence is: build $C = \{C^1, C^2, \dots, C^K\},$9, optionally apply semantic replacement to get $K=4$0, perturb to obtain $K=4$1, and train the diffusion model conditioned on $K=4$2.

The ablations reported in the source text assign distinct roles to these components. Without semantic replacement, edits spill over globally and the system behaves more like a global enhancer even when only one region’s parameters are changed. Without perturbation, the model overfits to segmentation boundaries and may produce unnatural transitions. With both components, the result is region-level controllability with smoother global coherence.

4. VLM-driven agent, instruction parsing, and long-term personalization

On top of the diffusion model, PerTouch introduces a VLM-driven agent that translates language into region-level control. The agent parses instructions, distinguishes weak instructions such as “Optimize this image” from strong instructions such as “Increase brightness of the eagle significantly,” invokes segmentation, modifies the relevant channels of the parameter map, and optionally performs feedback-driven rethinking (Chang et al., 17 Nov 2025).

For weak instructions, the system initializes all attributes at midpoint values, queries a scene-aware memory, and adjusts the parameter map according to historical preferences for similar scenes. For strong instructions, the agent extracts the target region, target attribute, and intensity descriptor, uses VLM detection plus SAM to obtain a mask, and modifies only the corresponding pixels in the relevant channel. The existing guidance map can therefore serve as a base, with strong instructions acting as local edits on top of prior global or memory-informed settings.

The feedback loop is formalized as

$K=4$3

followed by iterative updates

$K=4$4

Here $K=4$5 is the instruction, $K=4$6 represents prior knowledge, and $K=4$7 denotes user-specific history. The intended effect is to refine coarse mappings from language such as “slightly” or “significantly” into better-aligned control values after observing intermediate outputs.

Personalization is implemented through scene-aware memory. For each confirmed edit, the system stores a pair $K=4$8, where $K=4$9 denotes scene semantics and $C^k \in \mathbb{R}^{H \times W}$0 the final accepted control parameters. For a new query scene with features $C^k \in \mathbb{R}^{H \times W}$1, the agent initializes from

$C^k \in \mathbb{R}^{H \times W}$2

thereby adapting weak instructions to long-term scene-dependent user preferences without retraining the diffusion model.

5. Training data, evaluation, and empirical results

PerTouch is trained primarily on MIT-Adobe FiveK, which contains 5,000 RAW images and five expert-retouched versions per image, labeled A/B/C/D/E (Chang et al., 17 Nov 2025). For each training sample, the original image is segmented with SAM, regional attribute scores are computed from the expert-retouched target, the scores are fused into a multi-channel parameter map, and semantic replacement plus perturbation are applied to build robust training pairs. Multi-style supervision is obtained because each image is paired with multiple expert outputs.

The reported baselines are PIENet, TSFlow, StarEnhancer, and DiffRetouch. Evaluation uses PSNR, LPIPS, and a user study. The quantitative comparison reported against DiffRetouch is as follows.

Expert	DiffRetouch	PerTouch
A	PSNR 24.51, LPIPS 0.0812	PSNR 25.14, LPIPS 0.0798
B	PSNR 26.15, LPIPS 0.0672	PSNR 27.47, LPIPS 0.0687
C	PSNR 25.91, LPIPS 0.0684	PSNR 26.75, LPIPS 0.0844
D	PSNR 24.51, LPIPS 0.0768	PSNR 25.97, LPIPS 0.0823
E	PSNR 24.74, LPIPS 0.0776	PSNR 25.66, LPIPS 0.0792

The reported interpretation is that PerTouch consistently has higher PSNR and competitive LPIPS while also providing region-level semantic control. The text also states that it is at least on par with or better than existing SOTA globally.

The user study uses 30 test images and 50 volunteers. Each participant views the original image and four retouched outputs—PerTouch, DiffRetouch, StarEnhancer, and TSFlow—and selects a preferred result. The reported summary is that most participants preferred PerTouch in nearly half of the cases, significantly more often than other methods.

Ablation studies identify the semantic replacement module and the perturbation mechanism as crucial. The first sharpens semantic localization; the second preserves global coherence and reduces boundary artifacts. The VLM-driven agent and scene-aware memory are evaluated mainly through qualitative examples and the user study, reflecting their role in interaction and personalization rather than strict reconstruction fidelity.

6. Broader research usage of the term across touch-centered systems

Outside image retouching, the name “PerTouch” appears in the supplied literature as a broader organizing label for touch-centered representation learning, perception, and interaction. This usage is descriptive rather than canonical, but it is recurrent enough to define a cross-domain pattern.

In tactile representation learning, “Contrastive Touch-to-Touch Pretraining” learns a shared embedding for GelSlim and Soft Bubble sensors by aligning paired tactile observations of the same physical grasp in a shared latent space, using separate ResNet-50 encoders, a 2-layer MLP, a 64-D contrastive space, and an InfoNCE objective. The accompanying discussion explicitly states that CTTP is “very close in spirit to the idea of ‘PerTouch’,” because it aims at a universal, sensor-agnostic tactile representation that supports downstream classification, pose estimation, and zero-shot cross-sensor transfer (Rodriguez et al., 2024).

In egocentric multimodal perception, OpenTouch is described as “essentially a PerTouch-style platform.” It aligns full-hand tactile maps, egocentric RGB video, and hand pose in a shared embedding space, provides 5.1 hours of synchronized video-touch-pose data and 2,900 curated clips, and shows that tactile signals can ground retrieval and grasp understanding in settings where vision is ambiguous or occluded (Song et al., 18 Dec 2025).

In robotic manipulation, “Improved Object Pose Estimation via Deep Pre-touch Sensing” presents a fingertip-mounted optical time-of-flight pre-touch sensor and a deep region-selection network that guides sparse, informative scans for ICP-based pose refinement. The description supplied for that work treats “PerTouch” as a learning-guided, pre-contact sensing strategy that improves end-effector-relative pose estimation while avoiding arm calibration errors (Lancaster et al., 2022).

In interactive systems, “Above Surface Interaction for Multiscale Navigation in Mobile Virtual Reality” is described as “very close to what you call PerTouch-style or around-device interaction,” because it uses the small volume above a touchscreen for rate-controlled navigation and the surface itself for selection and clutching (Menzner et al., 2020). “TouchFusion: Multimodal Wristband Sensing for Ubiquitous Touch Interactions” extends the same logic to wrist-worn sensing, combining sEMG, bioimpedance, inertial, and optical sensing to support touch state, gestures, and tracking on arbitrary nearby surfaces and on the body (Whitmire et al., 16 Feb 2026). A separate line of work on depth-based measurement of human-to-human touch provides high-resolution time-series attributes such as contact area, indentation depth, absolute velocity, three orthogonal velocity components, and contact duration, showing how contact mechanics can be quantified without interfering with direct skin contact (Xu et al., 2022). At the evaluation level, “Metrics for Multi-Touch Input Technologies” proposes a factorized framework with $C^k \in \mathbb{R}^{H \times W}$3, $C^k \in \mathbb{R}^{H \times W}$4, $C^k \in \mathbb{R}^{H \times W}$5, $C^k \in \mathbb{R}^{H \times W}$6, $C^k \in \mathbb{R}^{H \times W}$7, $C^k \in \mathbb{R}^{H \times W}$8, $C^k \in \mathbb{R}^{H \times W}$9, $C^k_{ij} \in [-1,1].$0, and compound metrics $C^k_{ij} \in [-1,1].$1 and $C^k_{ij} \in [-1,1].$2, furnishing a domain-independent language for comparing multi-touch systems (Arif, 2020).

Taken together, these adjacent usages imply a stable conceptual core. In one explicit instantiation, PerTouch is a VLM-mediated semantic retouching system built on diffusion and parameter maps. In another, more informal but technically coherent sense, it denotes research programs that seek explicit localized control, shared latent spaces across heterogeneous embodiments, and touch-grounded perception or interaction that remains reusable across tasks, sensors, scenes, or users.