Unified Sketch: Shared Representations

• Unified Sketch is a design principle using sketch representations as structured intermediates that bridge appearance, semantics, and spatial organization.
• It employs diverse methodologies such as latent sketch tokens in multimodal LLMs, shared scene encodings, and fused backbones to improve retrieval, synthesis, and biometric tasks.
• Unified Sketch also informs unified evaluation benchmarks and continual learning frameworks, paving the way for richer generative controls and multi-task integration.

Unified Sketch denotes a recurring design principle in which sketch-like representations are treated as a common interface across modalities, tasks, or reasoning states rather than as a standalone drawing format. In recent work, this principle appears in several technically distinct forms: latent sketch tokens interleaved with text inside multimodal LLMs, shared scene encodings that support both sketch-based retrieval and scene synthesis, multi-task evaluation protocols for sketch synthesis, continual sketch biometric identification across tasks, universal sketch features derived from fused foundation models, and earlier middle-level video representations that combine explicit primitives with implicit texture models (Tong et al., 18 Dec 2025, Ribeiro et al., 2021, Lin et al., 2024, Liu et al., 17 May 2026, Koley et al., 18 Mar 2025, Han et al., 2015). Across these formulations, the unifying claim is that sketches can function as structured intermediates that preserve geometry, support abstraction, and remain compatible with generation, matching, or reasoning.

1. Conceptual scope and historical antecedents

The term does not refer to a single canonical architecture. Instead, it names a family of approaches in which sketch serves as a shared representational substrate. Some works unify internal reasoning modes, some unify downstream tasks, some unify evaluation, and some unify heterogeneous biometric domains. The common thread is that sketch is used neither as a mere raster output nor as a narrow retrieval query, but as a structured latent or explicit representation that mediates between appearance, semantics, and spatial organization.

A useful historical antecedent is Video Primal Sketch, which extends Marr’s primal sketch to video by combining two regimes within one probabilistic representation: an explicit sparse coding model for moving corners, lines, and feature points, and implicit ST-FRAME or MA-FRAME models for textured motion such as water and fire (Han et al., 2015). Its video lattice is partitioned into explicit and implicit regions, with model selection driven by sketchability and trackability. Explicit regions are represented by learned primitives with parameters such as position, orientation, scale, velocity, and coefficient, while implicit regions are represented by histograms of spatio-temporal filter responses and velocity distributions. The resulting VPS representation is both generative and parsimonious, and is explicitly described as compatible with high-level action representations (Han et al., 2015).

Work	What is unified	Principal mechanism
Video Primal Sketch (Han et al., 2015)	explicit structure and implicit textured motion	sparse primitives + ST-FRAME / MA-FRAME
Scene Designer (Ribeiro et al., 2021)	scene search and scene synthesis from sketch	OLR → GNN → Transformer
SketchRef (Lin et al., 2024)	benchmark and metrics across sketch synthesis tasks	mOKS, CLIP similarity, SR, mRS/mRC
SketchFusion (Koley et al., 18 Mar 2025)	universal sketch features across tasks	SD + CLIP fusion
SkiLa (Tong et al., 18 Dec 2025)	textual and visual thoughts in MLLMs	latent sketch tokens
UFSB (Liu et al., 17 May 2026)	face and pedestrian sketch biometric tasks	SIM + MPM + synthetic sketches

This breadth is important because it prevents an overly narrow reading of “unified.” In one line of work, unification is about a single latent space; in another, about a single backbone or benchmark; in another, about a hybrid generative model that automatically switches between explicit and implicit regimes. This suggests that Unified Sketch is best understood as a representational strategy rather than a single method name.

2. Unified latent reasoning in multimodal LLMs

In multimodal LLMs, Unified Sketch is instantiated most directly by Sketch-in-Latents (SkiLa), which argues that because projected visual tokens and text tokens already inhabit the same embedding space inside an MLLM, the model should be able to generate continuous visual-like vectors as internal “visual thoughts” during autoregressive reasoning (Tong et al., 18 Dec 2025). SkiLa is built on Qwen2.5-VL 7B, freezes the base vision encoder and multimodal projector, and trains the LLM backbone together with a sketch projector. Its central construct is the latent sketch token: a continuous hidden-state vector generated by the LLM itself, supervised not by vocabulary cross-entropy but by an MSE target derived from intermediate sketch images.

The unified reasoning trace is defined as

$$\mathcal{S} = \{s_1, \dots, s_N\}, \qquad s_i \in \{t_i, h_i\},$$

where each position is either a discrete text token or a continuous latent sketch token. During textual thinking mode, the model performs standard next-token prediction; during visual sketching mode, triggered by <|sketch_start|>, it self-feeds hidden states until <|sketch_end|> is generated. Semantic grounding is imposed through latent visual semantics reconstruction:

$$\mathcal{L}_{SkiLa} = \mathcal{L}_{NTP} + \lambda_{Sketch} \cdot \mathcal{L}_{Sketch},$$

with $\lambda_{Sketch}=0.5$ and 27 sketch tokens in the reported setting (Tong et al., 18 Dec 2025).

A key property of SkiLa is that it does not decode latent sketches into pixels at inference. The sketch encoder and sketch projector exist only to produce semantic targets during training. This distinguishes SkiLa from tool-based approaches that dispatch crop or draw operations, and from pixel-generative approaches that synthesize full images during reasoning. The paper characterizes SkiLa instead as a “latent internal sketchboard,” in which image inputs, textual outputs, and generated sketch embeddings all occupy positions in the same transformer sequence (Tong et al., 18 Dec 2025).

Empirically, this internal unification improves vision-centric reasoning. Relative to Qwen2.5-VL 7B, SkiLa reports gains of +9.3 on MMVP, +4.5 on MMStar, +5.8 on V*Bench, +3.5 on HR$_{4K}$, +1.5 on HR$_{8K}$, and +9.8 overall on MME-RealWorld-Lite, alongside improvements on TextVQA, MME, and POPE (Tong et al., 18 Dec 2025). The ablation over the number of latent sketch tokens peaks around 27, with too few tokens underrepresenting visual semantics and too many crowding out textual reasoning. A related misconception is therefore that more internal visual tokens are necessarily better; the reported behavior is explicitly non-monotonic (Tong et al., 18 Dec 2025).

3. Shared scene representations and universal sketch features

A second major sense of Unified Sketch appears in systems that use one representational backbone to support multiple tasks from sketch input. Scene Designer is an early and explicit example: a single model learns both a cross-modal search embedding for sketch–image scene retrieval and object-level embeddings that drive box and mask generation for scene synthesis (Ribeiro et al., 2021). Its pipeline consists of modality-specific MobileNet object encoders followed by shared fully connected layers for Object-Level Representation, a 6-layer GNN for Constrained-Correlated Representation over a scene graph, and a 3-layer 16-head Transformer that yields per-object Freely-Correlated Representations and a scene-level Scene Representation. The same OLR, GNN, and Transformer are shared across tasks and modalities, while the scene-level vector supports retrieval and the object-level vectors support layout synthesis through box and mask generators (Ribeiro et al., 2021).

This joint design is not merely architectural reuse. The model is trained with a scene-level contrastive loss, bounding-box and mask-generation losses, and classification losses, and the ablations show that removing either the contrastive loss or the generation losses degrades both search and synthesis. On QuickDrawCOCO-92c, the ablation without contrastive loss reduces Recall@1 from 62.15 to 45.48 and worsens FID from 76.64 to 129.88, while the ablation without generation losses reduces Recall@1 to 55.84 and worsens FID to 129.98 (Ribeiro et al., 2021). On SketchyCOCO, Scene Designer reports Recall@1 of 75.53 for compositional sketch-based retrieval, with 66.09 on Extended SketchyCOCO and 62.15 on QuickDrawCOCO-92c; for mixed-domain queries combining sketch and image objects, Recall@1 rises to 89.69, 87.12, and 93.22 respectively (Ribeiro et al., 2021).

SketchFusion advances a different but related notion of unification: a universal sketch feature representation built by fusing Stable Diffusion v2.1 with CLIP rather than retraining either foundation model (Koley et al., 18 Mar 2025). The paper’s analysis identifies two limitations: SD struggles to extract meaningful features from abstract sketches, and its denoising U-Net exhibits a pronounced high-frequency bias that suppresses low-frequency structure needed for sketch understanding. CLIP is semantically strong on sketches but spatially coarse. SketchFusion therefore keeps both SD and CLIP frozen, injects CLIP visual features into SD’s upsampling blocks by layer-specific learnable 1D convolutions, and adaptively aggregates multi-level SD features into a shared $60 \times 60$ feature map (Koley et al., 18 Mar 2025).

The resulting backbone is reused across zero-shot sketch-based retrieval, fine-grained retrieval, sketch recognition, sketch-based segmentation, and sketch–photo correspondence. In the abstract, SketchFusion reports state-of-the-art gains of +3.35% in sketch retrieval, +1.06% in recognition, +29.42% in segmentation, and +21.22% in correspondence learning (Koley et al., 18 Mar 2025). An important clarification follows from the paper itself: the universality is architectural and empirical, not a single multitask training run. The same fused backbone is reused across tasks, but the task objectives are optimized separately (Koley et al., 18 Mar 2025).

4. Unified evaluation and metric design

Unified Sketch also names an evaluative program. SketchRef is proposed as the first comprehensive multi-task evaluation benchmark for sketch synthesis, motivated by the absence of a unified dataset, the overreliance on category-level recognizability, and the lack of control for the recognizability–simplicity trade-off (Lin et al., 2024). It organizes evaluation around two primary tasks—category prediction and structural consistency estimation—which are instantiated as five sub-tasks across four domains: human body, human face, animals, and things.

The benchmark comprises 4,537 reference photos and 36,296 synthesized sketches, with each reference photo rendered by 8 sketch synthesis methods. Human Body contains 1,137 reference photos and 9,096 sketches; Human Face 950 photos and 7,600 sketches; Animals 950 photos and 7,600 sketches; Things 1,500 photos and 12,000 sketches. All reference photos are segmented by U$^2$Net and resized to 224×224 (Lin et al., 2024).

SketchRef’s distinctive contribution is its simplicity-aware metric suite. Structural recognizability is measured by mOKS, category-level recognizability by CLIP similarity, and simplicity by the Simplicity Ratio

$$\text{SR}(x^{\text{skt}}) = \frac{C(x^{\text{ref}})}{C(x^{\text{skt}})},$$

where $C(\cdot)$ is chosen as Compression Ratio. These are combined into mRS@$\alpha$ and mRC@$$\mathcal{L}_{SkiLa} = \mathcal{L}_{NTP} + \lambda_{Sketch} \cdot \mathcal{L}_{Sketch},$$0, which restrict evaluation to sketches with $$\mathcal{L}_{SkiLa} = \mathcal{L}_{NTP} + \lambda_{Sketch} \cdot \mathcal{L}_{Sketch},$$1. This directly addresses the confound that more lines and more detail usually increase recognizability (Lin et al., 2024).

The benchmark’s 7,920 human responses validate both the structural and simplicity metrics. For structure-level recognizability, pose-based mOKS correlates strongly with human judgment, with ViTPose achieving Spearman $$\mathcal{L}_{SkiLa} = \mathcal{L}_{NTP} + \lambda_{Sketch} \cdot \mathcal{L}_{Sketch},$$2 and Kendall $$\mathcal{L}_{SkiLa} = \mathcal{L}_{NTP} + \lambda_{Sketch} \cdot \mathcal{L}_{Sketch},$$3, whereas Inception-based similarities show weak alignment. For simplicity, Compression Ratio-based SR achieves Spearman $$\mathcal{L}_{SkiLa} = \mathcal{L}_{NTP} + \lambda_{Sketch} \cdot \mathcal{L}_{Sketch},$$4 and Kendall $$\mathcal{L}_{SkiLa} = \mathcal{L}_{NTP} + \lambda_{Sketch} \cdot \mathcal{L}_{Sketch},$$5, outperforming alternatives such as Harris corners, FAST, entropy-based measures, and ICNet features (Lin et al., 2024). The resulting rankings also expose domain-specific trade-offs: Anime is best on mRS when detail is allowed, while CLIPasso consistently leads category-level recognizability on Animals and Things (Lin et al., 2024). A common misconception addressed by this framework is that sketch quality can be reduced to classifier accuracy alone; SketchRef formalizes a stricter notion in which category, structure, and abstraction level must be considered jointly.

5. Continual cross-task sketch biometric identification

In biometric identification, Unified Sketch is formulated as a lifelong, cross-task, cross-modal problem. The paper “Bridging Data Trials and Task Barriers” defines sketch biometric identification as training one model that can handle multiple sketch-based biometric re-identification tasks across different domains and modalities in a task-sequential manner, with only partial access to past data (Liu et al., 17 May 2026). The reported tasks are sketch-face identification and sketch-pedestrian re-identification.

The proposed Unified Framework for Sketch Biometric Identification, UFSB, combines three components: a Sketch Biometric Identification Foundation Module, a Multi-Task Preservation Module, and large-scale synthetic sketch generation. The shared backbone is ClipReID on CLIP ViT-B/16, producing 768-dimensional global features. SIM optimizes identity classification, triplet loss, image-to-text cross-entropy, and a JMMD term for cross-modal alignment:

$$\mathcal{L}_{SkiLa} = \mathcal{L}_{NTP} + \lambda_{Sketch} \cdot \mathcal{L}_{Sketch},$$6

MPM then preserves prior tasks through trusted sample replay: after finishing a task, the model builds a sketch bank and a photo bank containing at most one sketch and one photo per identity, selected by low-uncertainty conformal prediction scores (Liu et al., 17 May 2026).

The supporting benchmark, SketchUnified-BioID, contains 13,942 identities and more than 480k images. Its pedestrian subset combines MaSk1K with synthetic CUHK03-Sketch; its face subset combines CUFSF or IIIT-D Viewed Sketch with synthetic CelebA-Sketch. Synthetic sketches are used only for training, not testing, and identities are non-overlapping across subsets (Liu et al., 17 May 2026).

The reported results are strong in both single-task and sequential settings. On MaSk1K, UFSB$$\mathcal{L}_{SkiLa} = \mathcal{L}_{NTP} + \lambda_{Sketch} \cdot \mathcal{L}_{Sketch},$$7 reaches mAP 28.25 and Rank-1 34.51, compared with BDG at mAP 19.61 and Rank-1 18.10. In Scheme A, UFSB$$\mathcal{L}_{SkiLa} = \mathcal{L}_{NTP} + \lambda_{Sketch} \cdot \mathcal{L}_{Sketch},$$8 attains Avg mAP 61.89 and Rank-1 62.97, close to Joint$$\mathcal{L}_{SkiLa} = \mathcal{L}_{NTP} + \lambda_{Sketch} \cdot \mathcal{L}_{Sketch},$$9 at 62.70 and 64.07. In Scheme B, UFSB$\lambda_{Sketch}=0.5$0 surpasses the reported joint upper bound, reaching Avg mAP 56.73 and Rank-1 57.63 versus Joint$\lambda_{Sketch}=0.5$1 at 55.32 and 55.11 (Liu et al., 17 May 2026). This suggests that, in this setting, sequential learning with replay and cross-modal alignment can act as a useful curriculum rather than only a concession to data access constraints.

6. Common principles, misconceptions, and future directions

Across these papers, at least four distinct meanings of “unified” recur. First, unification can mean a single latent space in which textual and visual thoughts coexist, as in SkiLa (Tong et al., 18 Dec 2025). Second, it can mean a single representational backbone shared across search and generation, as in Scene Designer, or across retrieval, recognition, segmentation, and correspondence, as in SketchFusion (Ribeiro et al., 2021, Koley et al., 18 Mar 2025). Third, it can mean a common evaluative lens spanning domains and tasks, as in SketchRef (Lin et al., 2024). Fourth, it can mean one continually trained model that preserves capability across multiple sketch biometric tasks, as in UFSB (Liu et al., 17 May 2026).

Several misconceptions follow from conflating these senses. Unified Sketch does not necessarily imply pixel rendering: SkiLa never decodes latent sketch tokens back into images at inference (Tong et al., 18 Dec 2025). It does not necessarily imply one joint multitask optimization run: SketchFusion reuses one fused backbone across tasks but trains task objectives separately (Koley et al., 18 Mar 2025). It does not necessarily imply generation: SketchRef is explicitly a benchmark and metric suite (Lin et al., 2024). Nor does it imply that sketch alone suffices without structured context: Scene Designer depends on scene graphs and Transformer-based global correlation, while Video Primal Sketch depends on explicit–implicit model selection through sketchability and trackability (Ribeiro et al., 2021, Han et al., 2015).

The open directions named in these works are correspondingly diverse. SkiLa proposes more task-adaptive selection among textual, visual, and joint reasoning patterns (Tong et al., 18 Dec 2025). Scene Designer points toward interactive creative workflows and richer generative controls (Ribeiro et al., 2021). SketchRef suggests training or fine-tuning evaluators on sketches and extending simplicity beyond pixel-level compression ratio (Lin et al., 2024). UFSB identifies more sophisticated synthetic sketch generation, more advanced continual learning, additional biometric tasks, and fairness analysis as future directions (Liu et al., 17 May 2026). SketchFusion proposes stronger foundation backbones, true multitask training, lightweight distillation, and zero-shot dense tasks (Koley et al., 18 Mar 2025).

Taken together, these lines of work show that Unified Sketch is not a narrow subfield but a cross-cutting technical motif. Whether instantiated as a latent internal sketchboard, a shared cross-modal scene representation, a simplicity-aware benchmark, a continual biometric embedding, or a hybrid middle-level video model, the central premise remains stable: sketch can serve as a compact, structurally meaningful representation that links abstraction to computation more directly than either raw pixels or text alone.