SemanticGarment: Structured Clothing Modeling

• SemanticGarment is a modeling paradigm that defines garments as semantically structured objects, integrating part-based, CAD, and 3D representations.
• It employs semantic parsing, geometric priors, and executable design programs to drive text- and image-guided garment generation and precise editing.
• Multiple representational families coexist, including part-aware visual methods, programmatic CAD semantics, and joint 2D–3D human-aware approaches for realistic simulation.

SemanticGarment is a line of garment computing in which clothing is modeled as a semantically structured object rather than as an undifferentiated image, texture map, or mesh. In this sense, semantic structure may refer to garment parts and attributes in image generation, to executable sewing-pattern programs in CAD, or to human-aware 3D representations for generation, editing, animation, and verification. The term also names a specific 3D Gaussian framework for text- or image-driven garment generation and semantic editing, but its broader significance lies in a convergence of methods that combine semantic parsing, geometric priors, sewing-pattern abstractions, and physics-aware representations (Wang et al., 21 Sep 2025). An early precursor is single-view garment recovery, which already combined statistical, geometric, and physical priors with parameter estimation, semantic parsing, shape recovery, and physics-based cloth simulation to reconstruct detailed garments from one photograph (Yang et al., 2016).

1. Conceptual scope and development

Semantic garment modeling emerged from a practical limitation of earlier pipelines: realistic clothing behavior and editability require more than appearance synthesis. Image-based reconstruction sought to infer global shape, geometry, wrinkles, folds, and occluded regions from limited observations, but this required strong priors about body shape, garment class, and cloth behavior. In the single-view setting, garment recovery was formulated as a combination of semantic parsing, parameter estimation, shape recovery, and physics-based cloth simulation, indicating that semantic understanding and physical plausibility were already coupled in garment analysis (Yang et al., 2016).

Subsequent work made that coupling explicit. In text-to-garment synthesis, semantic control shifted from whole-prompt conditioning toward part-aware alignment. DiffCloth represents garments in the visual modality as segmented semantic parts and in the linguistic modality as Attribute-Phrases, then aligns them by bipartite matching and bundled cross-attention (Zhang et al., 2023). GarmentAligner advances the same direction by treating quantity, position, and interrelations of garment components as explicit training targets, rather than hoping they will emerge from generic text-to-image attention (Zhang et al., 2024). In parallel, CAD-oriented work redefined garments as structured programs. GarmentCode introduced a DSL in which bodices, sleeves, collars, skirts, darts, gathers, interfaces, and stitching rules are first-class objects (Korosteleva et al., 2023), while Design2GarmentCode and ChatGarment used multimodal LLMs to synthesize parametric sewing-pattern programs or JSON garment specifications from text, sketches, or photographs (Zhou et al., 2024, Bian et al., 2024).

A recurring divide in the literature is therefore not between “semantic” and “non-semantic” garments, but between different loci of semantics. Some systems place semantics in segmented parts, attention maps, and attribute phrases; others place them in executable programs, scene graphs, or joint 2D–3D particle representations. The literature does not converge on a single canonical representation, and this plurality is central to the topic.

2. Representational families

Several representational families organize current SemanticGarment research.

Representation family	Core objects	Representative papers
Part-aware visual semantics	Parts, masks, Attribute-Phrases, counts, spatial attention	(Zhang et al., 2023, Zhang et al., 2024, Shen et al., 2024, Yin et al., 5 Aug 2025)
Programmatic and CAD semantics	DSL components, JSON schemas, design parameters, stitches	(Korosteleva et al., 2023, Zhou et al., 2024, Bian et al., 2024, Teikari et al., 6 Jan 2026)
Joint 2D–3D and human-aware semantics	5D particles, 3D Gaussians, SMPL-X semantic regions, GSMs	(Nakayama et al., 25 May 2026, Wang et al., 21 Sep 2025, Zhuang et al., 2024)

At the image-semantic level, DiffCloth defines a visual structure

$\mathbf{V} = [V_{full}, V_1, V_2, \dots, V_m]$

and a linguistic structure

$\mathbf{W} = [W_{full}, W_1, W_2, \dots, W_m],$

where each $V_i$ is a garment part image and each $W_i$ is an Attribute-Phrase obtained by constituency parsing. The basic semantic unit is therefore a part-plus-attributes phrase such as a color, pattern, or length descriptor bound to a noun naming a garment part (Zhang et al., 2023). GarmentAligner adopts a related but more explicitly counted representation: components are detected, segmented, assigned positions by box centers, and associated with per-component text, masks, and quantities $q_i$ (Zhang et al., 2024).

Programmatic systems move semantics from image regions into executable garment logic. Design2GarmentCode defines a sewing pattern as

$$\mathcal{S} = \langle \mathcal{F}, \mathcal{D}, B \rangle,$$

where $\mathcal{F}$ is a set of symbolic programs, $\mathcal{D}$ is a semantically meaningful design configuration, and $B$ is body measurements (Zhou et al., 2024). GarmentCode provides the underlying DSL: garments are built hierarchically from Component, Panel, Edge, EdgeSequence, Interface, and StitchingRule, making sleeves, bodices, collars, skirts, darts, and seams explicit programmable objects rather than raw curves (Korosteleva et al., 2023). ChatGarment further adapts this paradigm to VLM output by generating a simplified JSON that mixes garment-type fields with normalized continuous parameters, which GarmentCodeRC converts into sewing patterns and draped 3D garments (Bian et al., 2024).

At the 2D–3D coupled end, Garment Particles model a garment as the graph of a mapping $\mathbf{r}: U \to \mathbb{R}^3$,

$\mathbf{W} = [W_{full}, W_1, W_2, \dots, W_m],$0

then discretize this as a 5D point cloud carrying both packed 2D sewing-pattern coordinates and 3D draped coordinates, plus a boundary flag (Nakayama et al., 25 May 2026). Textile IR generalizes semantic representation into a scene-graph / constraint system connecting PatternPiece, SeamEdge, Dart, Notch, and MaterialRegion nodes to manufacturability, physics, and lifecycle-assessment constraints (Teikari et al., 6 Jan 2026). A plausible implication is that SemanticGarment is best understood not as one data structure, but as an architectural principle: semantics must remain manipulable across design, geometry, and verification.

3. Generation and reconstruction

The generation problem is treated differently depending on whether the target is an image, a sewing pattern, or a 3D garment. In part-aware image generation, DiffCloth performs structural cross-modal semantic alignment by solving a Hungarian bipartite matching problem between segmented garment parts and Attribute-Phrases, using CLIP similarity and a structural semantic consensus guidance during denoising (Zhang et al., 2023). GarmentAligner instead augments a latent diffusion model with an automatic component extraction pipeline, retrieval-augmented contrastive learning, and multi-level correction losses for semantic, spatial, and quantitative alignment; these losses directly supervise component text alignment, attention-to-mask alignment, and component counts (Zhang et al., 2024).

Garment restoration treats semantics as the inverse of virtual try-on. IGR restores a canonical garment image from a person image by combining two garment extractors: an IP-Adapter branch for low-level appearance features and a GarmNet branch for high-level garment semantics. These are fused into a Stable Diffusion v1.5 denoiser through garment fusion blocks that combine self-attention and cross-attention, and the model is trained with a coarse-to-fine strategy to improve fidelity and authenticity (Shen et al., 2024). This decomposition is noteworthy because it distinguishes appearance semantics from structural semantics within the same restoration process.

Pattern generation systems treat semantic inputs as program synthesis targets. Design2GarmentCode uses a Multi-Modal Understanding Agent and a DSL Generation Agent to map images, text, sketches, or combinations thereof into GarmentCode programs and a 122-token design configuration, with execution producing size-precise sewing patterns with correct stitches (Zhou et al., 2024). ChatGarment uses a fine-tuned LLaVA model to emit a JSON containing garment types, style descriptors, and 76 normalized float values, which are then decoded into simulation-ready sewing patterns (Bian et al., 2024). In both cases, generation is less about hallucinating garment appearance than about inferring a semantic program that determines geometry.

In 3D generation, the namesake SemanticGarment method uses 3D Gaussian Splatting with a 3D semantic clothing model derived from SMPL-X. Garment Gaussians are initialized on semantic body regions, optimized by Score Distillation Sampling under MVDream or Stable-Zero123 guidance, and edited without regenerating or relying on existing mesh templates (Wang et al., 21 Sep 2025). DAGSM follows a related but distinct route: body and each garment are separate GS-enhanced meshes, and garments are first optimized as free 2D Gaussians, then reconstructed as garment meshes and refined for view-consistent texture (Zhuang et al., 2024).

4. Editing and semantic control

Editing is one of the main reasons semantic representations are introduced at all. DiffCloth edits garments by replacing Attribute-Phrases in the prompt, re-running diffusion with attention-map replacement, and restricting modifications via blended masks derived from old and new AP attention maps. The method is explicitly designed so that manipulation-irrelevant regions remain unchanged (Zhang et al., 2023). GarmentAligner addresses a more granular regime: because its training losses supervise quantity, position, and component relations, the same machinery supports edits such as changing counts, relocating logos, or altering specific parts while preserving the rest of the garment (Zhang et al., 2024).

EditGarment turns editing into a dataset and evaluation problem. It defines six instruction categories aligned with real-world fashion workflows—Color Alteration, Material Replacement, Structural Alteration, Object Addition, Object Removal, and Object Replacement—and introduces Fashion Edit Score, a semantic-aware metric built from instruction-critical, instruction-dependent, and context-preserving binary questions over a semantic dependency graph (Yin et al., 5 Aug 2025). The dataset pipeline generated 52,257 candidate triplets and retained 20,596 high-quality triplets after filtering with $\mathbf{W} = [W_{full}, W_1, W_2, \dots, W_m],$1 (Yin et al., 5 Aug 2025). This directly counters the misconception that garment editing is reducible to generic image-editing prompts; the paper treats fabric, silhouette, and object-level changes as dependent semantic operations rather than isolated token substitutions.

Programmatic editing is even more explicit. ChatGarment supports multi-turn dialogue in which a source garment JSON is revised according to instructions and recompiled into sewing patterns, while Design2GarmentCode includes a “design comparison” loop in which a multimodal agent compares the generated garment with the input and proposes targeted edits that are translated back into code and parameters (Bian et al., 2024, Zhou et al., 2024). In 3D, SemanticGarment supports both global and local editing: global texture edits update color-related Gaussian attributes under SDS, whereas local edits prune Gaussians in a semantic region, densify new ones from the semantic clothing model, and optimize only that region (Wang et al., 21 Sep 2025).

Garment Particles generalize editing across modalities. Because each particle jointly encodes 2D pattern coordinates and 3D draped coordinates, diffusion posterior sampling can impose constraints either on $\mathbf{W} = [W_{full}, W_1, W_2, \dots, W_m],$2 for sewing-pattern edits or on $\mathbf{W} = [W_{full}, W_1, W_2, \dots, W_m],$3 for 3D geometry edits, with Particles-to-Pattern Flow reconstructing simulation-ready patterns afterward (Nakayama et al., 25 May 2026). This suggests a broader view of semantic editing: it is not merely prompt-conditioned appearance change, but constrained posterior inference in a representation that keeps multiple garment views synchronized.

5. 3D, animation, and physics-aware semantics

The 3D turn in SemanticGarment research is driven by three persistent issues: garment fitting to bodies, multi-view consistency, and animation. SemanticGarment addresses these through an SMPL-X-based semantic clothing model. The posed body is represented as

$\mathbf{W} = [W_{full}, W_1, W_2, \dots, W_m],$4

and garment Gaussians are initialized on fine-grained semantic regions such as torso, arms, legs, chest pattern areas, or accessories (Wang et al., 21 Sep 2025). This makes fitting and local editing body-aware from the outset. The same paper adds a self-occlusion optimization strategy for single-image reconstruction, with position, color, and smoothness losses to reduce holes and artifacts when animating self-occluded garments (Wang et al., 21 Sep 2025).

DAGSM treats semantic disentanglement as the central design principle for clothed avatars. The body in underwear is one GSM, each garment is another GSM, and SAM-based semantic filtering removes Gaussians that do not belong to the current garment during optimization. The resulting representation supports clothing replacement and realistic cloth animation because garment meshes are separate from the body mesh and textures are carried by attached 2D Gaussians (Zhuang et al., 2024). Textile IR pushes the same idea into fashion engineering: garments are scene graphs under geometric, physical, and sustainability constraints, checked through a seven-layer Verification Ladder ranging from syntax and seam compatibility to simulation stability, fit / pressure analysis, robotic feasibility, and LCA / DPP validation (Teikari et al., 6 Jan 2026).

These systems show that semantics in 3D garment research are not limited to naming parts. They also encode where a garment may lie on the body, which regions may be edited independently, how a simulation failure maps back to a pattern modification, and how uncertainty should be attached to material properties or impact estimates. Textile IR’s emphasis on explicit confidence bounds and compound uncertainty makes this point especially clear (Teikari et al., 6 Jan 2026).

6. Evaluation, limits, and open directions

Evaluation practices vary with representation. On CM-Fashion, DiffCloth reported FID $\mathbf{W} = [W_{full}, W_1, W_2, \dots, W_m],$5, IS $\mathbf{W} = [W_{full}, W_1, W_2, \dots, W_m],$6, and CLIPScore $\mathbf{W} = [W_{full}, W_1, W_2, \dots, W_m],$7, while GarmentAligner reported FID $\mathbf{W} = [W_{full}, W_1, W_2, \dots, W_m],$8, CLIPScore $\mathbf{W} = [W_{full}, W_1, W_2, \dots, W_m],$9, AestheticScore $V_i$0, and HPSv2 $V_i$1 (Zhang et al., 2023, Zhang et al., 2024). Design2GarmentCode evaluated text-guided pattern generation with SSR $V_i$2, Agreement $V_i$3, and Aesthetic $V_i$4, and image-guided generation with SSR $V_i$5 and Agreement $V_i$6 (Zhou et al., 2024). The 3D Gaussian SemanticGarment reported CLIP-L/14 $V_i$7 in text-guided generation and user-study scores of 4.6 for 3D consistency, 4.313 for text alignment, and 4.475 for image quality (Wang et al., 21 Sep 2025). These metrics are heterogeneous, but they collectively show that semantic garment methods are evaluated not only by realism, but also by alignment, structural validity, and edit locality.

Important limitations recur across the literature. DiffCloth is sensitive to noisy text because constituency parsing and Attribute-Phrase extraction can fail (Zhang et al., 2023). Design2GarmentCode and ChatGarment inherit the expressiveness limits of GarmentCode-like DSLs; if the underlying garment program does not support a concept, the semantic system cannot represent it faithfully (Zhou et al., 2024, Bian et al., 2024). SemanticGarment’s current animation uses LBS and works better for fitted garments than for loose garments such as skirts or flowing coats (Wang et al., 21 Sep 2025). EditGarment depends on the reliability of foundation models such as Qwen-VL and DeepSeek R1 for both data synthesis and semantic scoring (Yin et al., 5 Aug 2025). Textile IR, finally, remains a conceptual intermediate representation rather than a finalized formal standard (Teikari et al., 6 Jan 2026).

A common misconception is that semantic garment modeling is equivalent to stronger prompt engineering. The literature instead indicates a more structural claim: semantics become operational only when tied to explicit part decompositions, executable sewing abstractions, or 3D human-aware representations. Another misconception is that a single representation can solve generation, editing, manufacturability, and simulation simultaneously. Current work suggests otherwise. Part-aware diffusion, program synthesis, scene-graph IRs, 3D Gaussian garments, and 5D garment particles each solve different parts of the problem, and their coexistence is a defining feature of the field (Nakayama et al., 25 May 2026, Teikari et al., 6 Jan 2026).

Taken together, SemanticGarment describes a transition from garments as visual outputs to garments as structured computational objects. Whether realized through Attribute-Phrases and masks, DSLs and JSON, scene graphs and verification ladders, or joint 2D–3D particles and body-aware Gaussians, the central aim is consistent: to make garment semantics explicit enough that they can drive generation, constrain editing, preserve manufacturability, and remain meaningful under drape, animation, and physical validation.