Semantic Image Communication (SIC)

• Semantic Image Communication (SIC) is a design principle that encodes key image semantics rather than pixel data to support perceptually consistent and task-oriented decoding.
• It integrates various paradigms—including neural JSCC, segmentation-driven, language-oriented, and generative approaches—to adapt transmission based on channel conditions and application needs.
• Research shows that SIC systems can gracefully degrade under low signal-to-noise ratios while delivering improved robustness and flexibility compared to traditional codecs.

Searching arXiv for recent and foundational papers on Semantic Image Communication. Semantic Image Communication (SIC) denotes a class of communication systems in which the transmitted object is not a pixel-faithful bitstream but a semantic representation sufficient for downstream perception, reconstruction, or decision-making. Across the recent literature, SIC has been instantiated as end-to-end neural joint source–channel coding (JSCC), as transmission of structured visual semantics such as segmentation maps, color palettes, texture maps, or scene graphs, and as language-oriented or generative systems in which captions or latent image embeddings condition a generative decoder at the receiver. The common departure from conventional source–channel separation is that reconstruction quality is judged by semantic consistency, perceptual quality, or task performance rather than by bit accuracy alone, and many SIC systems are explicitly reported to degrade gracefully with signal-to-noise ratio (SNR) instead of exhibiting catastrophic decoder failure (Zhang et al., 2022, Hosonuma et al., 2024, Khalid et al., 23 Jul 2025).

1. Conceptual scope and defining characteristics

SIC is not a single architecture but a design principle. In one line of work, an image $\mathbf{s}$ is mapped directly by a neural encoder to a channel input vector $\mathbf{x}$, with the receiver applying a neural decoder to reconstruct $\hat{\mathbf{s}}$ without an explicit bitstream, channel code, or standard modulation. In another line, the transmitter extracts discrete or interpretable semantic carriers—semantic segmentation labels, captions, scene graphs, texture maps, color palettes, or category-specific feature maps—and the receiver reconstructs either an image or a task output from those carriers. In a third line, SIC is explicitly generative: only semantic descriptors are transmitted, and a diffusion model, GAN, or other generative model synthesizes a plausible image conditioned on them (Zhang et al., 2022, Lokumarambage et al., 2023, Wei et al., 2024).

A central distinction from conventional image communication is the target of preservation. Conventional systems perform source coding, then channel coding, then modulation, and are optimized for bit-level or pixel-level fidelity. SIC instead targets “what is in the image,” “where it is,” and “what is needed for the task.” This shift appears in formulations based on semantic segmentation for AIoT and IoV, in language-oriented portrait transmission, in scene-graph transmission, and in latent diffusion systems conditioned on compact image embeddings rather than on raw pixels (Qian et al., 2023, Pan et al., 2022, Zhu et al., 16 Jul 2025).

A persistent misconception is that SIC is synonymous with caption-only image generation. The literature is broader. Some systems transmit only text, but others transmit segmentation maps, class-level color palettes, local texture descriptors, scene-graph triplets with layouts, shared semantic bases, or compact latent tensors such as $[16,12,12]$ or $[16,24,24]$, and several systems combine multiple modalities precisely because a single semantic view is often insufficient for stable reconstruction (Wei et al., 2024, Fan et al., 2024, Khalid et al., 23 Jul 2025).

2. Representational paradigms

The literature currently clusters around a small number of recurrent semantic representations.

Paradigm	Transmitted representation	Representative papers
Neural JSCC latent	Continuous latent semantic representation or channel symbols	(Zhang et al., 2022, Chen et al., 30 Apr 2025, Khalid et al., 23 Jul 2025)
Segmentation-driven SIC	Semantic segmentation maps, masks, or segmented slices	(Lokumarambage et al., 2023, Qian et al., 2023, Pan et al., 2022)
Language-oriented SIC	Captions or multi-text descriptions extracted by BLIP or LLaVA	(Wei et al., 2024, Huang et al., 25 Mar 2025)
Multi-modal generative SIC	Caption + segmentation + palette, or text + texture + color	(Hosonuma et al., 2024, Fan et al., 2024)
Symbolic semantic SIC	Scene graphs, layouts, and probability-graph-compressed triplets	(Zhu et al., 16 Jul 2025)
Explicit semantic-basis SIC	Sebs, usage indices, and residuals	(Zheng et al., 2023)

In JSCC-style SIC, the latent itself is the semantic carrier. A representative formulation writes

$$\mathbf{x} = f(\mathbf{s}; \boldsymbol{\Phi}), \quad \mathbf{x} \in \mathbb{R}^{M \times 1},$$

with the decoder reconstructing $\hat{\mathbf{s}}$ from the noisy received signal. Residual blocks, GDN, and sub-pixel convolutions are used to improve rate–distortion efficiency, and in latent-diffusion variants the transmitted signal is not an image-space representation but an image embedding that conditions a generative model (Zhang et al., 2022, Khalid et al., 23 Jul 2025).

In segmentation-driven SIC, the semantic carrier is an object- or pixel-level label field. One implementation transmits a semantic segmentation map over a wireless channel and uses a SPADE GAN to regenerate a realistic image at the receiver; another uses a high-precision segmentation network at the transmitter and a semantic restoration GAN at the receiver; a third transmits task-oriented features and reconstructs the segmentation itself rather than the image (Lokumarambage et al., 2023, Qian et al., 2023, Pan et al., 2022). These systems treat semantic layout as the essential information and delegate appearance synthesis to a learned prior.

Language-centered SIC compresses image semantics into text. In portrait transmission, an Img2Txt model based on BLIP produces a short description, a transformer-based semantic text codec maps it to channel symbols, and Stable Diffusion reconstructs a portrait from the decoded text. Multi-text transmission extends this idea by partitioning the image into blocks and using a modified LLaVA to extract several region-specific descriptions, which are later fused with a segmented main-body slice through IP-Adapter (Wei et al., 2024, Huang et al., 25 Mar 2025).

Multi-modal generative SIC arises because text alone is often underdetermined. One framework transmits caption, segmentation array, and class-level color palette; another decomposes images into natural language description, texture semantic feature maps derived from LBP, and heavily downsampled color maps. These systems make explicit that different semantic modalities encode different invariants: captions capture object and context semantics, segmentation captures spatial arrangement, texture captures local structure, and palettes preserve dominant appearance cues (Hosonuma et al., 2024, Fan et al., 2024).

Symbolic SIC replaces dense features with explicit relational semantics. Scene-graph systems transmit objects, relations, and layouts, then exploit shared probability graphs to omit predictable relations and entities; explicit semantic-base systems cluster latent patch features into Sebs, transmit Sebs and usage indices, and reconstruct a reference image plus residuals. Both directions attempt to externalize semantic knowledge instead of leaving it entirely implicit in network weights (Zhu et al., 16 Jul 2025, Zheng et al., 2023).

3. Architectures and channel-aware system models

End-to-end JSCC remains one of the main architectural templates. In the secure residual-based image JSCC system, the encoder alternates residual blocks and convolution layers, performs three stages of downsampling, applies GDN after each block, and outputs a latent vector of length $M$ under an average power constraint

$\frac{1}{M}\mathbb{E}\{x_i^2\} \le p.$

Over an AWGN wiretap channel,

$$\mathbf{y}_b = \mathbf{x} + \mathbf{n}_b,\qquad \mathbf{y}_e = \mathbf{x} + \mathbf{n}_e,$$

and in the MISO case with maximal ratio transmission,

$\mathbf{x}$0

The decoder mirrors the encoder with residual blocks, convolutions, and sub-pixel convolutions for upsampling (Zhang et al., 2022).

Task-oriented SIC over IoV adopts a different objective but a comparable end-to-end channel-aware pipeline. The ISSC encoder is a Swin Transformer–based multi-scale semantic feature extractor followed by a semantic feature aggregator; the channel is modeled as

$\mathbf{x}$1

with $\mathbf{x}$2; and the receiver consists of a semantic feature decoder and a reconstructor producing a segmentation map rather than an RGB image (Pan et al., 2022).

Generative SIC replaces deterministic decoders with conditional generators. In image generative semantic communication, the receiver constructs a colored segmentation image from the segmentation array and palette, feeds it together with a caption into Stable Diffusion, generates multiple candidates, re-extracts captions and segmentations from them, and selects an output by semantic similarity. A related text-based system uses a BLIP caption, a transformer JSCC text codec, and a Stable Diffusion decoder. Another system uses IP-Adapter to fuse a noisy segmented main-object slice with multiple region-specific captions through decoupled text and image cross-attention

$\mathbf{x}$3

These designs treat generation as semantic decoding rather than as post-processing (Hosonuma et al., 2024, Wei et al., 2024, Huang et al., 25 Mar 2025).

Recent diffusion-oriented SIC increasingly operates in latent rather than pixel space. The Stable Cascade system uses a pretrained EfficientNet-V2 encoder to extract a compact image embedding $\mathbf{x}$4, transmits $\mathbf{x}$5 over AWGN as

$\mathbf{x}$6

then conditions Stage B latent diffusion on $\mathbf{x}$7 to generate a VQGAN latent and reconstruct the image. The reported embedding for $\mathbf{x}$8 images has shape $\mathbf{x}$9, corresponding to 0.29% of the original dimensionality. This shifts the bottleneck from pixel compression to semantic conditioning strength (Khalid et al., 23 Jul 2025).

Other generative systems preserve interpretability by decomposing the condition itself. TCSCI decomposes images into text, texture, and color; ControlNet branches inject texture and color conditions into Stable Diffusion; and semantic-feature transmission is separated from image restoration. In explicit-semantic-base systems, reference image generation is itself semantic decoding: Sebs are latent patch prototypes decoded into reference patches, concatenated into a reference image, and refined by a residual branch (Fan et al., 2024, Zheng et al., 2023).

4. Learning objectives and evaluation methodology

Loss design in SIC reflects a shift from pixel reconstruction toward semantic and perceptual objectives. In plain JSCC image reconstruction, a baseline objective is mean squared error:

$\hat{\mathbf{s}}$0

This objective appears in semantic JSCC systems that target image fidelity at the legitimate receiver, but several works argue that MSE alone is misaligned with semantic preservation (Zhang et al., 2022).

A more explicit semantic formulation appears in RL-ASC, where the optimization criterion is

$\hat{\mathbf{s}}$1

Here $\hat{\mathbf{s}}$2 is task-specific semantic degradation computed from a downstream network $\hat{\mathbf{s}}$3, and $\hat{\mathbf{s}}$4 is a VGG-based perceptual loss. The semantic representation unit is a class-specific semantic concept $\hat{\mathbf{s}}$5, and an RL policy allocates quantization levels to semantic concepts so as to maximize improvement in rate–semantic–perceptual performance (Huang et al., 2022).

For semantic robustness against adversarial perturbations, DeepSC-RI introduces the Image Semantic Impairment Intensity (ISII),

$\hat{\mathbf{s}}$6

and trains with

$\hat{\mathbf{s}}$7

The architecture couples a fine-grained branch with semantic importance masking, a coarse-grained branch with hierarchical pooling, and cross-attention fusion, explicitly targeting semantic rather than merely physical impairments (Peng et al., 2024).

Diffusion-based SIC often uses denoising objectives on latent or pixel-space trajectories. In the Stable Cascade SIC system, the conditional latent diffusion loss is

$\hat{\mathbf{s}}$8

while LRISC adopts a rate–distortion–perception objective that combines MSE-like distortion, LPIPS, and entropy-model rate terms. LRISC also introduces SNR-adaptive modulation modules in the latent JSCC codec so that one model operates across varying channel states (Khalid et al., 23 Jul 2025, Chen et al., 30 Apr 2025).

Evaluation practice in SIC is therefore heterogeneous by design. Image reconstruction works use PSNR, SSIM, LPIPS, and FID; segmentation-oriented works use mIoU; text-centered systems use BLEU; generative semantic selection systems use BERTScore and Segmentation Matching Rate (SMR),

$\hat{\mathbf{s}}$9

scene-graph systems use CLIP similarity and LPIPS; and trustworthy or task-oriented systems often prioritize downstream task accuracy over pixel-level metrics (Hosonuma et al., 2024, Wang et al., 2024, Zhu et al., 16 Jul 2025). A recurrent theme is that PSNR or SSIM alone is insufficient once reconstruction is intentionally generative.

5. Security, robustness, explainability, and controllability

A major conceptual correction in SIC is that semantic efficiency does not imply semantic security. In security-aware semantic JSCC for wireless image transmission, the same property that makes semantic communication effective at low SNR—recoverability of high-level content from noisy observations—also increases privacy leakage. Under the public decoder assumption $[16,12,12]$0, an eavesdropper can exploit the same decoder architecture as the legitimate user, and a conventional hard-failure threshold no longer protects semantics (Zhang et al., 2022).

To address this, SecureMSE augments Bob’s reconstruction loss with a privacy penalty that drives Eve’s reconstruction toward an all-black image. The proposed objective is

$[16,12,12]$1

with

$[16,12,12]$2

This yields an efficiency–privacy trade-off rather than a secrecy-capacity guarantee, and the paper explicitly notes the absence of a closed-form secrecy characterization (Zhang et al., 2022).

Robustness has similarly expanded beyond AWGN tolerance. DeepSC-RI studies semantic impairments induced by adversarial perturbations and shows that robustness can be built into the communication layer through multi-grained semantic extraction and fusion rather than only through adversarially retraining downstream models. Scene-graph SIC exploits shared probability graphs $[16,12,12]$3 and $[16,12,12]$4 to recover omitted relations and entities, making robustness partly a function of shared semantic priors rather than solely of channel coding (Peng et al., 2024, Zhu et al., 16 Jul 2025).

Explainability and controllability are active themes because many JSCC systems are opaque. Trustworthy ISC replaces uninterpretable latent vectors with image semantic text, A-seg, B-seg, and sub-images, enabling semantic-level multi-rate transmission and task-conditioned semantic selection. TCSCI similarly argues for Semantic Feature Decomposition (SeFD), in which text, texture, and color are separate, human-interpretable channels. These explicit semantic carriers make editing possible: color-only or texture-only control can alter corresponding aspects of the generated image while keeping other semantics fixed (Wang et al., 2024, Fan et al., 2024).

A related misconception is that explainability and compatibility are natural consequences of using neural representations. The trustworthy-ISC literature argues the opposite: complex-valued or opaque JSCC features are hard to integrate with existing digital stacks and difficult to inspect, whereas discrete text, segmentation maps, and other structured carriers are easier to transmit, debug, and reuse across tasks (Wang et al., 2024).

6. Applications, limitations, and open research directions

Application domains in SIC are broad but structurally similar. IoV work targets cooperative perception, where front vehicles communicate semantic features so rear vehicles can reconstruct segmentation maps for driving decisions under spectrum constraints. AIoT work targets smart factories, surveillance, autonomous driving, and smart cities, emphasizing segmentation-to-image restoration, compression ratio, and total delay. Generative SIC papers explicitly mention monitoring specific objects, event viewing, underwater sensing, satellite sensing, and portrait transmission under severe resource constraints (Pan et al., 2022, Qian et al., 2023, Hosonuma et al., 2024, Wei et al., 2024).

Edge/cloud partitioning is another recurring deployment model. Stable Cascade SIC is explicitly compatible with an edge device that encodes an image into $[16,12,12]$5 and a cloud that runs heavy latent diffusion and VQGAN decoding. Trustworthy ISC similarly separates explainable semantic extraction at the transmitter from GenAI-based downstream inference at the receiver. This suggests that SIC is increasingly becoming an architectural interface between communication and foundation models rather than merely a better codec (Khalid et al., 23 Jul 2025, Wang et al., 2024).

The limitations are equally consistent across papers. Many systems remain domain-dependent: Cityscapes-trained generative SIC shows visible style bias on DIV2K; portrait-oriented language SIC relies on few-shot DreamBooth fine-tuning of SDXL on CelebA; segmentation-based systems depend on the ontology and distribution of COCO-Stuff, Cityscapes, or Linnaeus 5 (Khalid et al., 23 Jul 2025, Wei et al., 2024, Lokumarambage et al., 2023). Several works explicitly lack formal information-theoretic guarantees, especially for privacy, semantic rate–distortion, or semantic capacity (Zhang et al., 2022, Zhu et al., 16 Jul 2025).

Computational cost remains a major systems bottleneck. Diffusion decoders are slower than CNN decoders even when latent-space models such as Stable Cascade materially reduce latency. TCSCI and trustworthy ISC both identify large generative models as deployment obstacles for resource-constrained devices. Training cost is also nontrivial; the resource-allocation study for training SIC networks frames time–energy trade-offs during distributed training as a primary systems challenge (Fan et al., 2024, Wang et al., 2024, Li et al., 8 Jan 2025).

Several open problems recur across the literature. One is stronger semantic metrics: object-focused or class-weighted SMR, learned multi-modal semantic similarity, and better approximations to mutual information at the semantic level are all proposed directions. Another is stronger adversaries and dynamic control: public-decoder security models do not cover retrained Eves, and fixed $[16,12,12]$6 or $[16,12,12]$7 in privacy-aware training does not yield on-the-fly security control. A third is broader semantic structure: scene-graph systems already explore multi-round probabilistic compression, while SeFD-based systems suggest adding depth or other semantic channels at higher bitrates. Video, speech, and multi-modal SIC are natural continuations, but temporal coherence, semantic consistency across frames, and scalable training remain open (Hosonuma et al., 2024, Zhang et al., 2022, Fan et al., 2024, Zhu et al., 16 Jul 2025).

Taken together, the literature shows SIC evolving along three simultaneous axes: from pixel fidelity to semantic adequacy, from black-box latent transmission to increasingly explicit semantic carriers, and from deterministic reconstruction to receiver-side generative priors. The field has not converged on a single canonical representation or metric, but the major directions—JSCC latents, structured semantics, language-centered semantics, and generative latent conditioning—now define a coherent technical landscape for semantic-first image communication.