SDComp: Semantic & Sample Distortion Compression

• SDComp is a dual-paradigm framework that combines semantically disentangled compression and sample distortion techniques to enhance machine-centric image analysis.
• It employs large multimodal models for semantic disentangling, enabling prioritized encoding of critical image regions for tasks like detection and segmentation.
• The sample distortion component defines theoretical MSE limits and optimizes sample allocation using methods such as reverse water-filling and turbo-AMP.

SDComp denotes two distinct but thematically linked paradigms in advanced image compression and compressed sensing: (1) Semantically Disentangled Compression for machine-centric image coding via large multimodal models (Liu et al., 2024) and (2) the Sample Distortion framework for compressive imaging, focused on sample allocation and theoretical limits for decoder performance (Guo et al., 2013). Both frameworks emphasize tailored representation and allocation strategies that diverge from purely human-centric or naive metric-optimized encoding, enabling more efficient support for downstream inference or reconstruction objectives.

1. Semantically Disentangled Compression for Machine Tasks

Traditional codecs (JPEG, HEVC, VVC) are optimized for pixel fidelity or perceptual quality under metrics such as mean-squared error (MSE) or SSIM, objectives fundamentally mismatched to vision tasks such as object detection or segmentation. In semantic compression for machine analysis, the goal shifts: the criterion becomes minimization of a joint rate and task-specific distortion,

$\min \; R + \lambda D_{\mathrm{task}}(X, \hat{X}),$

where $D_{\mathrm{task}}$ reflects task-driven performance loss (e.g., $1-\mathrm{mAP}$ for detection, $1-\mathrm{mIoU}$ for segmentation, or $1-\mathrm{accuracy}$ for classification).

SDComp implements this paradigm through sequential semantic disentangling and prioritized encoding, orchestrated by large multimodal models (LMMs) with detailed visual grounding and importance ranking, yielding a structured, semantically interpretable bitstream for flexible machine analysis (Liu et al., 2024).

2. Architecture and Semantic Protocol

The SDComp codec proceeds in two principal stages:

(A) Semantic Disentangling via LMMs

• Visual grounding (Grounding-DINO + SAM) isolates objects/regions, generating binary masks for each candidate object and background.
• InternVL-Chat-V1.5, a powerful LMM, is prompted for hierarchical semantic annotation:
  • Short and long captions summarizing scene content.
  • Open-vocabulary importance ranking of objects using structured prompts referencing bounding box and label information.

(B) Semantically Structured Image Compression (SSIC)

• Objects are partitioned into $L_1$ (most critical), $L_2$, $L_3$ (least critical), and background.
• Each importance level is encoded via a dedicated autoencoder branch (ELIC backbone), with bit allocation proportional to their LMM-derived importance scores $s_i = (4 - \ell(i))/\sum_j (4 - \ell(j))$.
• The bitstream header contains explicit semantic metadata (captions, labels, importance levels), followed by region-wise arithmetic-coded payloads in descending order of priority.
• Decoding is progressive: downstream tasks requiring only coarse scene understanding can decode $L_1$ alone for substantial bitrate savings, whereas tasks needing full fidelity (e.g., dense segmentation) can progressively decode $D_{\mathrm{task}}$0, $D_{\mathrm{task}}$1, and background as needed (Liu et al., 2024).

3. Sample Distortion Function and Theoretical Envelope

In compressive imaging, SDComp as "Sample Distortion" framework (Guo et al., 2013) formalizes fundamental reconstruction limits for i.i.d. sources $D_{\mathrm{task}}$2 under encoder–decoder pairs $D_{\mathrm{task}}$3 and sampling ratio $D_{\mathrm{task}}$4. The SD function $D_{\mathrm{task}}$5 is defined as: $D_{\mathrm{task}}$6 characterizing the minimum achievable MSE at sampling rate $D_{\mathrm{task}}$7.

Universal lower bounds:

• The Entropy-Based Bound (EBB), generalizing Shannon's source coding theorem to underdetermined sampling:

$D_{\mathrm{task}}$8

• The Model-Based Bound (MBB), for Gaussian mixture models, allocating samples to coefficients with largest prior variances.

Convexity property:

• $D_{\mathrm{task}}$9 is strictly convex; any non-convex empirical curve can be convexified via zeroing protocols, partitioning the signal and sampling subblocks at distinct rates, to time-share between allocations.

4. Optimized Sample Allocation and Bandwise Strategy

For images with structured statistical models (e.g., wavelet decompositions), SDComp prescribes bandwise sample allocation by reverse water-filling: $1-\mathrm{mAP}$0 where $1-\mathrm{mAP}$1 directs sampling resources to bands with greatest marginal distortion reduction.

Integration with hidden Markov tree (HMT) priors (Som & Schniter turbo-AMP) further propagates structure by alternately updating band/posterior coefficients via approximate message passing and refining support probabilities over the wavelet tree (Guo et al., 2013).

5. Training Objectives and Evaluation Protocols

In semantically disentangled compression, end-to-end training minimizes

$1-\mathrm{mAP}$2

per importance bucket. $1-\mathrm{mAP}$3 is tuned per downstream task, trading off bitrate and task performance.

Evaluation protocols leverage mainstream vision tasks:

• Instance segmentation: Mask R-CNN (X101-FPN), mIoU vs. bpp.
• Object detection: Faster R-CNN, AP vs. bpp.
• Classification: ResNet-50, Top-1 accuracy vs. bpp.
• VQA: LLaVA on VizWiz, answer accuracy vs. bpp (Liu et al., 2024).

In compressive imaging, predictive accuracy of the SD curve is validated empirically with simulations on natural images, comparing predicted versus observed PSNR under various sample allocation strategies (uniform, InfoMax, SA/GSA, turbo AMP with HMT). SD-optimal allocation consistently outperforms non-adaptive heuristics, particularly at low sampling ratios (Guo et al., 2013).

6. Comparative Results and Interpretability

SDComp shows significant bitrate savings for machine tasks compared to human-centric codecs:

• SDComp vs. VTM: BD-rate improvement of –31.4% (COCO segmentation), –33.2% (COCO detection), –12.8% (CUB classification).
• Ablation: Decoding only $1-\mathrm{mAP}$4 achieves optimal rate–accuracy for fine-grained classification; $1-\mathrm{mAP}$5 marginally benefit dense prediction tasks (Liu et al., 2024).

Visualization confirms interpretability: regions prioritized by LMM ranking are the most semantically salient per InternVL explanations, and even with aggressive bitrate reduction (only $1-\mathrm{mAP}$6 transmitted), scene-level semantic content is retained in the global caption.

In compressive imaging, rigorous simulation confirms close match between theoretical SD bounds (state evolution) and actual recovery for the Bayes-optimal decoder. Turbo-AMP with HMT guidance approaches the model-based lower bound at low $1-\mathrm{mAP}$7 (Guo et al., 2013).

7. Limitations, Extensions, and Implications

SDComp for semantic compression incurs computational overhead due to LMM prompting and encoding of headers. Applicability is currently limited to static images; extension to video demands temporal object tracking and dynamic bit allocation, as well as potential integration of lightweight, jointly-trained LMM adapters (Liu et al., 2024).

Sample Distortion as a unifying analytic framework is predicated on accurate prior/statistical models. Oracle sample allocation requires knowledge of per-image statistics; using average statistics yields robust, general gains, but ultimate performance is adaptive to model fit.

A plausible implication is that both paradigms illustrate the necessity of task-adaptive allocation—whether semantic, statistical, or both—for efficient visual representation in automated systems. Future research directions include semantically guided video compression and end-to-end joint optimization of codecs and semantic annotators, as well as dynamic, on-the-fly adaptivity to emerging tasks or priors (Liu et al., 2024, Guo et al., 2013).