ImgCoder: Semantic & Code-Driven Image Coding

• ImgCoder is a collection of frameworks that redefines image coding by focusing on semantic, task-specific representations rather than pixel-perfect reconstruction.
• It employs layered, latent, and code-driven methods to efficiently support machine vision tasks and scientific image synthesis with significant bitrate savings.
• The approach integrates scalable architectures and unified feature coding to optimize for diverse tasks such as detection, segmentation, and reconstruction.

ImgCoder is best understood as a family of image representation and coding frameworks in which compression or generation is organized around downstream use rather than around a single human-visual rate–distortion objective. In one major line of work, it denotes image coding for machines (ICM): compressed representations are designed to preserve what downstream machine vision models need for classification, detection, segmentation, VQA, or captioning, often by transmitting semantic structure, latent features, or task-conditioned subsets instead of uniformly preserving all pixels (Liu et al., 2024, Chen et al., 2021). In a different line of work, “ImgCoder” denotes a logic-driven framework for scientific image synthesis that follows an explicit “Understand → Plan → Code” workflow and renders Python/Matplotlib programs rather than sampling pixels directly (Lin et al., 17 Jan 2026). The literature therefore suggests that the term does not denote a single standardized codec, but a broader research program in which images are coded, structured, or generated according to semantic, task, or logical constraints.

1. Scope and nomenclature

Two recurring usages dominate the literature. The first usage concerns machine-oriented image coding, where the coded representation is optimized for downstream vision models and may optionally remain decodable for humans. The second usage concerns code-driven image generation, where an image is produced from executable code rather than from direct pixel synthesis (Liu et al., 2024, Lin et al., 17 Jan 2026).

Usage of “ImgCoder”	Core representation	Representative work
Image coding for machines	Semantic streams, latent subsets, feature codecs, or structured bitstreams	(Chen et al., 2021, Choi et al., 2021, Ozyilkan et al., 2023, Feng et al., 2022, Feng et al., 2023, Liu et al., 2024)
Logic-driven scientific image synthesis	Python/Matplotlib code produced from scientific problem text	(Lin et al., 17 Jan 2026)
Adjacent but distinct image-to-code or style encoding	SVG-to-HTML+CSS or a 20-symbol base64 style code	(Toth-Czifra, 5 Sep 2025, Rowles, 2024)

This distinction matters because several adjacent systems operate on images yet are not codecs in the ICM sense. “Reverse Browser” is a vector-image-to-code generator whose output is HTML+CSS from SVG input, while “StyleCodes” is a style encoder that expresses image style as a 20-symbol base64 code; neither is presented as a general-purpose image compression method (Toth-Czifra, 5 Sep 2025, Rowles, 2024).

2. Machine-first and human-machine codec architectures

A foundational branch of ImgCoder research replaces monolithic pixel reconstruction with layered or semantically structured representations. “A New Image Codec Paradigm for Human and Machine Uses” extracts an instance segmentation map $s = H(x;\xi_H)$, converts it into a 16-bit grayscale profile $\tilde{s} = R(s)$, derives low-level signal features $y=L(\tilde{s},x;\xi_L)$, predicts a general-quality image $\tilde{x}=I(\tilde{s},y;\xi_I)$, and then codes the residual $r=x-\tilde{x}$. The transmitted representation is split into three streams: a losslessly coded semantic profile, losslessly coded low-level features, and a lossy residual coded with VVC. As a result, $stream1$ alone is enough for several machine tasks, $stream1+stream2$ are enough to synthesize a general-quality image, and all three streams together give a higher-quality reconstruction (Chen et al., 2021).

A second line organizes scalability directly in latent space. “Scalable Image Coding for Humans and Machines” partitions the latent tensor as $\mathbfcal{Y}=\{\mathbfcal{Y}_1,\mathbfcal{Y}_2,\ldots,\mathbfcal{Y}_S\}$, assigning the simplest task to the base layer and more complicated tasks to enhancement layers. In the 2-layer model, the base layer supports object detection and the enhancement layer supports input reconstruction; in the 3-layer model, the base layer supports detection, the base plus first enhancement supports segmentation, and all layers support reconstruction. The paper reports 37%-80% bitrate savings on machine vision tasks compared to best alternatives while being comparable to state-of-the-art image codecs in terms of input reconstruction (Choi et al., 2021).

A closely related formulation explicitly disentangles machine-task and reconstruction information. “Learned Disentangled Latent Representations for Scalable Image Coding for Humans and Machines” defines a base latent $\boldsymbol{y}_1$ for object detection and an enhancement latent $\boldsymbol{y}_2$ for input reconstruction, with the encoder conditioned on task-relevant feature data $\tilde{s} = R(s)$0 extracted from a pretrained detector front-end. Its practical objective is $\tilde{s} = R(s)$1, and the paper reports a bit rate savings of 40.6% on the primary object detection task compared to the current state-of-the-art, albeit with some degradation in performance for the secondary input reconstruction task (Ozyilkan et al., 2023).

3. Unified feature coding, prompting, and rate–distortion–cognition control

Another major trajectory treats ImgCoder as a shared feature codec rather than as an image codec with task-specific side streams. “Image Coding for Machines with Omnipotent Feature Learning” learns omnipotent features by combining contrastive self-supervision with an Information Filtering module between the backbone head and tail. The stage-1 loss is $\tilde{s} = R(s)$2 with $\tilde{s} = R(s)$3, and a separate learned feature codec is then trained on the resulting features. The same decoded feature representation is used directly for object detection, semantic segmentation, instance segmentation, panoptic segmentation, and pose estimation, and the paper reports BD-rate values including -35.1 for VOC detection, -43.9 for COCO detection, -42.8 for COCO instance segmentation, and -72.0 for Cityscapes semantic segmentation (Feng et al., 2022).

Prompt-based adaptation pushes this unification further. “Prompt-ICM” keeps a single shared feature extractor and a single shared controllable feature codec, then introduces compression prompts as task-driven importance maps $\tilde{s} = R(s)$4 and task-adaptive prompts $\tilde{s} = R(s)$5 for downstream transfer. The codec loss becomes $\tilde{s} = R(s)$6 with $\tilde{s} = R(s)$7, so bitrate allocation varies spatially according to the task. The parameter-efficiency result is especially notable for classification: 87.61M trainable parameters under full tuning versus 0.87M for Prompt-ICM (Feng et al., 2023).

A complementary approach strengthens the encoder rather than the prompt interface. “Improving Image Coding for Machines through Optimizing Encoder via Auxiliary Loss” attaches an auxiliary branch just before the decoder and optimizes $\tilde{s} = R(s)$8, or adds $\tilde{s} = R(s)$9 for semantic segmentation. The auxiliary branch is training-only, so the method yields 27.7% BD-rate improvement for object detection and 20.3% for semantic segmentation without inference-time overhead (Iino et al., 2024).

The most explicit attempt at tri-objective control is “Rate-Distortion-Cognition Controllable Versatile Neural Image Compression.” It uses a cognition-oriented primary branch, a scalable auxiliary residual branch, and the interpolation rule $y=L(\tilde{s},x;\xi_L)$0, equivalent to a $y=L(\tilde{s},x;\xi_L)$1 trade-off between a cognition-oriented image and a distortion-oriented image. Rate is adjusted by channel-controllable latent gain units, distortion is improved by the auxiliary branch, and cognition is emphasized as $y=L(\tilde{s},x;\xi_L)$2 (Liu et al., 2024).

4. Semantically structured codecs and foundation-model guidance

A distinctive recent usage of ImgCoder treats semantic reasoning itself as a control signal for compression. “Tell Codec What Worth Compressing: Semantically Disentangled Image Coding for Machine with LMMs” proposes SDComp, a pipeline in which Grounded-SAM decomposes an input image $y=L(\tilde{s},x;\xi_L)$3 into $y=L(\tilde{s},x;\xi_L)$4, InternVL-Chat-V1.5 generates a short caption $y=L(\tilde{s},x;\xi_L)$5 and a detailed caption $y=L(\tilde{s},x;\xi_L)$6, and a second prompt ranks grounded objects into three levels $y=L(\tilde{s},x;\xi_L)$7 with $y=L(\tilde{s},x;\xi_L)$8. The codec then groups regions into $y=L(\tilde{s},x;\xi_L)$9, $\tilde{x}=I(\tilde{s},y;\xi_I)$0, $\tilde{x}=I(\tilde{s},y;\xi_I)$1, plus “other”, and encodes them sequentially with an upgraded SSIC built on ELIC. The resulting bitstream is semantically ordered and progressively decodable: only $\tilde{x}=I(\tilde{s},y;\xi_I)$2 may be decoded for very low bitrate, while $\tilde{x}=I(\tilde{s},y;\xi_I)$3 and “other” can be added for richer analysis or fuller reconstruction. Reported gains against VVC include BD-rate reductions of -31.38% for segmentation, -33.22% for detection, and -12.83% for classification, while the abstract states an average performance improvement of 32.3% over VTM on segmentation and detection tasks (Liu et al., 2024).

SDComp also makes explicit several current limitations of LMM-guided coding. Grounded-SAM plus an LMM like InternVL 1.5 is substantially heavier than a traditional front end; the method depends on grounding quality, caption quality, and ranking quality; and the paper does not provide a rigorous bitstream syntax specification, detailed entropy-coding design, or training recipe for the SSIC/ELIC integration. The work is therefore best understood as a semantic pre-analysis plus structured region coding framework rather than as a new end-to-end transform or entropy model (Liu et al., 2024).

Another semantic-supervision strategy is edge-centric rather than language-centric. “Image Coding for Machines with Edge Information Learning Using Segment Anything” uses SAM to generate segmentation masks, applies Canny edge detection to obtain $\tilde{x}=I(\tilde{s},y;\xi_I)$4, and trains the codec with $\tilde{x}=I(\tilde{s},y;\xi_I)$5. The method is designed to reconstruct object-part edge information instead of full texture, is evaluated with YOLOv5, Mask-RCNN, and Panoptic-DeepLab, and is claimed to be robust to changes in input data while also removing human facial information on the encoder’s side. The same idea extends to video through SA-NeRV, which improves object detection mAP on 7 of 8 reported SFU-HW-Objects-v1 sequences (Shindo et al., 2024).

5. Decoder refinement, feature-preserving optimization, and ambiguity-aware decoding

Not all ImgCoder work changes the coded representation itself; some work alters how a fixed representation is decoded or optimized. “Bridging the gap between image coding for machines and humans” starts from a learned ICM codec whose decoder exhibits checkerboard artifacts at low bitrate and applies decoder-only PatchGAN finetuning. The encoder is frozen, bitrate is unchanged, parameter count at inference is unchanged, and finetuning only the first 2 layers of the decoder is reported to suppress artifacts. At an average bitrate of 0.057 bpp, the base codec has mAP 0.766, PSNR 27.695, SSIM 0.709, and VGG 0.495; a typical finetuned setting reaches PSNR about 28.5 and SSIM about 0.76 with about -1.6% relative task-performance change (Le et al., 2024).

A more classical codec-oriented line embeds machine awareness directly into encoder RDO. “Image Coding for Machines via Feature-Preserving Rate-Distortion Optimization” begins from feature-distance RDO, linearizes the feature extractor with a Jacobian, and introduces input-dependent squared error (IDSE) as a blockwise approximation that can be evaluated in the transform domain. The method requires no decoder changes, only about a 7% encoder complexity increase, and reports up to 10% bitrate savings for the same computer vision accuracy compared to SSE-RDO (Fernández-Menduiña et al., 3 Apr 2025).

A different response to lossy ambiguity is not to optimize a single reconstruction at all. “What’s in the Image? Explorable Decoding of Compressed Images” treats JPEG decompression as a constrained set-valued problem. Its decoder predicts DCT-domain residuals $\tilde{x}=I(\tilde{s},y;\xi_I)$6 such that $\tilde{x}=I(\tilde{s},y;\xi_I)$7 with $\tilde{x}=I(\tilde{s},y;\xi_I)$8, ensuring exact JPEG consistency, and generates images as $\tilde{x}=I(\tilde{s},y;\xi_I)$9, where $r=x-\tilde{x}$0 is a dense control tensor. User tools then solve $r=x-\tilde{x}$1 to explore hypotheses of interest. This work is not an ICM codec in the narrow sense, but it is important for ImgCoder because it reframes decoding as controllable exploration of all natural images consistent with the compressed code (Bahat et al., 2020).

6. ImgCoder as logic-driven scientific image synthesis

The explicit title usage of ImgCoder appears in “Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility,” where ImgCoder is a code-driven framework for scientific image synthesis rather than a compression codec. The system follows an “Understand → Plan → Code” workflow: it parses the original scientific question, produces a four-part plan comprising Image Content, Layout, Labels, and Drawing Considerations, and then generates standalone Python/Matplotlib code for deterministic rendering. The paper evaluates scientific image generation with SciGenBench, a 1.4K-problem benchmark across 5 domains and 25 image types, and reports inverse validation rates of 56.38 for Qwen3-ImgCoder, 76.93 for Gemini-3-Flash-ImgCoder, and 77.87 for Gemini-3-Pro-ImgCoder, exceeding the best pixel-based baseline Nanobanana-Pro at 73.41 (Lin et al., 17 Jan 2026).

This scientific-synthesis version of ImgCoder is motivated by “visual–logic divergence”: pixel-based text-to-image systems often produce diagrams that are visually plausible yet scientifically wrong. ImgCoder addresses that problem by replacing unconstrained pixel synthesis with explicit planning plus executable code. The paper’s downstream study further reports that fine-tuning Qwen3-VL-8B-Instruct on verified synthetic images improves reasoning benchmarks over the base model, with all synthetic-image training conditions outperforming the base average score of 54.5 on Geometry3K$r=x-\tilde{x}$2 and MathVision$r=x-\tilde{x}$3 (Lin et al., 17 Jan 2026).

The scientific-synthesis usage also clarifies the boundary of the term. Adjacent systems such as “Reverse Browser” and “StyleCodes” likewise convert image-derived information into compact symbolic or programmatic forms, but their targets are different: Reverse Browser maps SVG webpage renderings to HTML+CSS, and StyleCodes maps image style to a 20-symbol base64 code for diffusion-based generation rather than to a full image representation (Toth-Czifra, 5 Sep 2025, Rowles, 2024). A plausible implication is that current usage of “ImgCoder” spans two linked ideas—machine-oriented image coding and code-driven structured image generation—unified less by a single architecture than by a common design principle: image representations should be organized around the structure, semantics, and downstream reasoning that matter.