JPNeO: Trainable JPEG Processing Operator

• JPNeO is a learnable JPEG operator that treats JPEG components (color transform, DCT, quantization) as differentiable modules, enabling joint optimization in pipelines.
• It supports both embedded differentiable modules for task-aware image processing and backward-compatible coding that maintains standard compliance while enhancing fidelity.
• JPNeO employs differentiable approximations—such as straight-through estimators and cubic surrogates—to enable gradient flow and optimize the rate–accuracy trade-off.

to=arxiv_search.search 利盛json {"query":"all: \"JPEG Processing Neural Operator\" OR id:(Han et al., 31 Jul 2025) OR id:(Qi et al., 2023) OR id:(Reich et al., 2023) OR id:(Santos et al., 2021) OR id:(Deguerre et al., 2019) OR id:(Liu et al., 2018) OR id:(Guo et al., 2023)","max_results":10,"sort_by":"relevance"} to=arxiv_search.search 天天中彩票qqjson {"query":"all:\"JPEG Processing Neural Operator\" (Han et al., 31 Jul 2025, Qi et al., 2023, Reich et al., 2023, Deguerre et al., 2019, Santos et al., 2021, Liu et al., 2018, Guo et al., 2023)","max_results":10,"sort_by":"relevance"} to=arxiv_search.search 彩神争霸官方json {"query":"JPEG Processing Neural Operator","max_results":5,"sort_by":"relevance"} JPEG Processing Neural Operator (JPNeO) denotes a family of formulations in which JPEG is treated as a learnable operator rather than as a fixed pre-processing or post-processing routine. In one established usage, JPNeO is the differentiable JPEG module embedded inside an end-to-end image classification pipeline, where quantization tables are optimized jointly with a downstream classifier under bandwidth constraints (Qi et al., 2023). In a later usage, JPNeO names a backward-compatible JPEG system that inserts neural operators before standard encoding and after standard decoding while keeping the emitted bitstream fully compliant with ISO/IEC 10918-1 (Han et al., 31 Jul 2025). Across these formulations, the common idea is to expose JPEG’s color transform, block DCT, quantization, chroma handling, and reconstruction pathway to optimization, approximation, or learned context modeling, thereby aligning a legacy codec with downstream task objectives or reconstruction fidelity.

1. Conceptual scope and uses

JPNeO is not a single architecture. The term covers at least two technically distinct but related conceptions. The first is an embedded differentiable JPEG operator inside a computational graph. In that setting, JPEG is modeled explicitly by level shifting, RGB-to-YCbCr conversion, block-wise $8\times 8$ DCT, quantization and dequantization using trainable tables, and optional IDCT plus chroma upsampling; the operator is then trained jointly with a classifier (Qi et al., 2023). The second is a backward-compatible coding pipeline in which a learned encoder-side operator, learned integer quantization tables, and a decoder-side neural operator improve fidelity without modifying JPEG syntax, markers, or entropy-coding protocol (Han et al., 31 Jul 2025).

A broader operator viewpoint also appears in adjacent compressed-domain literature. JPEG-domain object detection can be described as a block-local operator that maps entropy-decoded DCT blocks into per-block feature vectors before standard spatial convolutions (Deguerre et al., 2019). Image classification from JPEG coefficients can likewise be formalized as a learned operator that aggregates the $8\times 8 \times 3$ coefficient tensor per block and then processes the resulting block grid with a CNN (Santos et al., 2021). Learned lossless recompression extends the same perspective from task inference to entropy modeling: the operator acts directly on quantized DCT coefficients and predicts symbol distributions for arithmetic or ANS-style coding (Guo et al., 2023).

This multiplicity of meanings has produced a common misconception that JPNeO refers only to a single codec or only to a single differentiable layer. The literature instead supports a narrower technical statement: JPNeO is best understood as a reusable design pattern in which JPEG-domain structure becomes an explicit learnable operator, with the exact role of that operator differing across classification, detection, restoration, recompression, and backward-compatible coding.

2. Mathematical structure and differentiable modeling

The differentiable JPNeO formulation in end-to-end learning follows the canonical JPEG pipeline closely. Each 8-bit component is centered by subtracting 128 before DCT. RGB-to-YCbCr conversion uses the standard linear transform

$Y = 0.299R + 0.587G + 0.114B,$

$Cb = -0.168736R - 0.331264G + 0.5B + 128,$

$Cr = 0.5R - 0.418688G - 0.081312B + 128.$

Each channel is partitioned into non-overlapping $8\times 8$ blocks, and the DCT is applied blockwise as

$$X_{u,v} = \frac{1}{4} C_u C_v \sum_{x=0}^{7}\sum_{y=0}^{7} x_{x,y} \cos\!\left(\frac{(2x+1)u\pi}{16}\right) \cos\!\left(\frac{(2y+1)v\pi}{16}\right),$$

with the corresponding IDCT used when spatial reconstruction is required (Qi et al., 2023).

Quantization is parameterized by trainable per-channel “compression kernels” $q_i(j,k)=Q_i(j,k)^{-1}$. For $i\in\{Y,Cb,Cr\}$,

$$\hat{X}^{(i)}_{u,v} = \operatorname{round}\!\big(X^{(i)}_{u,v} q_i(u,v)\big),\qquad \tilde{X}^{(i)}_{u,v} = \hat{X}^{(i)}_{u,v}\frac{1}{q_i(u,v)}.$$

In the 2023 end-to-end classification system, this yields $8\times 8 \times 3$0 trainable JPEG parameters, shared globally across all images and learned per dataset or task rather than per class or per image (Qi et al., 2023).

Because rounding is non-differentiable, differentiable JPEG work uses surrogates. The end-to-end JPNeO adopts a straight-through estimator in which the forward pass uses $8\times 8 \times 3$1 and the backward pass treats the operation as identity, so gradients pass through unchanged (Qi et al., 2023). A more detailed differentiable JPEG formulation later argued that fidelity depends critically on modeling not only rounding, but also integer arithmetic in quality scaling, quantization-table floor operations, and output clamping. It uses the cubic surrogate

$8\times 8 \times 3$2

for rounding, a shifted cubic approximation for floor, and differentiable clipping with slope $8\times 8 \times 3$3 outside the valid range, while remaining differentiable with respect to the input image, the JPEG quality, the quantization tables, and the color conversion parameters (Reich et al., 2023).

A recurrent modeling choice is to omit zigzag scan, run-length encoding, and Huffman coding during training. In the end-to-end JPNeO for classification, no explicit entropy model is used in the loss; instead, regularization on $8\times 8 \times 3$4 is used as a bitrate surrogate, while actual bandwidth is measured post hoc with RLE plus Huffman coding and customized Huffman tables generated from 50k training images (Qi et al., 2023). This omission is methodologically central rather than incidental: it preserves differentiability, but it also creates a gap between the training objective and realized bitstream rate.

3. End-to-end task-aware optimization

The clearest instantiation of JPNeO as a trainable JPEG module appears in the end-to-end optimization of JPEG for image classification. A sensing device produces JPEG-compressed images using fixed 4:2:0 chroma subsampling, and the embedded JPNeO is trained jointly with the classifier to optimize the rate–accuracy trade-off. For CIFAR-100 and Tiny ImageNet, JPNeO includes IDCT and bilinear chroma upsampling and feeds reconstructed spatial YCbCr images into WRN-28. For ImageNet, the operator exports quantized DCT coefficients, upsamples Cb and Cr coefficient maps via learnable transposed convolutions, concatenates them with Y, and feeds the resulting representation directly to a Deconvolution-RFA model without IDCT (Qi et al., 2023).

The training objective is Lagrangian: $8\times 8 \times 3$5 For CIFAR-100 and Tiny ImageNet, $8\times 8 \times 3$6 combines a hinge term that pushes $8\times 8 \times 3$7 below $8\times 8 \times 3$8 with an $8\times 8 \times 3$9 term that promotes sparsity; for ImageNet, it reduces to an $Y = 0.299R + 0.587G + 0.114B,$0 penalty on $Y = 0.299R + 0.587G + 0.114B,$1. There is no explicit constraint of the form $Y = 0.299R + 0.587G + 0.114B,$2 during training. Instead, $Y = 0.299R + 0.587G + 0.114B,$3, $Y = 0.299R + 0.587G + 0.114B,$4, and $Y = 0.299R + 0.587G + 0.114B,$5 are tuned offline to reach target bandwidth regimes, and bandwidth budgets in KB per image are measured after training via actual RLE plus Huffman coding (Qi et al., 2023).

Empirically, the method improves classification performance at low to moderate bandwidth. On CIFAR-100 with WRN-28, JPNeO improves accuracy by up to 2.4 percentage points over baseline JPEG at bandwidths between 0.75 and 1.5 KB/image, while PSNR and SSIM remain similar to baseline JPEG at matched bandwidths. On Tiny ImageNet, the gain is up to 4 percentage points at 0.9 to 1.6 KB/image, again with comparable PSNR and SSIM. On ImageNet with Deconvolution-RFA, JPNeO improves top-5 accuracy by up to 3.7 percentage points for rates below approximately 11 KB/image; above 11 KB/image, baseline JPEG and JPNeO perform similarly, while JPNeO yields higher PSNR and SSIM in that regime (Qi et al., 2023).

The behavior of the learned tables is consistent with earlier evidence that task-aware quantization differs from human-visual quantization. DeepN-JPEG had already shown that standard JPEG, being optimized for the Human Visual System, can be suboptimal for DNN pipelines at high compression ratios, and that a DNN-aware redesign of quantization based on dataset-level DCT statistics can achieve approximately $Y = 0.299R + 0.587G + 0.114B,$6 higher compression over baseline JPEG QF=100 with no measurable top-1 accuracy degradation across AlexNet, GoogLeNet, VGG-16, ResNet-34, and ResNet-50 on ImageNet (Liu et al., 2018). JPNeO generalizes that task-aware intuition into a differentiable, jointly optimized operator.

4. Compressed-domain inference and recompression variants

Related work has extended the JPNeO viewpoint to pipelines that bypass full pixel reconstruction. In JPEG-domain object detection, entropy-decoded DCT coefficients are arranged into a “DCT image,” and a single JPEG-aware convolution with kernel $Y = 0.299R + 0.587G + 0.114B,$7, stride 8, and 192 output channels replaces the first three VGG16 blocks in an SSD300-style detector. The output enters VGG16 block 4, so each $Y = 0.299R + 0.587G + 0.114B,$8 JPEG block is mapped to one spatial location in the CNN feature map. On a GTX 1060 6GB, this SSD$Y = 0.299R + 0.587G + 0.114B,$9 variant achieves 111.3 FPS versus 54.3 FPS for the baseline SSD, with 14 GO versus 31 GO, corresponding to a $Cb = -0.168736R - 0.331264G + 0.5B + 128,$0 speedup and a $Cb = -0.168736R - 0.331264G + 0.5B + 128,$1 FLOPs reduction. Accuracy is lower than RGB SSD, particularly for small objects, but the results establish compressed-domain detection as a practical operator design rather than merely an efficiency trick (Deguerre et al., 2019).

Image classification directly from JPEG coefficients has been pushed further by learning how to fuse all 192 DCT inputs rather than discarding high-frequency coefficients manually. Three channel-reduction mechanisms were evaluated for the $Cb = -0.168736R - 0.331264G + 0.5B + 128,$2 projection before the second stage of a modified ResNet-50: linear projection, local attention, and cross-channel parametric pooling via $Cb = -0.168736R - 0.331264G + 0.5B + 128,$3 convolutions plus ReLU. On ImageNet, DCT plus LA reaches 69.96% top-1, DCT plus LP 69.62%, and DCT plus CCPP 69.73%, while DCT plus CCPP with the first and second ResNet stages skipped reaches 70.49% top-1 at 2.86 GFLOPs and 25.1M parameters. This establishes a compressed-domain operator that reduces compute relative to pixel-domain ResNet-50 while retaining competitive accuracy (Santos et al., 2021).

Lossless JPEG recompression provides a different JPNeO-style endpoint. Efficient learned lossless JPEG recompression models quantized DCT coefficients directly and replaces Huffman coding with learned conditional probability estimation plus entropy coding. Its ML-PCM architecture uses separate Y-Net and CbCr-Net branches, PPCM for luma, and CCCM for chroma, enabling component-level and intra-component parallelism. At QP 75, the reported savings are 29.96% on Kodak 4:2:0 and 31.72% on Kodak 4:4:4, with software-optimized throughput of 57 FPS encode and 57 FPS decode for 1080p images on an NVIDIA T4 GPU after optimization. This variant does not approximate JPEG for backpropagation or task inference; instead, it treats JPEG coefficients as the state space of a learned entropy operator (Guo et al., 2023).

These lines of work suggest that JPNeO can denote three separable operator roles: a differentiable surrogate of JPEG for end-to-end optimization, a compressed-domain feature ingress operator for task networks, and a learned probabilistic operator over JPEG coefficient streams for recompression.

5. Backward-compatible coding architecture

The 2025 formulation of JPNeO defines a next-generation JPEG algorithm that preserves full backward compatibility with the current JPEG format. The system has three components: JENO, a JPEG Encoding Neural Operator $Cb = -0.168736R - 0.331264G + 0.5B + 128,$4 applied before standard encoding; $Cb = -0.168736R - 0.331264G + 0.5B + 128,$5, learned integer quantization tables stored as conventional DQT markers; and JDNO, a JPEG Decoding Neural Operator $Cb = -0.168736R - 0.331264G + 0.5B + 128,$6 applied after standard decoding. The emitted bitstream remains a valid baseline JPEG stream with conventional markers, standard Huffman tables, integer quantized coefficients, and standard-compliant DQT tables in $Cb = -0.168736R - 0.331264G + 0.5B + 128,$7; no custom markers or auxiliary side information are required (Han et al., 31 Jul 2025).

JENO is a residual pre-transform in the pixel domain. It uses an EDSR-baseline feature extractor with 16 residual blocks and 64 channels, a sampling operator $Cb = -0.168736R - 0.331264G + 0.5B + 128,$8 with four-neighbor coordinate augmentation and anti-aliasing factor $Cb = -0.168736R - 0.331264G + 0.5B + 128,$9, and a Galerkin-attention operator $Cr = 0.5R - 0.418688G - 0.081312B + 128.$0 with 256 channels, 16 heads, and 2 iterations. Its output is

$Cr = 0.5R - 0.418688G - 0.081312B + 128.$1

and it is trained so that the neural operator behaves as a high-pass predictor complementary to the low-pass path induced by chroma subsampling. Learned quantization tables are trained through

$Cr = 0.5R - 0.418688G - 0.081312B + 128.$2

with a cubic approximation to rounding during optimization (Han et al., 31 Jul 2025).

JDNO operates after decoding and is explicitly quantization-aware. It uses a group embedding built from de-quantized luma and chroma spectra, a SwinV2-based feature extractor with 256 channels, a cosine neural operator $Cr = 0.5R - 0.418688G - 0.081312B + 128.$3 with $Cr = 0.5R - 0.418688G - 0.081312B + 128.$4, and the same Galerkin-attention operator family as JENO. The cosine operator is conditioned jointly on local features and the quantization table, enabling continuous, $Cr = 0.5R - 0.418688G - 0.081312B + 128.$5-aware reconstruction of high-frequency structure and color transitions. JDNO supports 4:4:4, 4:2:2, and 4:2:0 through explicit coordinate conditioning and the upsampling kernel embedded in the group representation (Han et al., 31 Jul 2025).

The reported gains are strongest in chroma restoration and very low-rate reconstruction. On LIVE-1 at $Cr = 0.5R - 0.418688G - 0.081312B + 128.$6, JPNeO achieves 23.15 PSNR, 22.64 PSNR-B, and 0.631 SSIM, compared with 21.70, 21.19, and 0.577 for FBCNN. At the same operating point, detailed chroma metrics on LIVE-1 are 32.30 dB chrominance PSNR and 0.125 chrominance SSIM for JPNeO, versus 29.86 dB and 0.068 for FBCNN. At $Cr = 0.5R - 0.418688G - 0.081312B + 128.$7, JPNeO reports 32.83 PSNR, 31.91 PSNR-B, and 0.929 SSIM on LIVE-1. The lightweight JPNeO− uses 8.0M parameters, 0.09 GB, and 222.95 ms for a 560×560 input; the full JPNeO uses 29.7M parameters, 0.26 GB, and 562.42 ms, while still requiring less memory than JDEC and fewer parameters than both FBCNN and JDEC (Han et al., 31 Jul 2025).

A second misconception is that “backward compatibility” implies neural operators inside the bitstream syntax. The technical claim is narrower: compatibility is preserved because neural processing occurs only before standard JPEG symbol generation and after standard JPEG decoding, while the middle of the pipeline remains strictly standard (Han et al., 31 Jul 2025).

6. Limitations, misconceptions, and directions

Several limitations recur across JPNeO formulations. In end-to-end classification, entropy coding is not modeled during training; bitrate is driven by surrogate penalties on trainable quantization parameters, and actual bandwidth is evaluated only after post hoc RLE and Huffman coding. The chroma subsampling ratio is fixed at 4:2:0, global quantization tables are shared across the dataset, and robustness may degrade under content shifts because the learned tables are tuned to the training distribution (Qi et al., 2023). In backward-compatible coding, JDNO improves extremely low-rate reconstruction but remains bounded by information already lost in quantized spectra, and gains vary with rate: JDNO contributes more at low bpp, whereas JENO contributes more at high bpp (Han et al., 31 Jul 2025).

Differentiable JPEG itself also has unresolved boundaries. Even detailed surrogate formulations omit the lossless syntax stage, so differentiability does not by itself provide file-size supervision. The strongest forward fidelity was obtained only after modeling floor, clamping, and quality scaling explicitly; earlier simplifications produced large forward errors at low quality. Strong adversarial results demonstrate effective gradients, but they do not eliminate the modeling gap between differentiable approximations and reference encoders (Reich et al., 2023).

A third misconception is that JPEG-domain processing necessarily sacrifices fidelity or task performance. The evidence is more specific. Compressed-domain detection can be approximately $Cr = 0.5R - 0.418688G - 0.081312B + 128.$8 faster while losing accuracy, especially on small objects (Deguerre et al., 2019). DCT-domain classification can reduce compute while retaining competitive ImageNet accuracy through learned channel fusion rather than manual coefficient dropping (Santos et al., 2021). Task-aware quantization can improve accuracy at matched bandwidth for classification (Qi et al., 2023), while backward-compatible JPNeO can improve reconstruction fidelity and chroma preservation without changing the source-coding protocol (Han et al., 31 Jul 2025). These are distinct operating points, not a single Pareto guarantee.

The literature identifies several concrete future directions. For task-aware JPNeO, the main proposals are differentiable entropy models or explicit bitrate predictors, adaptive per-image or per-class quantization tables, extension to detection and segmentation, and broader sensor–codec–task co-design (Qi et al., 2023). For backward-compatible coding, promising directions include interoperability with JPEG XL, learned adaptive quantization, and deployment modes tuned for embedded ISPs and browsers (Han et al., 31 Jul 2025). For recompression, further reduction of CPU-side entropy-coder overhead remains an engineering target (Guo et al., 2023). Taken together, these directions suggest that JPNeO is evolving less as a single model family than as a unifying formalism for making JPEG trainable, task-aware, and structurally interoperable with neural computation.