CompressAI-Vision Benchmark
- CompressAI-Vision is an evaluation platform that benchmarks machine-centric visual compression methods using remote and split inference pipelines.
- It employs a modular, YAML-driven architecture to integrate codecs, datasets, and vision models for comprehensive rate-accuracy analysis.
- Empirical results demonstrate significant bitrate savings at comparable task performance, supporting MPEG’s FCM standardization efforts.
CompressAI-Vision is an open-source evaluation platform for studying compression methods in neural-network-based computer vision pipelines, with a particular focus on how compression affects downstream task accuracy rather than only reconstructed visual quality. It is presented as a common software framework for benchmarking coding tools across multiple computer vision tasks, datasets, and inference deployments, especially in settings where compressed data is consumed by machines rather than humans. The platform is implemented in Python, is publicly available at https://github.com/InterDigitalInc/CompressAI-Vision, and has been adopted by MPEG for the development of the Feature Coding for Machines (FCM) standard (Choi et al., 25 Sep 2025).
1. Conceptual position
CompressAI-Vision is situated within the broader shift from human-vision-oriented compression toward machine-oriented evaluation. Its motivating premise is that conventional image and video compression has historically been optimized for human visual perception, whereas neural-network-based pipelines often require preservation of task-relevant information rather than visually pleasing reconstruction. The platform therefore treats bitrate versus task accuracy as the primary evaluation axis, rather than PSNR-centered rate-distortion alone (Choi et al., 25 Sep 2025).
This emphasis is consistent with earlier software and methodological developments around learned compression. CompressAI introduced a PyTorch-native ecosystem for end-to-end learned compression research, including analysis and synthesis transforms, quantization operations, entropy models, pretrained checkpoints, and benchmarking utilities, while also explicitly noting that learned compression could optimize “machine-tasks related metrics” and support pipelines such as the compression of deep features (Bégaint et al., 2020). CompressAI-Vision extends that general compression substrate into a benchmark layer organized around downstream vision workloads rather than only reconstructed-image fidelity.
A plausible implication is that CompressAI-Vision functions as infrastructural support for “compression for machines”: it does not define a single codec, but rather standardizes how heterogeneous coding tools, datasets, task models, and deployment scenarios are compared.
2. Inference scenarios and compressed entities
The platform supports three inference scenarios: local inference, remote inference, and split inference. Local inference serves as a no-compression benchmark with default task accuracy. The two compression-focused scenarios are remote inference and split inference (Choi et al., 25 Sep 2025).
In remote inference, the raw visual input is compressed and transmitted to a remote server, where the bitstream is decoded and the full neural network runs on reconstructed pixels. What is compressed here is pixel-domain visual data. This matches common edge-to-cloud deployments and permits straightforward interfacing with standard codecs such as AVC, HEVC, VVC, and FFmpeg-supported codecs.
In split inference, the neural network is partitioned into an on-device front end and a server-side back end. The compressed object is no longer pixels but intermediate feature tensors. The paper represents such tensors as
where is the number of channels and is the spatial resolution of each channel. By default, CompressAI-Vision reshapes a 3D feature tensor into a 2D frame by tiling feature channels in raster-scan order, allowing existing video codecs to be reused for feature coding experiments (Choi et al., 25 Sep 2025).
That distinction between pixel-domain remote inference and feature-domain split inference is central. It means that CompressAI-Vision is not restricted to one representation level; it evaluates compression both before neural processing and at explicit split points inside a model.
3. Software architecture and supported model inventory
The platform is modular and configuration-driven. A YAML configuration selects the inference pipeline, codec, neural vision model, evaluator, and dataset. The software includes dataset loaders, preprocessing logic, model wrappers, codec interfaces, split-point handling, and task-specific evaluators (Choi et al., 25 Sep 2025).
Dataset loaders support both image and video datasets, including COCO and VOC-like annotation formats. For video input, videos can be split into individual frame files when models operate frame by frame. The preprocessing pipeline may also include color-space conversion, for example from YUV420 to RGB, since many computer vision networks expect RGB input. Evaluators compute task-specific metrics such as mAP and MOTA.
The currently described model support spans Detectron2, JDE, YOLOX, and MMPose, with predefined split tags for split inference.
| Model family | Task | Split tags |
|---|---|---|
Detectron2 (faster_rcnn_*, mask_rcnn_*) |
Object detection, instance segmentation | "r2"(1), "c2"(1), "fpn"(4) |
JDE (jde_1088x608) |
Multiple people tracking | "dn53"(3), "alt1"(3) |
YOLOX (yolox_darknet53) |
Object detection | "l13"(1), "l37"(1) |
MMPose (rtmo_multi_person_pose_estimation) |
Pose estimation | "backbone"(2), "neck"(2) |
These split tags are not merely labels; they define the points at which intermediate tensors are exposed for compression. In R-CNN architectures, for example, the "fpn" split produces four multi-scale tensors. This makes the platform suitable for comparing codecs that operate on very different internal representations while keeping the downstream task pipeline fixed (Choi et al., 25 Sep 2025).
4. Evaluation methodology and benchmark datasets
CompressAI-Vision evaluates coding methods through rate-accuracy benchmarking. The main reported performance axes are bit-rate and task accuracy, typically visualized as Rate-Accuracy (RA) curves. The paper also reports BD-Bitrate differences at equal task accuracy, although it does not provide explicit formulas for BD-Bitrate, mAP, or MOTA (Choi et al., 25 Sep 2025).
The benchmark datasets summarized in the paper span both image and video domains and are linked to MPEG’s FCM and VCM test conditions.
| Dataset | Data type | Task |
|---|---|---|
| Open Images | Image | Instance segmentation, object detection |
| FLIR | Image | Object detection |
| SFU-HW-Obj | Video | Object detection |
| TVD | Video | Object tracking |
| HiEVE | Video | Object tracking |
The comparative protocol borrows coding conditions from MPEG common test conditions. From VCM CTC, the paper uses random access, low delay, and all-intra with end-to-end configuration; from FCM CTTC, it uses low delay. Quantization parameter is adjusted to ensure comparable task accuracy when comparing FCTM and VCM-RS under VCM conditions (Choi et al., 25 Sep 2025).
This evaluation design is significant because it places remote inference, split inference, standard codecs, and under-development machine-oriented codecs inside one common experimental frame. A plausible implication is that CompressAI-Vision is less a codec benchmark in the narrow sense than a benchmark of end-to-end compressed inference systems.
5. Empirical use cases and standardization role
The paper’s main experiments demonstrate how the platform is used to compare MPEG machine-oriented coding tools. A central comparison is between FCTM v6.1 and VCM-RS v0.12 under VCM CTC conditions (Choi et al., 25 Sep 2025).
For object detection on SFU-HW-Obj, FCTM saved 79.35% bits for Class C and 69.02% for Class D compared to VCM-RS at the same task accuracy. Average BD-Bitrate differences for FCTM versus VCM-RS were -58.33% for RA (e2e), -41.43% for LD (e2e), and -72.70% for AI (e2e). For object tracking on TVD, the outcome differed: FCTM increased bitrate relative to VCM-RS by 26.63% under RA and 18.69% under LD. The paper attributes much of this degradation to poor performance on sequence TVD-01.
The platform is also used to compare different inner codecs inside FCTM. Using VTM as reference, bitrate penalties for HM and JM are reported as 4.90% and 39.40% on SFU-HW-Obj Class AB, 12.86% and 53.28% on Class C, and 12.42% and 38.55% on Class D. The paper concludes that VTM is better as an inner codec, while also noting that machine-task rate-accuracy comparisons can yield smaller gaps than conventional bitrate-versus-PSNR comparisons (Choi et al., 25 Sep 2025).
These use cases explain why MPEG adopted CompressAI-Vision for FCM standardization. The software is not only a research toolkit; it functions as a common evaluation environment for comparing under-development coding tools under shared test conditions.
6. Relation to adjacent machine-centric compression research
CompressAI-Vision belongs to a broader research program in which compression is judged by machine utility. One influential direction formulates learned image compression directly with a rate–distortion–utility objective and allows downstream models to consume the compressed latent representation rather than reconstructed RGB. “Learned Image Compression for Machine Perception” defines a rate–distortion–utility formulation and trains task heads directly on compressed latents, showing that semantically informed compressed representations can improve detection and segmentation relative to JPEG at substantially lower bitrate (Codevilla et al., 2021).
A second direction preserves standard codec deployment while learning a task-aware frontend. “Preprocessing Enhanced Image Compression for Machine Vision” inserts a neural preprocessing module before BPG and optimizes it with downstream task loss using a proxy codec for backpropagation, reporting about 20% bitrate savings at equal task accuracy on detection and classification tasks (Lu et al., 2022). A related microscopy study evaluates pretrained CompressAI codecs not only by reconstruction metrics but also by downstream fluorescence-prediction robustness, showing that learned codecs preserve task-relevant structure much better than classical lossy codecs in 2D microscopy workloads (Zhou et al., 2023).
These lines of work clarify the design space that CompressAI-Vision is meant to evaluate. Some methods modify the codec itself; others modify the signal before coding; others operate on compressed latents or intermediate features. This suggests that a platform centered on remote and split inferencing, standardized datasets, and RA curves is necessary precisely because “compression for machines” now spans pixel-domain coding, feature coding, and hybrid task-aware front ends.
7. Current limitations and planned extensions
The platform is explicitly described as open-source and extensible, but several limitations are also identified. It does not include datasets; users must prepare them separately. Supported tasks and models are broad but not exhaustive. Direct comparison across remote and split inference remains conceptually difficult because the compressed entities and optimization targets differ. Existing standard codecs are still suboptimal for machine vision because they were designed for human perception, and split inference requires explicit tensor reshaping and split-point selection that may affect coding efficiency (Choi et al., 25 Sep 2025).
The paper also identifies future directions. It specifically proposes support for vision transformers, in order to study how compression noise affects embedding spaces and downstream tasks, and support for multi-task networks, to better understand compression under simultaneous task demands. Those extensions are consistent with the platform’s existing modular organization around models, split points, codecs, evaluators, and YAML-driven experiment configuration.
In that sense, CompressAI-Vision is best understood as a benchmark layer for machine-oriented visual compression rather than as a single algorithmic contribution. Its distinctive technical role is to provide a shared environment in which remote pixel coding, split feature coding, standard codecs, and task-aware coding tools can be compared under common rate-accuracy criteria (Choi et al., 25 Sep 2025).