ImageNeXt: Next-Generation Multi-Modal Image Systems

Updated 12 July 2026

ImageNeXt is a multi-modal image framework that extends ImageNet with additional modalities and supports next-generation image modeling.
It improves visual tasks by fusing RGB and supplementary modalities (Depth, Event, LiDAR, Thermal) through targeted supervised pretraining and modular designs.
It also represents an encoder-centric pattern used in architectures like ConvNeXt and ResNeXt, powering segmentation, view synthesis, and neural-field based image generation.

ImageNeXt is a non-standard term that appears in recent vision literature in several related senses. In its most explicit usage, OmniSegmentor defines ImageNeXt as a large-scale multi-modal extension of ImageNet-1K constructed for supervised pretraining over five visual modalities: RGB, Depth, Event, LiDAR, and Thermal (Yin et al., 18 Sep 2025). In parallel, other papers use “ImageNeXt,” “ImageNeXt-style,” or “ImageNeXt framing” more loosely to denote modern image systems built around ConvNeXt or ResNeXt backbones, vision foundation models, neural basis expansion, neural fields, or pixel-space generative modeling (Florindo et al., 21 May 2026). This suggests that the term functions both as a formal dataset name and as a broader label for next-generation image modeling pipelines.

1. Terminological scope

The literature represented here does not treat ImageNeXt as a single universally standardized object. Rather, it spans a named dataset, a family resemblance among encoder–decoder systems, and a broader “next-generation image” framing across synthesis, compression, and generation.

Usage	Representative paper	Core object
Formal dataset name	OmniSegmentor (Yin et al., 18 Sep 2025)	Multi-modal ImageNet-1K extension
Architectural shorthand	ConvNeXt-FD (Florindo et al., 21 May 2026)	ConvNeXt “ImageNeXt-style” backbone
Conceptual framing	NeX (Wizadwongsa et al., 2021)	Next-generation view synthesis
Foundation-guided translation	Pix2Next (Jin et al., 2024)	VFM-augmented RGB→NIR translation
Neural-field image modeling	ANI (Hoshikawa et al., 2024)	Adaptive neural images
Pixel-space generation	PixNerd (Wang et al., 31 Jul 2025)	Patch-wise neural field diffusion
ResNeXt lineage	MixedSN (Nyasaka et al., 2020)	Broader ResNeXt-style image paradigm

A common misconception is to read ImageNeXt as a single benchmark or model family. The papers instead support a narrower factual statement: only OmniSegmentor uses ImageNeXt as a formal dataset name, whereas several other works invoke it as a framing for modern image architectures or tasks (Yin et al., 18 Sep 2025).

In OmniSegmentor, ImageNeXt is constructed by augmenting every RGB image in ImageNet-1K. The resulting dataset preserves the 1000 categories and the standard train/val partition, with 1.2M training images and 50K validation images, while attaching four additional modalities to each sample: Depth, Event, LiDAR, and Thermal (Yin et al., 18 Sep 2025).

The construction pipeline is modality-specific. Depth is estimated for every ImageNet image using Omnidata; Event is taken from N-ImageNet; LiDAR is generated through pseudo-LiDAR from synthetic depth and represented in range-view image format; Thermal is generated by a thermal estimation network trained on VT821, VT1000, VT5000, and FLIR ADAS dataset (Yin et al., 18 Sep 2025). The dataset is therefore large-scale and modality-diverse, but it also inherits an important limitation: three of the supplementary modalities are synthetic or derived rather than directly sensor-acquired.

The associated pretraining strategy is deliberately not “all modalities at once.” During supervised classification pretraining, the model always receives RGB plus one randomly selected supplementary modality. The loss is standard ImageNet cross-entropy,

$\mathcal{L}_{\text{pretrain}} = - \sum_{c=1}^{1000} y_c \log p_c,$

and the multi-modal encoder is based on DFormer. During finetuning, the shared supplementary branch is duplicated into modality-specific stems and MLPs, enabling arbitrary combinations such as RGB-D, RGB-E, RGB-L, RGB-D-E, or RGB-D-E-L (Yin et al., 18 Sep 2025).

Empirically, the paper reports that this ImageNeXt pretraining improves downstream semantic segmentation across NYU Depthv2, SUNRGBD, MFNet, KITTI-360, EventScape, and DeLiVER. Representative results include 57.6% mIoU on NYU Depthv2 with OmniSegmentor + DFormer-L, 60.6% mIoU on MFNet, 69.2% mIoU on KITTI-360, and 68.0% mIoU on DeLiVER with RGB-D-E-L (Yin et al., 18 Sep 2025). A further ablation shows that RGB + random one modality pretraining outperforms both RGB-only pretraining and simultaneous all-modality pretraining, while reducing FLOPs relative to the latter.

3. ImageNeXt as an encoder-centric architectural pattern

Outside OmniSegmentor, ImageNeXt most often denotes a design pattern: a strong modern encoder, usually pre-trained, combined with a comparatively simple task head and task-specific supervision. ConvNeXt-FD is explicit about this by describing its encoder as a ConvNeXt “ImageNeXt-style” backbone inside a U-Net-like segmentation model (Florindo et al., 21 May 2026).

ConvNeXt-FD uses tu-convnext_tiny, tu-convnext_base, and tu-convnext_large encoders, with tu-convnext_large with ImageNet pre-training as the best-performing configuration on most datasets. A ConvNeXt block is described through large-kernel depthwise convolution, LayerNorm, pointwise convolution, GELU, a second pointwise convolution, and a residual connection. The encoder produces hierarchical features $\{E_1,E_2,E_3,E_4,E_5\}$ , and the decoder is conventional U-Net-style with skip connections. The distinctive addition is an auxiliary Fractal Dimension head and a hybrid loss

$\mathcal{L} = \mathcal{L}_{seg} + \lambda_{fd}\mathcal{L}_{fd},$

where $\mathcal{L}_{seg}$ combines BCE + Dice and $\mathcal{L}_{fd}$ is an MSE loss on a predicted FD map (Florindo et al., 21 May 2026).

The reported gains from pretraining are large. For example, Dice improves from 0.6418 to 0.8040 on BUSI, from 0.6330 to 0.8013 on DDTI, and from 0.9015 to 0.9619 on IDRiD when moving from scratch training to ImageNet pre-training (Florindo et al., 21 May 2026). Best configurations include 0.8040 ± 0.0324 Dice on BUSI, 0.9619 ± 0.0000 Dice on IDRiD, and 0.8223 ± 0.0000 Dice on MoNuSeg.

A closely related pattern appears in tampering localization, where a ConvNeXt-B encoder is paired with top-down feature pyramid fusion and a Pyramid Pooling Module. The decoder fuses $X_1,X_2,X_3,X_4$ through lateral $1\times1$ convolutions, upsampling, and $3\times3$ smoothing. With combined loss $L_{\text{Focal}} + \lambda L_{\text{Lovasz}}$ , ImageNet-22K pretrained weights, and heavy augmentation, the method reaches average F1 52.1% and IoU 47.7% across Columbia, CASIAv1, NIST, DSO, and IMD, outperforming OSN in average F1 and IoU (Zhu et al., 2022).

Pix2Next generalizes the same encoder-centric logic to spectral translation. It uses a pre-trained InternImage foundation model as a feature extractor, injects the final InternImage representation into a U-Net–style encoder–decoder by cross-attention at encoder, bottleneck, and decoder stages, and supervises generation with GAN, feature matching, and SSIM losses (Jin et al., 2024). On the RANUS dataset, Pix2Next reports FID 28.01, SSIM 0.8031, PSNR 20.83, LPIPS 0.107, and DISTS 0.1252, improving the FID score by 34.81% relative to the best baseline CycleGAN at 42.97. This supports a broader interpretation of ImageNeXt as a regime in which global pre-trained representations are fused with local decoding machinery rather than learned from scratch.

4. ImageNeXt as next-generation view synthesis

In the context of neural rendering, NeX is presented as an ImageNeXt-style approach in the sense of next-generation image / view synthesis. NeX starts from the classic multiplane image (MPI) representation but replaces constant per-pixel RGB with a view-dependent function expressed as a linear combination of learned basis functions over viewing direction,

$\mathcal{C}^{\mathbf{p}}(\mathbf{v}) = k_0^{\mathbf{p}} + \sum_{n=1}^{N} k_n^{\mathbf{p}} H_n(\mathbf{v}),$

thereby introducing view-dependent effects while preserving the explicit, homography-based MPI rendering structure (Wizadwongsa et al., 2021).

The method is hybrid implicit–explicit. The base color coefficient $\{E_1,E_2,E_3,E_4,E_5\}$ 0 is stored explicitly per pixel, whereas alpha and the view-dependent coefficients $\{E_1,E_2,E_3,E_4,E_5\}$ 1 are predicted by a per-pixel MLP $\{E_1,E_2,E_3,E_4,E_5\}$ 2, and the global basis functions $\{E_1,E_2,E_3,E_4,E_5\}$ 3 are predicted by a second MLP $\{E_1,E_2,E_3,E_4,E_5\}$ 4 conditioned on view direction. At runtime, the heavy neural computation is precomputed into textures, so rendering reduces to texture lookup, homography warping, and alpha compositing (Wizadwongsa et al., 2021).

This design yields a specific quality–efficiency profile. For a typical scene with 17 input photos at 1008×756, NeX reports ~18 hours of training on a single NVIDIA V100, but runtime rendering reaches ~300 FPS in WebGL on an NVIDIA RTX 2080 Ti, corresponding to ~0.16 MFLOPs per pixel. The paper contrasts this with NeRF at ~55 seconds per frame and ~226 MFLOPs per pixel, concluding that NeX is over 1000× faster in rendering while achieving the best overall scores across PSNR, SSIM, and LPIPS on the evaluated forward-facing datasets (Wizadwongsa et al., 2021).

In this strand of work, ImageNeXt does not denote a dataset. It denotes a move beyond Lambertian MPI and beyond expensive implicit-only radiance fields: explicit geometry and raster structure are preserved, while view dependence is introduced through learned basis expansion.

5. ImageNeXt as neural-field compression and generation

A separate cluster of papers uses ImageNeXt to describe neural-field-based image representations and generators. In Extreme Compression of Adaptive Neural Images, the core object is the Adaptive Neural Image (ANI), defined as a once-for-all neural image: a single INR super-network trained for one image but adaptable in depth, width, and quantization configuration (Hoshikawa et al., 2024). The search space contains $\{E_1,E_2,E_3,E_4,E_5\}$ 5 and $\{E_1,E_2,E_3,E_4,E_5\}$ 6, producing 16 subnet combinations. The paper adopts LSQ+ quantization-aware training and reports that successful 4-bit neural representations reduce bits-per-pixel by roughly 4× relative to full precision while preserving much of the signal. For instance, a SIREN model moves from 1.812 bpp and 27.98 dB PSNR in full precision to 0.257 bpp and 25.35 dB PSNR with QAT 4-bit, whereas naive PTQ 4-bit collapses to 18.24 dB (Hoshikawa et al., 2024).

PixNerd extends the neural-field logic into large-scale generative modeling. It is a single-scale, single-stage, end-to-end pixel-space diffusion framework that removes the VAE bottleneck and avoids a cascade pipeline by replacing the usual patch decoder with a patch-wise neural field. Each transformer patch token predicts the weights of a local MLP, and the MLP maps coordinate encoding and noisy pixel value to a velocity prediction (Wang et al., 31 Jul 2025). The paper reports 2.15 FID on ImageNet $\{E_1,E_2,E_3,E_4,E_5\}$ 7 and 2.84 FID on ImageNet $\{E_1,E_2,E_3,E_4,E_5\}$ 8 without any complex cascade pipeline or VAE. For text-to-image, PixNerd-XXL/16 reaches 0.73 overall on GenEval and 80.9 overall on the DPG benchmark (Wang et al., 31 Jul 2025).

These two works share a common representational move: the image is no longer treated only as a fixed discrete tensor. It becomes a coordinate-conditioned neural object whose rate, resolution, or decoding detail can be controlled by architecture, quantization, or per-patch neural fields. A plausible implication is that ImageNeXt, in this usage, denotes a transition from pixel grids to adaptive neural image parameterizations.

6. Lineage, limitations, and standardization

The term also has a lineage-oriented usage. In hyperspectral classification, MixedSN states that “ImageNeXt” can be understood as a broader paradigm of applying ResNeXt-style networks to general image feature extraction and classification (Nyasaka et al., 2020). MixedSN is a 3D–2D hybrid ResNeXt network: early 3D convolutions and 3D ResNeXt blocks model spectral–spatial information, then the architecture transitions to 2D ResNeXt blocks for higher-level semantic abstraction. On Indian Pines, Pavia University, and Salinas, it reaches 99.75, 99.97, and 100.0 overall accuracy respectively while using 332,864 parameters, compared with 5,122,176 for HybridSN on Indian Pines (Nyasaka et al., 2020). This usage anchors ImageNeXt in the ResNeXt-to-ConvNeXt lineage of hierarchical convolutional encoders.

Across the papers, several limitations recur. In OmniSegmentor, Depth, LiDAR, and Thermal are synthetic or derived, and the paper notes a lack of comprehensive real-world datasets containing all five modalities jointly (Yin et al., 18 Sep 2025). ConvNeXt-FD notes the computational overhead of constructing FD maps and the relative heaviness of ConvNeXt-Large, while also restricting the method to binary segmentation (Florindo et al., 21 May 2026). NeX inherits MPI’s “stack of cards” artifacts and struggles with extremely sharp, sparse highlights (Wizadwongsa et al., 2021). ANI reports that ternary quantization gave unsatisfactory results and that full-resolution INR training can be memory-intensive (Hoshikawa et al., 2024). PixNerd states that current models can show unclear details and occasional blurry or unnatural artifacts (Wang et al., 31 Jul 2025).

Taken together, these limitations reinforce the terminological point made at the outset. ImageNeXt is not yet a closed, canonical framework. It is best understood as an umbrella term spanning: a concrete multi-modal ImageNet extension; a set of encoder-centric architectural practices built around ConvNeXt, ResNeXt, and VFMs; and a broader class of next-generation image representations based on neural basis expansion, neural fields, and pixel-native generative modeling.