Image-to-Spectrum Encoding Overview

• Image-to-Spectrum Encoding is a method that transforms visual data into spectral coefficients, enabling compact representations and accurate reconstruction.
• It employs diverse implementations such as Fourier and wavelet transforms, neural implicit representations, and calibrated linear models to optimize performance.
• This approach supports applications from hyperspectral and multispectral imaging to audio and MR imaging, while addressing key trade-offs in noise and resolution.

Searching arXiv for recent and core papers on image-to-spectrum encoding and closely related spectral encoding methods. Image-to-Spectrum Encoding denotes a family of representations and measurement schemes in which image-domain information is converted into a spectral, transform-domain, or spectrum-like code that can later be inverted to recover the original scene, a higher-dimensional field, or a distribution over an internal variable. In the cited literature, this appears in several exact forms: hyperspectral images are modeled as mappings from pixel coordinates to full spectral signatures and compressed by implicit neural representations; single-pixel and fiber systems encode spatial structure into Fourier coefficients or broad spectral codes; computational spectral imagers map $I(x,y,\lambda)$ into one or a few 2D sensor measurements; complex audio spectrograms encode amplitude and phase into color pixels; and magnetic-resonance velocity spectrum imaging encodes a velocity distribution inside each voxel and recovers it by Fourier inversion (Rezasoltani et al., 2023, Bian et al., 2015, Liu et al., 2023, Hernandez-Garcia et al., 27 Aug 2025).

1. Formal scope and mathematical models

A compact way to formalize the topic is as an encoding operator that maps an object, image, or latent field into measurements whose coordinates are spectral coefficients, wavelength responses, transform coefficients, or spectrum-like variables. In the hyperspectral compression setting, the object is explicitly written as a mapping

$I:(p_x,p_y)\rightarrow(ch_0,ch_1,\ldots,ch_n),$

and the encoder learns a coordinate-to-spectrum approximation $I(p_x,p_y)\approx f_\Theta(p_x,p_y)$, with the network parameters acting as the compressed code (Rezasoltani et al., 2023). In computational spectral imaging, a broader physical forward model is

$$I_{rgb}=\int W(\lambda)\cdot P(x,y,\lambda)\ast\big(A(x,y,\lambda)\cdot I(x,y,\lambda)\big)\,d\lambda,$$

which unifies amplitude, phase, and wavelength encoding into a single operator from a 3D spectral cube to a 2D measurement (Liu et al., 2023).

A second canonical form is the linear inverse model

$\mathbf{y}=\mathbf{A}\mathbf{x}+\boldsymbol{\eta},$

where $\mathbf{x}$ is a discretized spectrum or multispectral cube, $\mathbf{y}$ is a measured image or detector output, and $\mathbf{A}$ is a calibrated encoding matrix. This appears in nanowire-mat spectrometers, single-fiber spread-spectrum imaging, and micro-structured diffractive spectral cameras, with reconstruction posed as a regularized inversion of $\mathbf{A}$ (French et al., 2017, Barankov et al., 2015, Wang et al., 2017).

A third form is explicitly transform-domain. Efficient single-pixel imaging treats the unknown scene $f(x,y)$ through its Fourier coefficients $I:(p_x,p_y)\rightarrow(ch_0,ch_1,\ldots,ch_n),$0, and designs sinusoidal probe patterns so that bucket measurements directly sample the scene’s spatial spectrum (Bian et al., 2015). Velocity spectrum imaging uses modified velocity-selective RF pulses to encode a voxel’s internal velocity distribution, yielding

$I:(p_x,p_y)\rightarrow(ch_0,ch_1,\ldots,ch_n),$1

so that the inverse Fourier transform over the encoding variable $I:(p_x,p_y)\rightarrow(ch_0,ch_1,\ldots,ch_n),$2 recovers a velocity spectrum $I:(p_x,p_y)\rightarrow(ch_0,ch_1,\ldots,ch_n),$3 per voxel (Hernandez-Garcia et al., 27 Aug 2025). In the audio domain, complex spectrogram coefficients $I:(p_x,p_y)\rightarrow(ch_0,ch_1,\ldots,ch_n),$4 are encoded into image pixels by mapping phase to hue and amplitude to brightness and saturation, with exact reversal of the mapping recovering the original sound (Wedekind et al., 2019).

Form	Encoding map	Representative papers
Coordinate-to-spectrum	$I:(p_x,p_y)\rightarrow(ch_0,ch_1,\ldots,ch_n),$5	(Rezasoltani et al., 2023, Rezasoltani et al., 2023)
Fourier-domain sampling	$I:(p_x,p_y)\rightarrow(ch_0,ch_1,\ldots,ch_n),$6	(Bian et al., 2015, Ryczkowski et al., 2020)
Linear spectral imaging	$I:(p_x,p_y)\rightarrow(ch_0,ch_1,\ldots,ch_n),$7	(French et al., 2017, Wang et al., 2017, Barankov et al., 2015)
Velocity-spectrum imaging	$I:(p_x,p_y)\rightarrow(ch_0,ch_1,\ldots,ch_n),$8 by Fourier inversion	(Hernandez-Garcia et al., 27 Aug 2025)
Complex spectrogram imaging	$I:(p_x,p_y)\rightarrow(ch_0,ch_1,\ldots,ch_n),$9 color pixel	(Wedekind et al., 2019)
Spectral tokenization	image $I(p_x,p_y)\approx f_\Theta(p_x,p_y)$0 DWT-based discrete tokens	(Esteves et al., 2024)

This suggests that “spectrum” in the literature is not restricted to wavelength alone. It also includes Fourier coefficients, wavelet coefficients, angular spectra, complex phase-amplitude encodings, and voxel-wise distributions over latent variables when the transform relation is explicit.

2. Optical and physical realizations

In physical optics, image-to-spectrum encoding often begins by selecting an orthogonal or approximately orthogonal basis and designing illumination or scattering so that measurements directly correspond to basis coefficients. Efficient single-pixel imaging projects sinusoidal patterns and uses two $I(p_x,p_y)\approx f_\Theta(p_x,p_y)$1-phase-shifted patterns to recover each Fourier coefficient in the most informative spatial frequency bands, exploiting centrosymmetric conjugation and sparsity of natural-image spectra; the method reduces requisite patterns by two orders of magnitude compared to conventional SPI (Bian et al., 2015). Closely related spectral-domain single-pixel imaging replaces spatial masks with spectral Fourier probe patterns imposed by a programmable optical filter, so that an unknown transmission $I(p_x,p_y)\approx f_\Theta(p_x,p_y)$2 is reconstructed from single-pixel measurements as a Fourier series in the spectral domain (Ryczkowski et al., 2020). In scan-less confocal phase microscopy, two-dimensional image pixels are encoded onto optical-frequency-comb modes via a VIPA–grating disperser and then decoded from mode-resolved amplitude and phase spectra using dual-comb spectroscopy (Hase et al., 2017).

A second physical family uses wavelength-dependent codes that are broad, overlapping, and later inverted computationally. In spread-spectrum single-fiber imaging, each spatial pixel is converted into a distinct spectral code spanning the full object bandwidth, the spectrum is transmitted through a single optical fiber, and the image is recovered numerically from the detected spectrum; this scheme is explicitly described as spatial information converted to spectral information, is insensitive to fiber bending, and contains no moving parts (Barankov et al., 2015). A closely related microfabricated spatio-spectral encoder uses a $I(p_x,p_y)\approx f_\Theta(p_x,p_y)$3 array of Fabry–Pérot–metasurface filters so that each spatial pixel is tagged with a narrowband spectral code over 560–625 nm and the original pattern is reconstructed by pseudo-inverse; the recovered image remains unchanged with fiber bending or moving (Xie et al., 2023). Computational multi-spectral video imaging places a micro-structured diffractive filter near the sensor so that spectral information is converted to a spatial code on sensor pixels and the multispectral cube is recovered from a single frame by regularization-based linear algebra (Wang et al., 2017).

Disordered and reconfigurable spectral encoders replace deterministic gratings or filter arrays with complex response matrices. In nanowire-mat hyperspectral imaging, strong diffuse scattering maps different wavelengths to distinct speckle patterns with nanometer sensitivity, and a transmission matrix is calibrated so that spectra are reconstructed from monochrome camera measurements; compressive sensing is especially effective for sparse spectra and noisy conditions (French et al., 2017). Bio-inspired photonic spectral encoders generalize this idea by optimizing orthogonality, completeness, and sparsity of the response matrix under a Bayesian expected-information-gain framework, and experimentally validate a dynamically reconfigurable encoder with 6 pm resolution over a 30 nm measurable bandwidth (Zhang et al., 18 Jan 2026).

3. Learned, implicit, and transform-domain representations

In implicit neural representations for hyperspectral compression, the encoder is no longer a physical optical element but a learned continuous function. A multilayer perceptron with sinusoidal activations learns the mapping from 2D coordinates to a full band vector, and the weights of the network become the compressed representation of the image; decoding is simply evaluation of the learned function at pixel coordinates (Rezasoltani et al., 2023). A later variant adds a sampling strategy based on window size and sampling rate to reduce compression time while preserving or improving PSNR and SSIM, and reports better low-bitrate performance than JPEG, JPEG2000, PCA-DCT, and several learning-based baselines on Indian Pines, Jasper Ridge, Pavia University, and Cuprite (Rezasoltani et al., 2023).

Other learned encoders operate in physically realized spectral spaces. Vortex-encoded spectral correlations use phase masks $I(p_x,p_y)\approx f_\Theta(p_x,p_y)$4 so that the sensor intensity becomes a quadratic polynomial in derivatives of the underlying Fourier field, and a learned linear transformation reconstructs or distills useful image features from the resulting correlation patterns (Perry et al., 2023). Diffractive networks controlled by illumination angular spectrum encode task identity in the angular components of illumination while the object image remains in spatial amplitude, enabling a single diffractive stack to perform multiple image-to-image mappings; the task is determined by the illumination mask, which serves as a unique task encoder (Kleiner et al., 8 Jan 2026). The unified encoding model in computational spectral imaging makes this encoder–decoder viewpoint explicit by jointly optimizing physical amplitude, phase, or wavelength encoders with digital decoders under the same loss (Liu et al., 2023).

A transform-domain digital counterpart appears in the Spectral Image Tokenizer. Here the image is first converted to a multi-level 2D discrete wavelet transform, each scale is patchified and vector-quantized, and the resulting sequence of discrete tokens is ordered from low frequency to high frequency (Esteves et al., 2024). The token sequence is therefore coarse-to-fine rather than raster-ordered, enabling partial decoding, multiscale generation, and text-guided upsampling. This suggests that image-to-spectrum encoding also functions as a sequence-design principle for autoregressive models when the “spectrum” is wavelet rather than optical.

4. Decoding and reconstruction methods

Decoding is usually posed as an inverse problem. In single-fiber spectral encoding and micro-structured spectral filter arrays, the reconstruction is the pseudo-inverse or least-squares solution of the calibrated linear system, often written as $I(p_x,p_y)\approx f_\Theta(p_x,p_y)$5 or $I(p_x,p_y)\approx f_\Theta(p_x,p_y)$6 (Barankov et al., 2015, Xie et al., 2023). In nanowire-mat spectrometers, Tikhonov regularization and $I(p_x,p_y)\approx f_\Theta(p_x,p_y)$7-based compressive sensing are compared directly; compressive sensing suppresses computational background and remains effective in underdetermined and noisy regimes (French et al., 2017). In the bio-inspired photonic encoder, reconstruction is cast as

$I(p_x,p_y)\approx f_\Theta(p_x,p_y)$8

with additional $I(p_x,p_y)\approx f_\Theta(p_x,p_y)$9, $$I_{rgb}=\int W(\lambda)\cdot P(x,y,\lambda)\ast\big(A(x,y,\lambda)\cdot I(x,y,\lambda)\big)\,d\lambda,$$0, and smoothness terms to adapt to sparse or continuous spectra (Zhang et al., 18 Jan 2026). In the unified encoding model, the decoder is a learned network such as Sim-Conv-Net, Res-U-Net, or an unfolding-based model trained jointly with the encoder (Liu et al., 2023).

Some systems admit direct or nearly direct inversion because the encoding is deliberately chosen in an orthogonal basis. Efficient single-pixel imaging reconstructs by populating Fourier coefficients and applying the inverse Fourier transform (Bian et al., 2015). Velocity spectrum imaging reconstructs the voxel-wise distribution by Fourier transforming over the sampled first gradient moments $$I_{rgb}=\int W(\lambda)\cdot P(x,y,\lambda)\ast\big(A(x,y,\lambda)\cdot I(x,y,\lambda)\big)\,d\lambda,$$1 (Hernandez-Garcia et al., 27 Aug 2025). Complex spectrogram color encoding is explicitly bijective: RGB is converted back to HSV, hue yields phase, saturation or brightness yields amplitude according to the appropriate branch, and inverse STFT recovers the waveform down to the original phases (Wedekind et al., 2019). In coordinate-based hyperspectral INRs, the “inverse” is not matrix inversion but function evaluation on the coordinate grid after loading the learned parameters (Rezasoltani et al., 2023).

Evaluation criteria vary with modality but consistently measure either fidelity of the recovered spectrum or utility of the encoded representation. Hyperspectral compression papers use MSE, PSNR, and SSIM (Rezasoltani et al., 2023). Optical spectral imagers use RMSE, cross-correlation, or spectral resolution measured by decorrelation width (Bian et al., 2015, Wang et al., 2017). The unified encoding model compares PSNR, SAM, and ERGAS across amplitude, phase, and wavelength encoders (Liu et al., 2023). Vortex-encoded learning uses MSE, SSIM, classification accuracy, $$I_{rgb}=\int W(\lambda)\cdot P(x,y,\lambda)\ast\big(A(x,y,\lambda)\cdot I(x,y,\lambda)\big)\,d\lambda,$$2, and SAD to characterize reconstruction quality, learnability, and discriminative usefulness of the encoded spectral correlations (Perry et al., 2023). Bio-inspired photonic encoders evaluate condition number, relative residual, mutual coherence, and reconstruction error in relation to orthogonality, completeness, and sparsity (Zhang et al., 18 Jan 2026).

5. Domain-specific applications

In hyperspectral imaging, image-to-spectrum encoding is used primarily for compression and compact representation. Implicit neural representations store the entire data cube as a continuous coordinate-to-spectrum function, achieve better compression than JPEG, JPEG2000, and PCA-DCT at low bitrates, and in the sampling-based variant also improve encoding speed (Rezasoltani et al., 2023, Rezasoltani et al., 2023). In computational spectral cameras, a micro-structured diffractive filter converts a visible or Vis–IR scene into a single coded sensor frame, enabling compact single-shot multispectral video imaging with 9.6 nm spectral resolution over 430–718 nm and software-defined trade-offs between field of view and spectral resolution (Wang et al., 2017).

In sensing and communication systems, the same principle underlies robust transmission through constrained hardware. Single-fiber spread-spectrum imaging converts a 2D self-luminous object into a measured spectrum and is insensitive to fiber bending (Barankov et al., 2015). Image sensor communication structures an optical time-domain waveform so that rolling-shutter images become a recoverable sampled representation of the signal even under variable frame rates, long exposures, and heterogeneous sensors (Nguyen et al., 2016). In audio, the complex-color spectrogram makes amplitude and phase simultaneously visible, allowing humans and machine-vision models to operate on a canonical image representation of total sound while preserving exact reversibility in principle (Wedekind et al., 2019).

In microscopy, tomography, and magnetic resonance, image-to-spectrum encoding exposes latent variables not directly accessible in a conventional image. Optical-frequency-comb microscopy encodes 2D pixels onto comb modes and decodes a scan-less confocal image from the mode-resolved amplitude spectrum and a phase image from the mode-resolved phase spectrum, combining a confocal depth resolution of 62.4 $$I_{rgb}=\int W(\lambda)\cdot P(x,y,\lambda)\ast\big(A(x,y,\lambda)\cdot I(x,y,\lambda)\big)\,d\lambda,$$3m with a phase depth resolution of 13.7 nm (Hase et al., 2017). Velocity spectrum imaging uses modified velocity-selective RF pulses so that each voxel yields a velocity distribution rather than a single mean velocity, with the distribution decoded via Fourier transform and demonstrated on flow phantoms and human participants along three laboratory axes (Hernandez-Garcia et al., 27 Aug 2025). Diffractive networks controlled by illumination angular spectrum use the spectrum itself as a task token, allowing one physical network to perform different image-to-image translations under different illumination masks (Kleiner et al., 8 Jan 2026).

6. Trade-offs, limitations, and design criteria

A recurring design principle is that good encoders maximize diversity in the measurement space while preserving invertibility. The bio-inspired photonic framework formalizes this through orthogonality, completeness, and sparsity: orthogonality is linked to low condition number and low cross-talk, completeness to small residual when representing the target spectral space, and sparsity to low mutual coherence and effective regularized recovery (Zhang et al., 18 Jan 2026). This criterion is consistent with Fourier-domain SPI, transmission-matrix spectrometers, diffractive coded cameras, and joint optical–digital encoders, even when the papers do not use the same vocabulary.

The main limitations are similarly recurrent. Intensity-only measurements discard phase unless special encodings preserve or reconstruct it, as in complex-color audio or dual-comb microscopy (Wedekind et al., 2019, Hase et al., 2017). Poorly conditioned encoders amplify noise and require regularization, as shown in transmission-matrix spectrometers and in the unified encoding comparison of amplitude, phase, and wavelength systems (French et al., 2017, Liu et al., 2023). Quantization and architectural choices matter in neural encoders: reducing INR weights from 32 to 16 bits caused essentially no increase in distortion, but reducing them further to 8 bits caused a large rise in distortion (Rezasoltani et al., 2023). Sampling limits cause aliasing or velocity wrap-around in velocity spectrum imaging, and high-resolution or multi-task diffractive systems face crosstalk when spectral supports overlap or the available angular bandwidth is small (Hernandez-Garcia et al., 27 Aug 2025, Kleiner et al., 8 Jan 2026).

Several directions recur across the literature. Hyperspectral INRs point to meta-learned base networks and PCA-augmented encoders (Rezasoltani et al., 2023). Diffractive systems show that angular spectrum encoding can be combined with wavelength control, suggesting multiplicative task spaces in optical computing (Kleiner et al., 8 Jan 2026). Spectral tokenization shows that coarse-to-fine spectral representations can improve conditioning for autoregressive prediction, partial decoding, and image upsampling (Esteves et al., 2024). A plausible implication is that future image-to-spectrum encoders will increasingly be hybrid systems: physically informative measurement operators paired with learned decoders, with calibration and optimization performed jointly in the spectral domain rather than treated as separate optical and computational stages.