---
title: Image-to-Spectrum Encoding Overview
url: https://www.emergentmind.com/topics/image-to-spectrum-encoding
type: topic
---

# Image-to-Spectrum Encoding Overview

Searching arXiv for recent and core papers on image-to-spectrum encoding and closely related spectral encoding methods.
Image-to-Spectrum Encoding denotes a family of representations and measurement schemes in which image-domain information is converted into a spectral, transform-domain, or spectrum-like code that can later be inverted to recover the original scene, a higher-dimensional field, or a distribution over an internal variable. In the cited literature, this appears in several exact forms: hyperspectral images are modeled as mappings from pixel coordinates to full spectral signatures and compressed by implicit neural representations; single-pixel and fiber systems encode spatial structure into Fourier coefficients or broad spectral codes; computational spectral imagers map \(I(x,y,\lambda)\) into one or a few 2D sensor measurements; complex audio spectrograms encode amplitude and phase into color pixels; and magnetic-resonance velocity spectrum imaging encodes a velocity distribution inside each voxel and recovers it by Fourier inversion [2302.04129] [1504.03823] [2312.13310] [2508.20218].

## 1. Formal scope and mathematical models

A compact way to formalize the topic is as an encoding operator that maps an object, image, or latent field into measurements whose coordinates are spectral coefficients, wavelength responses, transform coefficients, or spectrum-like variables. In the hyperspectral compression setting, the object is explicitly written as a mapping
\[
I:(p_x,p_y)\rightarrow(ch_0,ch_1,\ldots,ch_n),
\]
and the encoder learns a coordinate-to-spectrum approximation \(I(p_x,p_y)\approx f_\Theta(p_x,p_y)\), with the network parameters acting as the compressed code [2302.04129]. In computational spectral imaging, a broader physical forward model is
\[
I_{rgb}=\int W(\lambda)\cdot P(x,y,\lambda)\ast\big(A(x,y,\lambda)\cdot I(x,y,\lambda)\big)\,d\lambda,
\]
which unifies amplitude, phase, and wavelength encoding into a single operator from a 3D spectral cube to a 2D measurement [2312.13310].

A second canonical form is the linear inverse model
\[
\mathbf{y}=\mathbf{A}\mathbf{x}+\boldsymbol{\eta},
\]
where \(\mathbf{x}\) is a discretized spectrum or multispectral cube, \(\mathbf{y}\) is a measured image or detector output, and \(\mathbf{A}\) is a calibrated encoding matrix. This appears in nanowire-mat spectrometers, single-fiber spread-spectrum imaging, and micro-structured diffractive spectral cameras, with reconstruction posed as a regularized inversion of \(\mathbf{A}\) [1705.02991] [1505.07491] [1705.09321].

A third form is explicitly transform-domain. Efficient single-pixel imaging treats the unknown scene \(f(x,y)\) through its Fourier coefficients \(F(u,v)\), and designs sinusoidal probe patterns so that bucket measurements directly sample the scene’s spatial spectrum [1504.03823]. Velocity spectrum imaging uses modified velocity-selective RF pulses to encode a voxel’s internal velocity distribution, yielding
\[
M_z(m_g)=\int M_{xy}(v)e^{i m_g v}\,dv,
\]
so that the inverse Fourier transform over the encoding variable \(m_g\) recovers a velocity spectrum \(M_{xy}(v)\) per voxel [2508.20218]. In the audio domain, complex spectrogram coefficients \(X_m[k]=A_m[k]e^{i\phi_m[k]}\) are encoded into image pixels by mapping phase to hue and amplitude to brightness and saturation, with exact reversal of the mapping recovering the original sound [1907.09936].

| Form | Encoding map | Representative papers |
|---|---|---|
| Coordinate-to-spectrum | \((p_x,p_y)\mapsto(ch_0,\ldots,ch_n)\) | [2302.04129], [2312.01558] |
| Fourier-domain sampling | \(f(x,y)\mapsto\{F(u,v)\}\) | [1504.03823], [2002.10789] |
| Linear spectral imaging | \(\mathbf{y}=\mathbf{A}\mathbf{x}+\boldsymbol{\eta}\) | [1705.02991], [1705.09321], [1505.07491] |
| Velocity-spectrum imaging | \(M_z(m_g)\mapsto M_{xy}(v)\) by Fourier inversion | [2508.20218] |
| Complex spectrogram imaging | \(X_m[k]\leftrightarrow\) color pixel | [1907.09936] |
| Spectral tokenization | image \(\mapsto\) DWT-based discrete tokens | [2412.09607] |

This suggests that “spectrum” in the literature is not restricted to wavelength alone. It also includes Fourier coefficients, wavelet coefficients, angular spectra, complex phase-amplitude encodings, and voxel-wise distributions over latent variables when the transform relation is explicit.

## 2. Optical and physical realizations

In physical optics, image-to-spectrum encoding often begins by selecting an orthogonal or approximately orthogonal basis and designing illumination or scattering so that measurements directly correspond to basis coefficients. Efficient single-pixel imaging projects sinusoidal patterns and uses two \(\pi/2\)-phase-shifted patterns to recover each Fourier coefficient in the most informative spatial frequency bands, exploiting centrosymmetric conjugation and sparsity of natural-image spectra; the method reduces requisite patterns by two orders of magnitude compared to conventional SPI [1504.03823]. Closely related spectral-domain single-pixel imaging replaces spatial masks with spectral Fourier probe patterns imposed by a programmable optical filter, so that an unknown transmission \(T(\omega)\) is reconstructed from single-pixel measurements as a Fourier series in the spectral domain [2002.10789]. In scan-less confocal phase microscopy, two-dimensional image pixels are encoded onto optical-frequency-comb modes via a VIPA–grating disperser and then decoded from mode-resolved amplitude and phase spectra using dual-comb spectroscopy [1705.07236].

A second physical family uses wavelength-dependent codes that are broad, overlapping, and later inverted computationally. In spread-spectrum single-fiber imaging, each spatial pixel is converted into a distinct spectral code spanning the full object bandwidth, the spectrum is transmitted through a single optical fiber, and the image is recovered numerically from the detected spectrum; this scheme is explicitly described as spatial information converted to spectral information, is insensitive to fiber bending, and contains no moving parts [1505.07491]. A closely related microfabricated spatio-spectral encoder uses a \(4\times4\) array of Fabry–Pérot–metasurface filters so that each spatial pixel is tagged with a narrowband spectral code over 560–625 nm and the original pattern is reconstructed by pseudo-inverse; the recovered image remains unchanged with fiber bending or moving [2305.17113]. Computational multi-spectral video imaging places a micro-structured diffractive filter near the sensor so that spectral information is converted to a spatial code on sensor pixels and the multispectral cube is recovered from a single frame by regularization-based linear algebra [1705.09321].

Disordered and reconfigurable spectral encoders replace deterministic gratings or filter arrays with complex response matrices. In nanowire-mat hyperspectral imaging, strong diffuse scattering maps different wavelengths to distinct speckle patterns with nanometer sensitivity, and a transmission matrix is calibrated so that spectra are reconstructed from monochrome camera measurements; compressive sensing is especially effective for sparse spectra and noisy conditions [1705.02991]. Bio-inspired photonic spectral encoders generalize this idea by optimizing orthogonality, completeness, and sparsity of the response matrix under a Bayesian expected-information-gain framework, and experimentally validate a dynamically reconfigurable encoder with 6 pm resolution over a 30 nm measurable bandwidth [2601.12228].

## 3. Learned, implicit, and transform-domain representations

In implicit neural representations for hyperspectral compression, the encoder is no longer a physical optical element but a learned continuous function. A multilayer perceptron with sinusoidal activations learns the mapping from 2D coordinates to a full band vector, and the weights of the network become the compressed representation of the image; decoding is simply evaluation of the learned function at pixel coordinates [2302.04129]. A later variant adds a sampling strategy based on window size and sampling rate to reduce compression time while preserving or improving PSNR and SSIM, and reports better low-bitrate performance than JPEG, JPEG2000, PCA-DCT, and several learning-based baselines on Indian Pines, Jasper Ridge, Pavia University, and Cuprite [2312.01558].

Other learned encoders operate in physically realized spectral spaces. Vortex-encoded spectral correlations use phase masks \(M(x,y)\propto e^{\pm i m\phi}\) so that the sensor intensity becomes a quadratic polynomial in derivatives of the underlying Fourier field, and a learned linear transformation reconstructs or distills useful image features from the resulting correlation patterns [2304.12172]. Diffractive networks controlled by illumination angular spectrum encode task identity in the angular components of illumination while the object image remains in spatial amplitude, enabling a single diffractive stack to perform multiple image-to-image mappings; the task is determined by the illumination mask, which serves as a unique task encoder [2601.04825]. The unified encoding model in computational spectral imaging makes this encoder–decoder viewpoint explicit by jointly optimizing physical amplitude, phase, or wavelength encoders with digital decoders under the same loss [2312.13310].

A transform-domain digital counterpart appears in the Spectral Image Tokenizer. Here the image is first converted to a multi-level 2D discrete wavelet transform, each scale is patchified and vector-quantized, and the resulting sequence of discrete tokens is ordered from low frequency to high frequency [2412.09607]. The token sequence is therefore coarse-to-fine rather than raster-ordered, enabling partial decoding, multiscale generation, and text-guided upsampling. This suggests that image-to-spectrum encoding also functions as a sequence-design principle for autoregressive models when the “spectrum” is wavelet rather than optical.

## 4. Decoding and reconstruction methods

Decoding is usually posed as an inverse problem. In single-fiber spectral encoding and micro-structured spectral filter arrays, the reconstruction is the pseudo-inverse or least-squares solution of the calibrated linear system, often written as \(\mathbf{a}=M^{+}\mathbf{b}\) or \(\min_{\mathbf{a}}\|M\mathbf{a}-\mathbf{b}\|^2\) [1505.07491] [2305.17113]. In nanowire-mat spectrometers, Tikhonov regularization and \(\ell_1\)-based compressive sensing are compared directly; compressive sensing suppresses computational background and remains effective in underdetermined and noisy regimes [1705.02991]. In the bio-inspired photonic encoder, reconstruction is cast as
\[
I^*=\arg\min_{I\in\mathbb{R}_+}\|TI-S\|_2^2
\]
with additional \(\ell_1\), \(\ell_2\), and smoothness terms to adapt to sparse or continuous spectra [2601.12228]. In the unified encoding model, the decoder is a learned network such as Sim-Conv-Net, Res-U-Net, or an unfolding-based model trained jointly with the encoder [2312.13310].

Some systems admit direct or nearly direct inversion because the encoding is deliberately chosen in an orthogonal basis. Efficient single-pixel imaging reconstructs by populating Fourier coefficients and applying the inverse Fourier transform [1504.03823]. Velocity spectrum imaging reconstructs the voxel-wise distribution by Fourier transforming over the sampled first gradient moments \(m_g\) [2508.20218]. Complex spectrogram color encoding is explicitly bijective: RGB is converted back to HSV, hue yields phase, saturation or brightness yields amplitude according to the appropriate branch, and inverse STFT recovers the waveform down to the original phases [1907.09936]. In coordinate-based hyperspectral INRs, the “inverse” is not matrix inversion but function evaluation on the coordinate grid after loading the learned parameters [2302.04129].

Evaluation criteria vary with modality but consistently measure either fidelity of the recovered spectrum or utility of the encoded representation. Hyperspectral compression papers use MSE, PSNR, and SSIM [2312.01558]. Optical spectral imagers use RMSE, cross-correlation, or spectral resolution measured by decorrelation width [1504.03823] [1705.09321]. The unified encoding model compares PSNR, SAM, and ERGAS across amplitude, phase, and wavelength encoders [2312.13310]. Vortex-encoded learning uses MSE, SSIM, classification accuracy, \(H_{SVD}\), and SAD to characterize reconstruction quality, learnability, and discriminative usefulness of the encoded spectral correlations [2304.12172]. Bio-inspired photonic encoders evaluate condition number, relative residual, mutual coherence, and reconstruction error in relation to orthogonality, completeness, and sparsity [2601.12228].

## 5. Domain-specific applications

In hyperspectral imaging, image-to-spectrum encoding is used primarily for compression and compact representation. Implicit neural representations store the entire data cube as a continuous coordinate-to-spectrum function, achieve better compression than JPEG, JPEG2000, and PCA-DCT at low bitrates, and in the sampling-based variant also improve encoding speed [2302.04129] [2312.01558]. In computational spectral cameras, a micro-structured diffractive filter converts a visible or Vis–IR scene into a single coded sensor frame, enabling compact single-shot multispectral video imaging with 9.6 nm spectral resolution over 430–718 nm and software-defined trade-offs between field of view and spectral resolution [1705.09321].

In sensing and communication systems, the same principle underlies robust transmission through constrained hardware. Single-fiber spread-spectrum imaging converts a 2D self-luminous object into a measured spectrum and is insensitive to fiber bending [1505.07491]. Image sensor communication structures an optical time-domain waveform so that rolling-shutter images become a recoverable sampled representation of the signal even under variable frame rates, long exposures, and heterogeneous sensors [1606.05666]. In audio, the complex-color spectrogram makes amplitude and phase simultaneously visible, allowing humans and machine-vision models to operate on a canonical image representation of total sound while preserving exact reversibility in principle [1907.09936].

In microscopy, tomography, and magnetic resonance, image-to-spectrum encoding exposes latent variables not directly accessible in a conventional image. Optical-frequency-comb microscopy encodes 2D pixels onto comb modes and decodes a scan-less confocal image from the mode-resolved amplitude spectrum and a phase image from the mode-resolved phase spectrum, combining a confocal depth resolution of 62.4 \(\mu\)m with a phase depth resolution of 13.7 nm [1705.07236]. Velocity spectrum imaging uses modified velocity-selective RF pulses so that each voxel yields a velocity distribution rather than a single mean velocity, with the distribution decoded via Fourier transform and demonstrated on flow phantoms and human participants along three laboratory axes [2508.20218]. Diffractive networks controlled by illumination angular spectrum use the spectrum itself as a task token, allowing one physical network to perform different image-to-image translations under different illumination masks [2601.04825].

## 6. Trade-offs, limitations, and design criteria

A recurring design principle is that good encoders maximize diversity in the measurement space while preserving invertibility. The bio-inspired photonic framework formalizes this through orthogonality, completeness, and sparsity: orthogonality is linked to low condition number and low cross-talk, completeness to small residual when representing the target spectral space, and sparsity to low mutual coherence and effective regularized recovery [2601.12228]. This criterion is consistent with Fourier-domain SPI, transmission-matrix spectrometers, diffractive coded cameras, and joint optical–digital encoders, even when the papers do not use the same vocabulary.

The main limitations are similarly recurrent. Intensity-only measurements discard phase unless special encodings preserve or reconstruct it, as in complex-color audio or dual-comb microscopy [1907.09936] [1705.07236]. Poorly conditioned encoders amplify noise and require regularization, as shown in transmission-matrix spectrometers and in the unified encoding comparison of amplitude, phase, and wavelength systems [1705.02991] [2312.13310]. Quantization and architectural choices matter in neural encoders: reducing INR weights from 32 to 16 bits caused essentially no increase in distortion, but reducing them further to 8 bits caused a large rise in distortion [2302.04129]. Sampling limits cause aliasing or velocity wrap-around in velocity spectrum imaging, and high-resolution or multi-task diffractive systems face crosstalk when spectral supports overlap or the available angular bandwidth is small [2508.20218] [2601.04825].

Several directions recur across the literature. Hyperspectral INRs point to meta-learned base networks and PCA-augmented encoders [2312.01558]. Diffractive systems show that angular spectrum encoding can be combined with wavelength control, suggesting multiplicative task spaces in optical computing [2601.04825]. Spectral tokenization shows that coarse-to-fine spectral representations can improve conditioning for autoregressive prediction, partial decoding, and image upsampling [2412.09607]. A plausible implication is that future image-to-spectrum encoders will increasingly be hybrid systems: physically informative measurement operators paired with learned decoders, with calibration and optimization performed jointly in the spectral domain rather than treated as separate optical and computational stages.

Source: https://www.emergentmind.com/topics/image-to-spectrum-encoding