---
title: Integer Discrete Flows (IDFs)
url: https://www.emergentmind.com/topics/integer-discrete-flows-idfs
type: topic
---

# Integer Discrete Flows (IDFs)

An Integer Discrete Flow (IDF) is a bijective, invertible transformation $f_\theta:\mathbb{Z}^D\rightarrow\mathbb{Z}^D$ parameterized for learning rich probability models over high-dimensional integer-valued random variables. Unlike normalizing flows for continuous data, IDFs work natively on the integer lattice, enabling exact log-likelihoods, invertible inference, and lossless compression without real-to-integer quantization artifacts. Architecturally, IDFs stack integer-valued coupling layers (often bipartite), where each layer performs an integer-only, learnable, invertible transformation such as additive shift with neural network parameterization. Their density evaluation, training, and compression procedures are tailored to the discrete structure, with practical advantages in both generative modeling and entropy coding. Modern IDF architectures incorporate improvements in expressivity, gradient estimation, and hardware efficiency, resulting in state-of-the-art lossless compression for image datasets and competitive density modeling performance.

## 1. Mathematical Definition and Change-of-Variables

At the core of the IDF framework is the composition of $L$ integer-valued bijective layers,
$$
f_\theta = f^{(L)}_{\theta_L} \circ f^{(L-1)}_{\theta_{L-1}} \circ ... \circ f^{(1)}_{\theta_1}\,,
$$
with $x\in\mathbb{Z}^D$ mapped to $z=f_\theta(x)\in\mathbb{Z}^D$. The discrete change-of-variables formula omits the usual Jacobian determinant, as there is no notion of infinitesimal volume in discrete space:
$$
p_X(x) = p_Z(z) = p_Z(f_\theta(x)),\qquad z=f_\theta(x)\,,
$$
where $p_Z$ is typically a simple factorized or mixture prior on $\mathbb{Z}^D$. Log-likelihood is thus
$$
\log p_X(x) = \log p_Z(f_\theta(x))\,.
$$
The invertibility of $f_\theta$ ensures exact one-to-one mapping between data and latent codes, allowing both density evaluation and invertible sample generation [1905.07376][2006.12459][2011.15056].

## 2. Core Layer Constructions: Coupling and Permutation

The canonical IDF building block is the integer discrete bipartite coupling layer. Splitting an input vector $x$ into $x=[x_a, x_b]$, the forward pass is
$$
y_a = x_a,\qquad y_b = x_b + \lfloor t_\theta(x_a)\rceil\,,
$$
where $t_\theta:\mathbb{Z}^{d_a}\rightarrow\mathbb{R}^{d_b}$ is typically a neural network and $\lfloor\cdot\rceil$ denotes rounding to nearest integer. The inverse recovers $x$ exactly:
$$
x_a = y_a,\qquad x_b = y_b - \lfloor t_\theta(y_a)\rceil\,.
$$
Alternating with channel permutations ensures all components eventually interact. Crucially, all operations are closed on $\mathbb{Z}^D$ and preserve bijectivity.

Recent IDF variants introduce more expressive multiway coupling, e.g., 4- and 8-part splits, where each output block $y_i$ is updated via a dedicated neural net, enabling richer dependency structures per layer at constant parameter budget [2011.15056].

For sequence or categorical data on $\{0,...,K-1\}^D$, modular arithmetic coupling is used:
$$
y_d = [\mu_d(y_{<d}) + \sigma_d(y_{<d}) x_d] \bmod K
$$
with modular inverse for the backward pass [1905.10347].

## 3. Training, Gradient Estimation, and Flexibility

IDFs admit exact maximum-likelihood training: the negative log-likelihood objective is minimized by backpropagation, using the discrete change-of-variables formula. The main technical challenge is the nondifferentiability of rounding:
- Forward: $u = \lfloor t_\theta\rfloor$;
- Backward: straight-through estimator $\partial L/\partial t := \partial L/\partial u$.

Empirical analysis demonstrates that, despite this bias, the straight-through gradient aligns well with finite-difference estimates in 8–16 bit image data; architecture depth and network conditioning are important for optimization stability [2006.12459].

A previously raised concern was reduced expressivity relative to continuous flows. This is refuted in the IDF++ analysis, which proves that embedding finite-support data in $\mathbb{Z}^D$ allows IDFs to exactly factorize (flatten) or transform any joint distribution via successive translation and permutation layers [2006.12459].

## 4. Lossless Compression and Entropy Coding

IDFs naturally support lossless compression by mapping data $x$ to latent code $z=f_\theta(x)$ and encoding $z$ under $p_Z(z)$ (via rANS or arithmetic coding). Decoding reverses the process exactly using $f_\theta^{-1}$. The strictly integer nature of all transformations guarantees no information loss and enables achieving bitrates competitive with or better than established codecs. The entire model is thus a learned, invertible statistical compressor with exact recovery [1905.07376][2006.12459][2206.08869].

Empirical results for IDF-based compressors on CIFAR-10 and ImageNet are included in Table 1:

| Model                | CIFAR-10 (bpd) | ImageNet32 (bpd) | ImageNet64 (bpd) |
|----------------------|----------------|------------------|------------------|
| PNG                  | 5.87           | 6.39             | 5.71             |
| FLIF                 | 4.19           | 4.52             | 4.19             |
| Bit-Swap             | 3.82           | 4.50             | —                |
| **IDF**              | 3.32           | 4.18             | 3.90             |
| **IDF++ (8 f/l)**    | 3.26           | 4.12             | 3.81             |
| IODF (INT8+pruned)   | 3.979          | 3.695            | —                |

## 5. Architectural Improvements and Hardware Efficiency

IDF++ introduces several architectural optimizations:
- Inverted channel permutations after each coupling to preserve spatial coherence.
- ReZero-style identity initialization: $y_b = x_b + \lfloor \alpha\cdot t_\theta(x_a)\rceil$ with learnable $\alpha$ initialized at zero.
- GroupNorm and Swish activations inside dense blocks.
- Zero initialization for logistic-mixture parameters in the conditional prior [2006.12459].

Hardware-friendly implementations such as Integer-Only Discrete Flows (IODF) replace floating-point subnetworks with integer-only ResNet blocks and add learnable binary gates for aggressive channel pruning. All convolutions and ReLUs are performed in INT8, producing an order-of-magnitude reduction in inference latency with minimal rate loss [2206.08869].

## 6. Alternative Flow Structures and Theoretical Relations

Alternate IDF architectures include:
- Discrete autoregressive and bipartite flows, with modular affine coupling for categorical data, supporting bidirectional context (DAF) and parallel non-autoregressive sampling (DBF) [1905.10347].
- Multi-way integer couplings, unifying reversible logic gates, discrete NVP layers, and general neural network couplings under a broad class of invertible templates [2011.15056].
- Measure-preserving and discrete (MAD) maps constructed from CDF-inverse actions on $\mathbb{Z}^d$, as in MAD Mix, yielding unbiased variational flows that are ergodic and parameter-free aside from mixture depth [2308.15613].

## 7. Applications, Extensions, and Empirical Observations

IDFs provide:
- Exact likelihoods on integer and categorical data without dequantization;
- Native and invertible image compression with state-of-the-art rates;
- Competitive performance in character-level language modeling, synthetic joint distributions, Potts models, and permutation-based data;
- Modular hybridization with continuous flows (e.g., interleaving integer and continuous coupling blocks), expanding applicability to mixed discrete/continuous domains [1905.10347][2011.15056][2204.01361].

Limitations include training challenges with large alphabet sizes ($K>200$), gradient bias from the straight-through estimator in extremely low-bit regimes, and sequential dependencies in autoregressive flow architectures.

## References

- "Integer Discrete Flows and Lossless Compression" [1905.07376]
- "IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression" [2006.12459]
- "Discrete Flows: Invertible Generative Models of Discrete Data" [1905.10347]
- "General Invertible Transformations for Flow-based Generative Modeling" [2011.15056]
- "Fast Lossless Neural Compression with Integer-Only Discrete Flows" [2206.08869]
- "Mixed Variational Flows for Discrete Variables" [2308.15613]
- "Discretely Indexed Flows" [2204.01361]

Source: https://www.emergentmind.com/topics/integer-discrete-flows-idfs