Transformation Autoregressive Networks (TANs)

• Transformation Autoregressive Networks (TANs) are defined by combining invertible mappings with autoregressive conditional models to flexibly estimate complex data distributions.
• They merge techniques from normalizing flows and autoregressive models, enabling superior performance on synthetic benchmarks, image modeling, and anomaly detection.
• TANs leverage components like RNN-rescale and additive shift layers to facilitate efficient density learning and support applications in meta-learning and outlier detection.

Transformation Autoregressive Networks (TANs) are a class of tractable density estimators designed for general density estimation tasks. TANs integrate two previously competing methodological families: (1) the change-of-variables formula via a composition of invertible smooth transformations as in normalizing flows, and (2) autoregressive modeling of conditional densities, which factorizes the joint density into a product of one-dimensional conditionals. By learning both a powerful invertible mapping and an expressive autoregressive factorization, TANs achieve greater flexibility than either normalizing flow or autoregressive models in isolation. This framework supports density modeling across synthetic, structured, and real-world settings, and extends to applications such as anomaly detection, image modeling, and meta-learning for families of distributions (Oliva et al., 2018).

1. Mathematical Formulation

TANs seek to model the density $p_X(x)$ for vectors $x \in \mathbb{R}^d$ via an invertible map $u = T(x)$ such that $u$ is more amenable to autoregressive factorization. The resulting density factorization applies the change-of-variables formula:

$$p_X(x) = p_U(u) \cdot |\det \partial u / \partial x|, \qquad u = T(x)$$

The transformed density $p_U(u)$ is modeled autoregressively:

$p_U(u) = \prod_{i=1}^d p(u_i | u_{<i})$

The composite log-likelihood objective is:

$$\log p_X(x) = \sum_{i=1}^d \log p(u_i | u_{<i}) + \log |\det J_T(x)|$$

where $J_T(x) = \partial T(x) / \partial x$ is the Jacobian. The model allows $T$ to be a composition of multiple layers $T^{(L)} \circ \ldots \circ T^{(1)}$, with the total log-determinant being a sum over per-layer determinants.

2. Core Architectural Components

Autoregressive Conditional Models

After applying $T$, the conditional densities $p(u_i | u_{<i})$ are modeled as mixtures of Gaussians with neural network-computed parameters. Two main architectures are proposed for summarizing $u_{<i}$ into the hidden state $h_i$:

• Linear Autoregressive Model (LAM): $h_i = W^{(i)} u_{<i} + b$, with separate $W^{(i)} \in \mathbb{R}^{p \times (i-1)}$ per $i$. This provides maximal capacity at parameter cost $\mathcal{O}(d^2 p)$.
• Recurrent Autoregressive Model (RAM): $h_i = g(u_{i-1}, h_{i-1})$, $h_1 = 0$ for a recurrent cell $g(\cdot)$ (e.g., GRU or LSTM), allowing parameter sharing and non-Markovian dependency modeling. The outputs of $h_i$ are passed through an MLP to produce mixture parameters.

Invertible Transformations

Several classes of $T^{(\ell)}(\cdot)$ are introduced:

• Linear (Global): $z = L x + t$, with $L=LU$ (triangular decomposition) for tractable determinant and inversion.
• Invertible RNN-Rescale: For $i=1\ldots d$, $z_i = r_\alpha(y x_i + w^\top s_{i-1} + b)$; $s_i = r(u x_i + v^\top s_{i-1} + a)$ with $r_\alpha$ leaky-ReLU, $r$ ReLU, and all parameters learnable. Determinant is tractable; inversion proceeds recursively.
• Additive RNN Shift: $z_i = x_i + m(s_{i-1})$, $s_i = g(x_i, s_{i-1})$, $m(\cdot)$ is a small MLP, $g$ is an RNN cell. The Jacobian is unit-triangular, so determinant is 1.
• Coupling Layers: Additive coupling (NICE/Real NVP-style) splits $x$ and applies MLP shift, a special case of additive RNN shift.
• Stacking and Reversal: Multiple transformation layers are stacked; reversal ($z = (x_d,\ldots,x_1)$) is interleaved to enable bidirectional dependencies.

Empirically, 4–5 stacked layers combining linear, invertible RNN, and additive shift perform best for complex density estimation.

3. Training and Sampling Procedures

TANs are trained by maximizing the likelihood via gradient descent. Training alternates forward passes through all transformation layers, summing log-determinants, and applying the autoregressive model in the transformed space. The conditional likelihood is computed as the sum of the mixture log-probabilities. The Adam optimizer is used, with initial learning rate $5 \times 10^{-3}$, decayed periodically, and gradient clipping at norm 1.

Sampling proceeds by ancestral sampling in the transformed ($u$) space using the autoregressive conditionals, followed by inversion through the sequence of transforms to obtain samples in $x$ space. All operations are fully differentiable.

4. Model Variants and Crossed Evaluations

TANs support systematic variation in both conditional modulator and transformation components:

Conditional Module	Transformation Stack
SingleInd (factorized Gauss)	None, Linear only
MultiInd (non-conditioning)	1× RNN-rescale, 2× RNN-rescale (with reversal)
Tied (NADE-style)	4× additive coupling (NICE), 4× additive RNN-shift
LAM	Linear $\rightarrow$ RNN-rescale $\rightarrow$ 4× additive shift
RAM	Stacked as above

Empirical observations establish that pure autoregressive models struggle with high non-Markovian or nonlinearly entangled data, while pure flows (e.g., NICE) are ineffective when the base distribution exhibits dependencies. The most capable models pair rich transformation stacks (linear + RNN + shift) with LAM or RAM conditioners.

5. Hyperparameter Settings

• Conditionals: Mixtures of 40 Gaussians per conditional.
• Hidden size: 120 for LAM/RAM states; 16 for RNN transforms.
• Optimization: Adam, initial learning rate $5 \times 10^{-3}$, learning rate decay (0.1 or 0.5 every 5,000 steps), 30,000 steps total.
• Batch size: 256 (increased to 1024 for large UCI datasets).
• Gradient clipping: norm 1.
• Leaky-ReLU slope $\alpha = 0.01$ for invertible RNNs.
• No normalization penalty for additive shift layers (unit-triangular Jacobian).

6. Empirical Validation and Applications

Synthetic Benchmarks

On Markovian sinusoids with random-walk noise ($d=32$), TANs using RAM+4×SRNN+Reversal achieve log-likelihood $\approx 16.2$, outperforming single-prong methods. On star-structured graphical models ($d=32, 128$), TANs handle both hub and peripheral structure, where pure transformation or conditional approaches struggle.

Real-world Datasets

For UCI datasets (POWER, GAS, HEPMASS, MINIBOONE, BSDS300 with up to $d=63$), TANs surpass MADE, Real NVP, and MAF models, e.g., on POWER, achieving $0.60$ nats compared to $0.24$ nats for MAF(10).

Image Modeling

Continuous pixel modeling on MNIST ($28 \times 28$) and CIFAR-10 ($32 \times 32 \times 3$) after dequantization and logit-transform shows bits-per-pixel improvements of $\sim0.1$–$0.2$ over MAF and Real NVP. Generated samples exhibit high visual quality and digit-like appearance.

Anomaly Detection

TANs, when used for outlier detection (ODDS datasets: forest, pendigits, satimage2), achieve the highest average precision, implying that the learned densities capture semantically meaningful structure.

Learning Distribution Families

TANs are combined with DeepSets encoders for modeling a continuum of distributions. For $N$ sets $X_1,\ldots,X_N$ each from $P_{\theta_n}$, a shared TAN $p_W(\cdot \mid \phi(X))$ is conditioned on a permutation-invariant embedding $\phi(X)$. For ShapeNet point clouds (category as family), the model generalizes to previously unseen objects.

7. Significance and Unified Perspective

TANs constitute a unified framework interpolating between normalizing flows and autoregressive models. By simultaneously learning invertible transformations (linear, RNN-rescale, additive shift) and rich autoregressive conditioners (LAM, RAM), TANs achieve state-of-the-art performance in density estimation, anomaly detection, image modeling, and meta-learning of distribution families. The empirical results demonstrate that jointly leveraging expressive transformations and conditional modeling provides superior flexibility and accuracy compared to using either alone (Oliva et al., 2018).