PHOTON: Hierarchical Model for Efficient Decoding

• PHOTON is a hierarchical autoregressive model that employs a bottom-up encoder and top-down decoder to efficiently reconstruct token details using multi-resolution latent streams.
• It significantly reduces memory-bound KV cache overhead by compressing sequences into coarser latent representations, leading to order-of-magnitude improvements in throughput.
• Experimental results demonstrate that PHOTON matches or exceeds vanilla Transformer quality while cutting memory footprint and boosting efficiency in long-context scenarios.

Parallel Hierarchical Operation for Top-down Networks (PHOTON) is a hierarchical autoregressive model for efficient language generation that fundamentally alters the standard paradigm of left-to-right, token-by-token traversal in Transformers. PHOTON introduces a vertical, multi-resolution context representation by maintaining a persistent hierarchy of latent streams, with a bottom-up encoder forming coarse-level summaries and a top-down, chunk-local autoregressive decoder reconstructing fine-grained token-level details as needed. This architectural transition from a flat to a hierarchical latent state significantly reduces memory-bound Key-Value (KV) cache overhead, enabling order-of-magnitude improvements in throughput per unit memory during long-context and high-concurrency decoding scenarios (Ichikawa et al., 22 Dec 2025).

1. Theoretical Foundation and Motivation

PHOTON leverages the intrinsic hierarchical structure of language (subwords → words → sentences → documents) to impose a hierarchy of latent states over the token sequence. In standard Transformers, each decoding step accesses and updates ever-growing token-level states, resulting in prefill latencies and a linear growth of the KV cache with sequence length $T$. This growth transitions inference to a memory-bound regime where memory bandwidth limitations, rather than computational throughput, become the primary bottleneck.

By compressing the sequence bottom-up into progressively coarser latent streams, and reconstructing fine token details only when required in a localized top-down fashion, PHOTON amortizes most global context operations over much shorter sequences. This translates to significant reductions in memory footprint, bandwidth, and ultimately KV-cache traffic, thus unlocking much higher throughput (Ichikawa et al., 22 Dec 2025).

2. Hierarchical Latent Stream Architecture

PHOTON’s multi-resolution latent representation comprises $L$ hierarchy levels. The central components are:

• Bottom-up Encoder: At each level $l$, the encoder groups $C_l$ consecutive vectors $X^{(l-1)}$ (token or lower-level representations) into chunk-level summaries via a context chunker $C^{(l)}$, then processes these with an autoregressive Transformer context encoder $F^{(l)}$. For $T$ input tokens and chunk sizes $C_1, ..., C_L$, the sequence at level $l$ has length $M_l = T / (C_1 \cdots C_l)$.
• Top-down Decoder: Given coarse latent representations $\hat{X}^{(l)}$ at level $l$, the top-down stack recursively reconstructs finer-grained representations for lower levels. Local context converters $U^{(l)}$ expand each chunk into conditioning vectors, and local autoregressive decoders $G^{(l)}$ perform strictly block-local, chunk-causal decoding for fine-grained token reconstructions.

All levels’ latent streams are maintained in the KV cache, but updates are infrequent, as each global/coarse level is only updated once per $C_{\leq l}$ tokens, where $C_{\leq l} = \prod_{k=1}^l C_k$ (Ichikawa et al., 22 Dec 2025).

3. Computational Complexity and Throughput-Memory Trade-offs

PHOTON significantly alters prefill and decode-time computational and memory requirements. The joint autoregressive factorization over tokens $t_{1:T}$ is preserved, but the internal state transitions leverage local causality and chunk-based updates.

Key metrics include:

• Prefill Cost: $$\sum_{l=1}^{L} O(M_l^2) = \sum_{l=1}^L O((T/C_{\leq l})^2)$$, lower than the $O(T^2)$ scaling in Transformers.
• KV-Cache Size: $\sum_{l=1}^L O(M_l) = \sum_{l=1}^L O(T/C_{\leq l})$, as opposed to $O(T)$.
• Decode-Time KV Reads per Token: Scaling as $\sum_{l=1}^L O(T / C_{\leq l}^2)$ for global reads, with a strictly bounded local term $O(1)$ with respect to $T$.

With two levels and $C_1 = C_2 = 4$, the reduction is as follows:

Model	Global KV Reads per Token
Baseline Transformer	$\sim 1 \cdot T$
PHOTON ($C_1$=$C_2$=4)	$\sim 0.07\,T$

This constitutes $\sim14\times$ fewer global KV reads per token (Ichikawa et al., 22 Dec 2025).

4. KV-Cache Traffic and Throughput per Memory (TPM)

The restructuring of cache usage is reflected in the metric “TPM” (Throughput per GiB of KV memory). Benchmark results demonstrate:

• For a 600M parameter model, in prefill regime (2048 input / 128 output), the vanilla Transformer achieves TPM ${\approx}3.24$K tok/s/GiB, whereas PHOTON achieves ${\approx}1262.6$K tok/s/GiB, a $390\times$ gain.
• In decode-heavy regime (128 input / 2048 output), TPM for PHOTON reaches $3062$K tok/s/GiB ($417\times$ gain).
• Similar improvements hold for 1.2B-scale models and across both prefill and decode-heavy contexts.

Comparisons against Block Transformers (one-level global + local design) show PHOTON consistently on the Pareto frontier in TPM versus perplexity and zero-shot accuracy, with strictly superior throughput and memory efficiency (Ichikawa et al., 22 Dec 2025).

5. Experimental Results and Quality

PHOTON’s empirical evaluations utilize models with 600M and 1.2B parameters, implemented with LLaMA-style blocks. Training on the Pile-uncopyrighted (134B tokens), the experimental protocol includes context windows of 2048 tokens, Adam optimizer ($3\times 10^{-4}$ learning rate), and auxiliary recursive reconstruction and next-context prediction losses.

Measured on Wikitext perplexity and zero-shot accuracy across HellaSwag, SciQ, and ARC-Easy, PHOTON attains quality on par with or superior to vanilla Transformers while dramatically reducing KV memory usage ($\sim 10\times$ less) and boosting raw throughput ($\sim 40$–$45\times$ higher) (Ichikawa et al., 22 Dec 2025).

6. Practical Considerations and Deployment

PHOTON reduces the per-sample memory footprint to $O(T/C_1 + T/(C_1 C_2) + \cdots)$—for a 2048-token prefix on a 600M parameter model, this is about $30$ MiB versus $275$ MiB for standard Transformers. Decode latency becomes compute-bound due to strictly local decoders (window size $\leq (R_l+C_l)$), and global state updates are needed only every $C_{\leq l}$ tokens.

Practical deployment advantages include:

• Substantial reduction of GPU-host memory traffic and PCIe/NVLink bandwidth.
• Higher concurrency of users per RAM budget.
• Maintainability of multiple small KV caches (one per hierarchy level).
• Modular optimization through specialized kernels for chunked, block-local attention.

The multi-resolution top-down generation framework shifts the inference bottleneck from memory bandwidth back to arithmetic throughput, enabling up to $10^3\times$ improvements in throughput per unit memory for long-context, multi-query scenarios (Ichikawa et al., 22 Dec 2025).

7. Relationship to Broader Top-Down and Hierarchical Modeling

PHOTON’s hierarchical-autoregressive approach shares high-level conceptual lineage with top-down information flow and coarse-to-fine representations found in vision architectures such as Top-Down Networks (TDN) (Lelekas et al., 2020). Both exploit the efficiency of working with coarser-grained latent representations for global context, refining only locally as needed. While TDN applies this principle to spatial resolution in convolutional networks, with benefits for robustness and explainability, PHOTON applies it to sequence modeling, yielding profound implications for efficiency and scalability in language generation. This suggests fertile ground for cross-domain innovation in hierarchical inference and representation learning (Ichikawa et al., 22 Dec 2025, Lelekas et al., 2020).