ProGen3 Protein Language Model

• ProGen3 is a protein language model characterized by a 10-layer transformer with sparse Mixture-of-Experts blocks, enabling both autoregressive generation and span infilling.
• The training regime combines causal language modeling and span-infilling objectives in a 2:1 ratio, facilitating robust next-token prediction and masked reconstruction.
• ProGenMech applies cross-layer transcoders to reconstruct ProGen3's latent computations, yielding interpretable circuits that recover biologically meaningful motifs.

Searching arXiv for the specified paper and closely related ProGen/interpretability context. arXiv search query: (Tsui et al., 14 Jun 2026) ProGen3 is a protein LLM (pLM) described as a 10-layer transformer with sparse Mixture-of-Experts (MoE) blocks and trained jointly for autoregressive generation and span infilling. In the mechanistic-interpretability study "Circuit Tracing in Autoregressive Protein LLMs" (Tsui et al., 14 Jun 2026), ProGen3 serves as the generative substrate for ProGenMech, a framework that extends cross-layer transcoders (CLTs) to autoregressive protein generation. Within that account, ProGen3 is characterized both as a generative model over protein sequences and as an object of circuit-level analysis, with sparse latent structure used to recover generative computation, zero-shot fitness behavior, and biologically meaningful motifs (Tsui et al., 14 Jun 2026).

1. Architectural definition

ProGen3 is specified as a 10-layer transformer with residual-stream dimension $d_{\text{model}} = 384$, with sparse MoE in each “MLP” block. At layer $\ell$, the residual stream $x^\ell \in \mathbb{R}^{d_{\text{model}}}$ is routed through $E$ expert sub-networks $M_i(\cdot)$, and a lightweight gating network computes a soft selection over experts:

$$g(x^\ell)=\operatorname{softmax}(W_g x^\ell + b_g)\in \mathbb{R}^E.$$

The MoE output is

$$y^\ell(x^\ell)=\sum_{i=1}^E g_i(x^\ell)\cdot M_i(x^\ell).$$

The transformer then adds $y^\ell$ to $x^\ell$, passes through layer norm, and proceeds to layer $\ell+1$ (Tsui et al., 14 Jun 2026).

Two heads are described. For autoregressive generation, each token $\ell$0 has hidden state $\ell$1, defined as the final residual activation at position $\ell$2, and the next-residue probability is

$\ell$3

For span infilling, spans of the input are masked, and the same head predicts the missing residues conditioned on their left and right context. This dual use of a common output head situates ProGen3 as a generalized sequence model rather than a model restricted to strictly left-to-right decoding.

A plausible implication is that the architecture is designed to support both local autoregressive continuation and bidirectionally constrained reconstruction, which is central to the later interpretability analysis: the same backbone must support distinct conditional regimes without changing the output interface.

2. Training objectives and generative regime

ProGen3 is trained on a mixture of causal language modeling (CLM) and span-infilling generalized language modeling (GLM) examples in a 2:1 ratio. Let $\ell$4 denote total sequence length and let $\ell$5 denote masked span positions. The causal loss is

$\ell$6

while the span-infilling loss is

$\ell$7

With $\ell$8 weight on CLM and $\ell$9 on GLM, the combined objective is

$x^\ell \in \mathbb{R}^{d_{\text{model}}}$0

This training specification defines ProGen3 as a mixed-objective protein sequence model rather than a purely causal generator (Tsui et al., 14 Jun 2026).

The distinction matters for the later experimental results. The mechanistic analysis explicitly evaluates both causal generation and span infilling, and the training design provides the formal basis for doing so. This suggests that ProGen3’s internal computation is expected to support both next-token prediction and masked-span reconstruction, which in turn motivates interpretability methods that can track inter-layer dependencies instead of treating each layer in isolation.

3. ProGenMech and cross-layer reconstruction

In the cited study, ProGen3 is not only a sequence model but also the target of a mechanistic replacement procedure. ProGenMech is introduced as a mechanistic interpretability framework for generative protein LLMs that extends cross-layer transcoders to ProGen3. The stated goal is to replace each MoE block by a sparse, interpretable bottleneck that reconstructs the original computation across layers (Tsui et al., 14 Jun 2026).

At layer $x^\ell \in \mathbb{R}^{d_{\text{model}}}$1, the encoder maps $x^\ell \in \mathbb{R}^{d_{\text{model}}}$2 to a high-dimensional latent pre-activation and sparsifies it via TopK:

$x^\ell \in \mathbb{R}^{d_{\text{model}}}$3

where TopK keeps the $x^\ell \in \mathbb{R}^{d_{\text{model}}}$4 largest magnitudes and zeroes out the rest. The decoder reconstructs the true MoE output $x^\ell \in \mathbb{R}^{d_{\text{model}}}$5 by linearly combining all previous latents:

$x^\ell \in \mathbb{R}^{d_{\text{model}}}$6

The training objective per layer is the sum of a reconstruction loss and an auxiliary “dead-latent” loss:

$x^\ell \in \mathbb{R}^{d_{\text{model}}}$7

and

$x^\ell \in \mathbb{R}^{d_{\text{model}}}$8

where $x^\ell \in \mathbb{R}^{d_{\text{model}}}$9 and $E$0 decodes the top $E$1 entries of $E$2. The full CLT loss is

$E$3

An equivalent formulation is also given: the CLT may be viewed as learning $E$4 to approximate the true hidden-state distribution $E$5 by minimizing

$E$6

where $E$7 denotes the sparse latents up to layer $E$8.

The contrast with per-layer methods is explicit. Unlike per-layer approaches, CLTs reconstruct each layer using sparse latent variables from all preceding layers, and this is presented as enabling faithful recovery of inter-layer generative computation. In that framing, ProGen3 is important because its autoregressive and span-infilling behavior is taken to require a cross-layer account of computation rather than a merely local one.

4. Zero-shot circuit discovery

After training a CLT replacement for ProGen3, the study seeks minimal “circuits,” defined as small subsets of the total latent space that still reproduce the model’s behavior in generation and fitness scoring. During circuit search, the true MoE output at each layer is replaced by the CLT reconstruction $E$9, while the attention sublayer outputs are kept fixed to the original ProGen3 activations in order to avoid error blowup (Tsui et al., 14 Jun 2026).

The task metrics are defined in two parts. For generation, including both CLM and GLM, the metric is average negative log-likelihood (NLL) of generated tokens under the original ProGen3. For zero-shot fitness, the metric is Spearman $M_i(\cdot)$0 between model log-likelihoods, averaged forward and backward, and experimental deep-mutational-scanning (DMS) fitness.

Circuit selection proceeds by a thresholded greedy procedure. Baseline performance is computed with all latents active, yielding either $M_i(\cdot)$1 for generation or $M_i(\cdot)$2 for fitness. A target threshold is then defined as $M_i(\cdot)$3 for generation or $M_i(\cdot)$4 for fitness, where $M_i(\cdot)$5 is the performance of the full-latent replacement. Each latent $M_i(\cdot)$6 at layer $M_i(\cdot)$7 is attributed by

$M_i(\cdot)$8

summing over all positions; for fitness tasks, $M_i(\cdot)$9 is over the entire logit matrix. Latents are then added greedily in batches of 32, in descending order of $$g(x^\ell)=\operatorname{softmax}(W_g x^\ell + b_g)\in \mathbb{R}^E.$$0, until the task metric meets $$g(x^\ell)=\operatorname{softmax}(W_g x^\ell + b_g)\in \mathbb{R}^E.$$1 or 1,000 latents are reached.

The biological mapping procedure is also defined operationally. For each selected latent, the top-activating sequences are retrieved from Swiss-Prot, aligned, and the activations are overlaid on known protein structures to identify conserved biochemical and structural motifs, with examples including HRD and DFG kinase motifs and GRB2 binding loops. This procedure links ProGen3’s sparse latent circuits to recognizable sequence and structural regularities rather than leaving them as abstract features.

5. Empirical behavior

The reported results characterize ProGen3 both as a baseline model and as the target distribution that ProGenMech seeks to recover. In CLM generation on $$g(x^\ell)=\operatorname{softmax}(W_g x^\ell + b_g)\in \mathbb{R}^E.$$2 Swiss-Prot test proteins, original ProGen3 attains NLL $$g(x^\ell)=\operatorname{softmax}(W_g x^\ell + b_g)\in \mathbb{R}^E.$$3. ProGenMech with all latents attains NLL $$g(x^\ell)=\operatorname{softmax}(W_g x^\ell + b_g)\in \mathbb{R}^E.$$4, described as approximately 60% likelihood recovery, while the PLT baseline with all latents attains NLL $$g(x^\ell)=\operatorname{softmax}(W_g x^\ell + b_g)\in \mathbb{R}^E.$$5. A ProGenMech circuit using approximately 719 latents attains NLL $$g(x^\ell)=\operatorname{softmax}(W_g x^\ell + b_g)\in \mathbb{R}^E.$$6, described as 58% recovery, and the PLT circuit attains NLL $$g(x^\ell)=\operatorname{softmax}(W_g x^\ell + b_g)\in \mathbb{R}^E.$$7 (Tsui et al., 14 Jun 2026).

In GLM span-infilling, all models—ProGen3, ProGenMech, and PLT—achieve NLL approximately $$g(x^\ell)=\operatorname{softmax}(W_g x^\ell + b_g)\in \mathbb{R}^E.$$8, and this is interpreted in the source as reflecting that the 112 M-parameter model struggles equally across representations. In zero-shot fitness across 8 DMS assays using the $$g(x^\ell)=\operatorname{softmax}(W_g x^\ell + b_g)\in \mathbb{R}^E.$$9 metric, original ProGen3 attains $$y^\ell(x^\ell)=\sum_{i=1}^E g_i(x^\ell)\cdot M_i(x^\ell).$$0. ProGenMech with all latents attains $$y^\ell(x^\ell)=\sum_{i=1}^E g_i(x^\ell)\cdot M_i(x^\ell).$$1, described as approximately 95% recovery, while PLT with all latents attains $$y^\ell(x^\ell)=\sum_{i=1}^E g_i(x^\ell)\cdot M_i(x^\ell).$$2. A ProGenMech circuit using approximately 256 latents attains $$y^\ell(x^\ell)=\sum_{i=1}^E g_i(x^\ell)\cdot M_i(x^\ell).$$3, described as 80% recovery, while the PLT circuit attains $$y^\ell(x^\ell)=\sum_{i=1}^E g_i(x^\ell)\cdot M_i(x^\ell).$$4.

Setting	Result
CLM, original ProGen3	NLL $$y^\ell(x^\ell)=\sum_{i=1}^E g_i(x^\ell)\cdot M_i(x^\ell).$$5
CLM, ProGenMech all latents	NLL $$y^\ell(x^\ell)=\sum_{i=1}^E g_i(x^\ell)\cdot M_i(x^\ell).$$6
CLM, PLT all latents	NLL $$y^\ell(x^\ell)=\sum_{i=1}^E g_i(x^\ell)\cdot M_i(x^\ell).$$7
CLM, ProGenMech circuit	NLL $$y^\ell(x^\ell)=\sum_{i=1}^E g_i(x^\ell)\cdot M_i(x^\ell).$$8
CLM, PLT circuit	NLL $$y^\ell(x^\ell)=\sum_{i=1}^E g_i(x^\ell)\cdot M_i(x^\ell).$$9
GLM span-infilling, all models	NLL $y^\ell$0
Zero-shot fitness, original ProGen3	$y^\ell$1
Zero-shot fitness, ProGenMech all latents	$y^\ell$2
Zero-shot fitness, PLT all latents	$y^\ell$3
Zero-shot fitness, ProGenMech circuit	$y^\ell$4
Zero-shot fitness, PLT circuit	$y^\ell$5

These findings are reported as evidence that ProGenMech outperforms local transcoder baselines in recovering ProGen3’s probability distribution and functional scoring behavior, while matching the original model’s generative distribution in span infilling tasks. A plausible implication is that ProGen3’s causal and fitness-related computations are more sensitive to cross-layer structure than its span-infilling behavior, at least under the measured replacement regime.

6. Biological motifs, latent organization, and interpretive scope

The qualitative analyses attribute biologically meaningful content to circuits recovered from ProGen3. For CLM circuits, known kinase motifs are recovered: early layers detect basic residues such as arginine, mid-layers assemble catalytic loops, and later layers isolate the HRD and DFG motifs. For zero-shot GRB2 circuits, the study reports stability and binding latents: the wild-type GRB2 circuit highlights the homodimer interface and GAB2-binding residues, the high-fitness mutation H26D upregulates a stability latent, and the low-fitness mutation Y51D downregulates a binding latent (Tsui et al., 14 Jun 2026).

Latent usage is described as following a “U-shaped” pattern, with high density in initial and final layers and sparse usage in middle layers, mirroring a progression from simple biochemical patterns to complex, task-specific features. This is presented as an organizational property of the recovered computation rather than as a property of the raw architecture alone.

These observations constrain several common simplifications. ProGen3 is not presented merely as a black-box sequence generator; within this interpretability framework, it exhibits sparse latent circuits tied to conserved sequence patterns and protein fitness landscapes. Conversely, the study does not claim perfect recovery of the native model. The reported NLL and $y^\ell$6 values show partial but not exact recovery in causal generation and fitness tasks, while span infilling remains uniformly difficult for the 112 M-parameter model. The resulting picture is therefore neither one of full transparency nor of uninterpretable opacity. Instead, ProGen3 appears as a generative pLM whose MoE computations can be approximated by sparse cross-layer latent structure with measurable fidelity and biologically grounded circuit interpretations.