--- title: Compute Aligned Training (CAT) url: https://www.emergentmind.com/topics/compute-aligned-training-cat type: topic --- # Compute Aligned Training (CAT) Searching arXiv for the requested topic and closely related work. Searching for "Compute Aligned Training" on arXiv. Compute Aligned Training (CAT) is a post-training framework for generative models in which the optimization target is the distribution induced by the **deployment-time inference procedure**, rather than the raw one-sample base policy. In the formulation introduced in "Compute Aligned Training: Optimizing for Test Time Inference" [2604.24957], CAT addresses a mismatch between standard supervised fine-tuning (SFT) or reinforcement learning (RL)—which optimize individual samples under a base policy—and modern test-time strategies such as Pass@\(\!N\), majority vote, filtering, reranking, and Best-of-\(N\), which consume multiple samples and return a single final output. The framework therefore treats inference as an operator on the base policy and derives training losses that maximize performance **after** that operator is applied. ## 1. Operator-level view of inference CAT begins from a distinction between the **base policy** \[ \pi_\theta(\cdot \mid x) \] and the **effective deployment policy** \[ \tilde{\pi}_\theta(y \mid x)=\mathcal{T}(\pi_\theta(\cdot \mid x),\phi)(y), \] where \(\mathcal{T}\) is a test-time strategy (TTS) and \(\phi\) denotes strategy hyperparameters such as sample budget \(N\), threshold \(k\), or verifier settings [2604.24957]. The central claim is that evaluation increasingly occurs under \(\tilde{\pi}_\theta\), not under \(\pi_\theta\). Standard training is therefore misaligned whenever deployment uses search, aggregation, or filtering. The practical motivation is clearest in search-based settings. Under Pass@\(\!N\), once the correct answer already appears with moderate probability, further pushing that probability toward \(1\) can have little marginal value, because success is already likely across \(N\) draws. Under majority vote, the relevant objective is not that the correct answer dominate the full probability mass, but that it exceed the strongest rival often enough to win plurality. CAT formalizes this intuition as a **marginal-utility reweighting** problem: updates should be concentrated where they matter for the final system returned by the chosen inference operator. This operator view also reframes “test-time scaling” as an optimization target rather than merely an evaluation trick. A plausible implication is that CAT treats extra inference compute as part of the model-definition pipeline: the deployable system is the composition of a policy and a TTS operator, not the policy alone. ## 2. Exact objectives and diagonal approximation For SFT, CAT replaces ordinary cross-entropy on the base policy with negative log-likelihood under the transformed policy: \[ \mathcal L_{\mathrm{CAT}}(\theta) = -\mathbb E_{(x,y^*)\sim\mathcal D}\big[\log \tilde{\pi}_\theta(y^*\mid x)\big]. \] If \[ p=\pi_\theta(y^*\mid x), \qquad \tilde p=\tilde{\pi}_\theta(y^*\mid x), \] the exact gradient is \[ \nabla_\theta \mathcal L_{\mathrm{CAT}} = -\frac{1}{\tilde p} \sum_{y'\in\mathcal Y} \frac{\partial \tilde p}{\partial \pi_\theta(y'\mid x)} \nabla_\theta \pi_\theta(y'\mid x). \] For RL, the CAT objective is expected reward under the transformed policy: \[ J_{\mathrm{CAT}}(\theta) = \mathbb E_{y\sim \tilde{\pi}_\theta(\cdot\mid x)}[R(y\mid x)] = \sum_{y\in\mathcal Y}\tilde{\pi}_\theta(y\mid x)R(y\mid x). \] Its exact gradient weights each sample by its effect on deployment-time utility rather than by one-sample utility alone [2604.24957]. Because the full Jacobian over \(\mathcal Y\) is generally impractical, CAT uses a **diagonal approximation** in which \(\tilde p\) is treated mainly as a function of the target’s own base probability \(p\), ignoring off-diagonal dependence on rivals. Under this approximation, SFT reduces to a scalar rescaling of the ordinary cross-entropy gradient: \[ \nabla_\theta \mathcal L_{\mathrm{CAT}} \approx w(p,\phi)\,\nabla_\theta \mathcal L_{\mathrm{CE}}, \qquad w(p,\phi)=\frac{p}{\tilde p}\frac{\partial \tilde p}{\partial p}. \] RL analogously becomes \[ \nabla_\theta J_{\mathrm{CAT}} \approx \mathbb E_{y\sim\pi_\theta} \left[ R(y\mid x)\frac{\partial \tilde p}{\partial p}\nabla_\theta\log\pi_\theta(y\mid x) \right]. \] The paper writes the exact gradient as \(g_{\mathrm{exact}}=g_{\mathrm{diag}}+\epsilon_{\mathrm{vec}}\) and proves a sufficient descent condition: \[ \|\epsilon_{\mathrm{vec}}\|<\|g_{\mathrm{diag}}\|. \] It further distinguishes **orthogonal** strategies, for which success depends only on the target probability and the approximation is exact, from **competitive** strategies, in which increasing rival probability harms the target and the diagonal approximation is conservative. Pass@\(\!N\) is orthogonal; majority vote is analyzed via a threshold relaxation rather than exact plurality [2604.24957]. ## 3. Compute-aligned supervised fine-tuning In CAT-SFT, the training target is not “maximize the probability of this demonstration under one sample,” but “maximize the probability that the deployed inference strategy returns the correct answer.” For reasoning tasks with chain-of-thought, the relevant marginal answer probability is \[ \pi_\theta(y\mid x)=\sum_{z\in\mathcal Z}\pi_\theta(y,z\mid x), \] but this is intractable. In practice, CAT uses the probability of the provided reasoning trace \((z^*,y^*)\) as a lower-bound proxy for \(p\), while retaining the operator-aware weight \(w(p,\phi)\) [2604.24957]. For Pass@\(\!N\), the transformed success probability is \[ \tilde p = 1-(1-p)^N, \] yielding the CAT-SFT weight \[ w_{\mathrm{pass,SFT}}(p,N) = \frac{Np(1-p)^{N-1}}{1-(1-p)^N}. \] This suppresses gradients on examples that are already likely to succeed within \(N\) trials. The implementation uses a log-space approximation when \(p<10^{-4}\): \[ \mathcal L_{\mathrm{Pass}@N}=\ell_{\mathrm{seq}}-\log N. \] For majority vote, the paper uses a threshold relaxation: success occurs when the correct answer appears at least \(k\) times in \(N\) samples, \[ \tilde p_{\mathrm{maj}} = \sum_{i=k}^{N}\binom Ni p^i(1-p)^{N-i}. \] The corresponding SFT weight is \[ w_{\mathrm{maj,SFT}}(p,N,k) = \frac{k\binom Nk p^k(1-p)^{N-k}} {\sum_{i=k}^{N}\binom Ni p^i(1-p)^{N-i}}. \] This concentrates learning near the consensus boundary. The paper also analyzes its variance: as \(p\to 0\), \(w_{\mathrm{maj}}\to k\), implying variance inflation that scales approximately as \(N^2/4\) for simple majority. That analysis is used to explain why majority-vote SFT exhibits a bias–variance tradeoff and why moderate \(N\) can outperform very large \(N\) [2604.24957]. The paper additionally sketches a token-level contrastive auxiliary loss for majority vote based on the token margin \[ \delta_t=z_t[y_t^*]-\max_{v\neq y_t^*}z_t[v], \] with \[ \mathcal L_{\mathrm{Contrast}} = \frac1T\sum_{t=1}^{T}\max(0,\eta-\delta_t). \] This is not the main CAT algorithm, but it reflects the same principle: majority-vote alignment depends on controlling strong rivals, not only increasing the correct trace. ## 4. Compute-aligned reinforcement learning CAT-RL optimizes reward after the deployment operator rather than reward of a single rollout. Under the diagonal approximation, the generic policy-gradient form becomes \[ \nabla_\theta J_{\mathrm{CAT}} \approx \mathbb E_{y\sim\pi_\theta} \left[ \tilde R(y\mid x)\nabla_\theta\log\pi_\theta(y\mid x) \right], \qquad \tilde R(y\mid x)=R(y\mid x)\frac{\partial \tilde p}{\partial p}. \] Thus CAT-RL can be interpreted as a strategy-specific per-sample modulation of reward or advantage [2604.24957]. For Pass@\(\!N\) with binary reward, \[ \tilde p=1-(1-p)^N, \qquad w_{\mathrm{pass,RL}}=N(1-p)^{N-1}. \] The paper reports that this exact RL-aligned weight can have severe variance at larger \(N\), especially on hard examples where \(p\ll 1/N\). To stabilize GRPO/PPO, it often uses a normalized “log-weighted” variant, \[ w_{\mathrm{Norm}} = \frac{p}{1-(1-p)^N}N(1-p)^{N-1}, \] which is biased but substantially more stable. For majority vote RL, the threshold-relaxed weight is \[ w_{\mathrm{maj,RL}}(p,N,k) = N\binom{N-1}{k-1}p^{k-1}(1-p)^{N-k}. \] Unlike the SFT weight, this acts as a boundary-localized spotlight: it is largest where a marginal increase in probability is most likely to flip consensus. In RL, \(k\) need not be fixed; the paper estimates it from rollout batches using the empirical frequency of the strongest rival. For Best-of-\(N\) RL, if \(P_{