Test-Time Constrained Optimization (TCO)

• Test-Time Constrained Optimization (TCO) is an inference-time adaptation framework that optimizes select test variables under explicit constraints like compute budgets, safety, and feasibility.
• It is applied across domains such as video classification, language modeling, and combinatorial optimization by updating only a subset of parameters to maintain stability and efficiency.
• Key implementations of TCO use partial parameter updates, self-supervised or reward-based objectives, and explicit constraint enforcement to balance adaptation with resource control.

Test-Time Constrained Optimization (TCO) denotes inference-time adaptation in which a pretrained system is not treated as fixed after deployment; instead, one optimizes test-time variables against self-supervised, reward-based, or prior-based objectives while respecting explicit constraints on supervision, compute, feasibility, safety, or modifiable parameters. In the surveyed literature, these optimized variables include normalization parameters and shallow layers for video models, latent solution probabilities for combinatorial optimization, a sample-specific vector for LLMs, a query embedding for multimodal retrieval, shared decoder adapters for multiview Transformers, policy mixtures for constrained meta-RL, and control sequences or trajectories for planning (Yi et al., 2023, Li et al., 5 Feb 2025, Hu et al., 18 May 2025, Uzan et al., 6 Oct 2025, Zhou et al., 4 Apr 2026, Ni et al., 29 Jan 2026, Xu et al., 5 Jun 2026).

1. Formal structure and problem classes

A canonical instantiation of TCO appears in adaptive compute allocation for reasoning models, where the objective is to maximize expected accuracy under an average compute budget: $$\boxed{ \begin{aligned} V(B) &:= \max_{\pi \in \Pi} \; E_{x \sim D}[Acc(x, \pi(x))] \ \text{s.t.}\quad & E_{x \sim D}[C(\pi(x))] \le B. \end{aligned}} \tag{1}$$ Here, the decision variable is a policy $\pi : X \to B$ that selects a test-time budget for each input, and the constraint is a global average-cost budget (Zhai et al., 16 Apr 2026).

Other TCO formulations preserve the same pattern but change the optimized variable and the constraint set. In robust video classification, TeCo adapts a pretrained classifier on unlabeled corrupted videos by minimizing prediction entropy on globally sampled clips together with attentive temporal coherence on densely sampled local clips: $L = L_{\text{ent}(I_u, f, g) + \beta\, L_{\text{att-co}(I_d, f, h).$ The optimization is constrained by partial parameter updates, mixed batch-normalization statistics,

$$\mu = \alpha \mu_s + (1 - \alpha) \mu_t, \qquad \sigma^2 = \alpha \sigma_s^2 + (1 - \alpha) \sigma_t^2,$$

and a small fixed number of test-time epochs (Yi et al., 2023).

In constrained meta-reinforcement learning, the test task is a CMDP,

$$\max_{\pi\in\Pi}\ V_r^{\mathcal{M}_{\text{test}}}(\pi) \quad \text{s.t.}\quad V_c^{\mathcal{M}_{\text{test}}}(\pi)\ge 0,$$

with the additional requirement that every deployed policy during adaptation remain feasible. The paper defines this as safe exploration and studies regret and sample complexity under that constraint (Ni et al., 29 Jan 2026).

These examples indicate that TCO is not tied to a single loss family. Its common structure is an inference-time optimization problem in which deployment constraints are part of the optimization rather than a downstream filter.

2. Optimized variables and constraint mechanisms

The optimized object in TCO is often deliberately smaller than the full parameter set. In TeCo, only shallow parameters, normalization parameters in deep layers, and attention parameters are updated, while deep convolutional weights remain fixed; this constrains deviation from pretrained behavior and improves stability (Yi et al., 2023). In SLOT, the optimized variable is a single sample-specific vector,

$\delta \in \mathbb{R}^{1 \times d},$

added to the last hidden layer,

$H' = H + \delta,$

with gradients computed only with respect to $\delta$ while the base LLM remains frozen (Hu et al., 18 May 2025).

In multimodal retrieval, Guided Query Refinement optimizes the primary retriever’s query embedding $z^{(t)}$ while keeping both retrievers and both document indexes fixed. The objective is a KL divergence between the primary distribution and a consensus distribution built from the primary and complementary retrievers: $$\mathcal{L}^{(t)} = \mathrm{KL}\!\big(p_{\mathrm{avg}}^{(t)} \,\|\, p_1(\cdot\mid z^{(t)})\big).$$ The update is applied only to the per-query embedding, not to encoder weights (Uzan et al., 6 Oct 2025).

In diffusion-based combinatorial optimization, Fast T2T operates on latent Bernoulli solution probabilities $\pi : X \to B$0, updating them by exponential gradient descent under a combined consistency and objective loss, then projecting soft outputs to feasible TSP tours or MIS sets through greedy and problem-specific post-processing (Li et al., 5 Feb 2025). In multiview 3D reconstruction, TCO updates only LoRA adapters attached to the shared decoder, while encoder and heads remain frozen; camera pose, intrinsics, and depth priors are enforced as penalty terms rather than as architectural inputs (Zhou et al., 4 Apr 2026).

Setting	Test-time variable	Constraint mechanism
Robust video classification	Shallow layers, normalization parameters, attention module	Partial updates, BN-statistics mixing, temporal coherence (Yi et al., 2023)
Sample-specific LM adaptation	$\pi : X \to B$1	Frozen base model, few optimization steps (Hu et al., 18 May 2025)
Multimodal retrieval	Query embedding $\pi : X \to B$2	Fixed indexes, small candidate pool, limited iterations (Uzan et al., 6 Oct 2025)
Diffusion CO solver	Latent solution probabilities $\pi : X \to B$3	Feasible decoding, rollout-step budgets (Li et al., 5 Feb 2025)
Multiview 3D reconstruction	Decoder LoRA parameters	Prior penalties and compatibility losses (Zhou et al., 4 Apr 2026)
Safe meta-RL	Policy choice and mixture weight $\pi : X \to B$4	Feasibility of every deployed policy (Ni et al., 29 Jan 2026)
Autonomous driving	Control sequence $\pi : X \to B$5 / trajectory $\pi : X \to B$6	Bicycle dynamics, comfort and anchor penalties (Xu et al., 5 Jun 2026)

This variability is central: TCO need not update “the model” in the full fine-tuning sense; it may instead optimize a latent, an embedding, a mixture coefficient, a decoder adapter, or an output trajectory.

3. Representative instantiations across domains

In video classification, TeCo uses global content from sparsely sampled frames for entropy minimization and local content from densely sampled consecutive frames for attentive temporal coherence regularization. It is model-agnostic across 2D CNNs, 3D CNNs, and transformers, and is evaluated on Mini Kinetics-C and Mini SSV2-C under 12 corruption types and 5 severity levels (Yi et al., 2023).

In language modeling, SLOT performs few optimization steps on the prompt itself using the prompt-stage objective

$\pi : X \to B$7

then freezes $\pi : X \to B$8 and reuses it throughout generation. The paper reports that Qwen2.5-7B with SLOT increases GSM8K accuracy from $\pi : X \to B$9 to $L = L_{\text{ent}(I_u, f, g) + \beta\, L_{\text{att-co}(I_d, f, h).$0, and DeepSeek-R1-Distill-Llama-70B with SLOT reaches $L = L_{\text{ent}(I_u, f, g) + \beta\, L_{\text{att-co}(I_d, f, h).$1 on GPQA (Hu et al., 18 May 2025).

In multimodal retrieval, GQR refines a vision-centric primary retriever with guidance from a dense text retriever. The candidate pool is the union of each retriever’s top-$L = L_{\text{ent}(I_u, f, g) + \beta\, L_{\text{att-co}(I_d, f, h).$2 results, and the refined primary query is optimized only over that pool. On ViDoRe 2, Colnomic-7B rises from NDCG@5 $L = L_{\text{ent}(I_u, f, g) + \beta\, L_{\text{att-co}(I_d, f, h).$3 to $L = L_{\text{ent}(I_u, f, g) + \beta\, L_{\text{att-co}(I_d, f, h).$4 with GQR using Jina text guidance, while avoiding the memory and latency profile of larger vision-centric baselines (Uzan et al., 6 Oct 2025).

In combinatorial optimization, Fast T2T uses consistency sampling plus consistency-based gradient search. The method treats test-time solving as optimization over the learned solution manifold under time-budget, step-budget, and feasibility constraints. On TSP and MIS, the paper reports that one-step generation and one-step gradient search can mostly outperform diffusion-based counterparts that require hundreds of steps, while achieving tens of times speedup (Li et al., 5 Feb 2025).

In generative modeling, Projected Coupled Diffusion modifies the sampling dynamics of multiple pretrained diffusion models by adding a coupled guidance term and a projection step at each diffusion step: $L = L_{\text{ent}(I_u, f, g) + \beta\, L_{\text{att-co}(I_d, f, h).$5 The reported applications are image-pair generation, PushT object manipulation, and multi-robot motion planning, with guaranteed constraint satisfaction obtained by projection construction (Luan et al., 14 Aug 2025).

In geometry-heavy vision, TCO for multiview Transformers incorporates camera poses, intrinsics, and depth as constraints on predictions. The optimization objective combines a self-supervised compatibility loss and prior penalties: $L = L_{\text{ent}(I_u, f, g) + \beta\, L_{\text{att-co}(I_d, f, h).$6 On ETH3D, 7-Scenes, and NRGBD, the paper reports that point-map distance error is reduced by more than half relative to base image-only models, and that the method outperforms retrained prior-aware feed-forward methods (Zhou et al., 4 Apr 2026).

In planning, TOAD treats the scorer of an end-to-end driving system as a learned trajectory-level reward and optimizes control sequences at test time with CEM. Across six base planners, the paper reports improvements to $L = L_{\text{ent}(I_u, f, g) + \beta\, L_{\text{att-co}(I_d, f, h).$7 PDMS on NAVSIM-v1, $L = L_{\text{ent}(I_u, f, g) + \beta\, L_{\text{att-co}(I_d, f, h).$8 EPDMS on NAVSIM-v2, and gains on the closed-loop HUGSIM benchmark (Xu et al., 5 Jun 2026).

4. Stability, efficiency, and resource control

A recurring design principle in TCO is to limit what can change at test time. TeCo reports that fully updating parameters decreases performance by $L = L_{\text{ent}(I_u, f, g) + \beta\, L_{\text{att-co}(I_d, f, h).$9–$$\mu = \alpha \mu_s + (1 - \alpha) \mu_t, \qquad \sigma^2 = \alpha \sigma_s^2 + (1 - \alpha) \sigma_t^2,$$0, whereas constraining updates to shallow layers and normalization parameters improves stability and computational efficiency. Its standard schedule is offline adaptation for one epoch, with SGD and batch size $$\mu = \alpha \mu_s + (1 - \alpha) \mu_t, \qquad \sigma^2 = \alpha \sigma_s^2 + (1 - \alpha) \sigma_t^2,$$1; $$\mu = \alpha \mu_s + (1 - \alpha) \mu_t, \qquad \sigma^2 = \alpha \sigma_s^2 + (1 - \alpha) \sigma_t^2,$$2 is reported as the best setting, and batch size $$\mu = \alpha \mu_s + (1 - \alpha) \mu_t, \qquad \sigma^2 = \alpha \sigma_s^2 + (1 - \alpha) \sigma_t^2,$$3 gives better mPC than $$\mu = \alpha \mu_s + (1 - \alpha) \mu_t, \qquad \sigma^2 = \alpha \sigma_s^2 + (1 - \alpha) \sigma_t^2,$$4 or $$\mu = \alpha \mu_s + (1 - \alpha) \mu_t, \qquad \sigma^2 = \alpha \sigma_s^2 + (1 - \alpha) \sigma_t^2,$$5 (Yi et al., 2023).

SLOT emphasizes parameter efficiency and feature caching. Because $$\mu = \alpha \mu_s + (1 - \alpha) \mu_t, \qquad \sigma^2 = \alpha \sigma_s^2 + (1 - \alpha) \sigma_t^2,$$6 is applied only to the final hidden layer, the last hidden features $$\mu = \alpha \mu_s + (1 - \alpha) \mu_t, \qquad \sigma^2 = \alpha \sigma_s^2 + (1 - \alpha) \sigma_t^2,$$7 are cached, and each optimization step requires only forward and backward through the LM head. On 30 GSM8K questions with Qwen2.5-7B, the reported runtime changes from $$\mu = \alpha \mu_s + (1 - \alpha) \mu_t, \qquad \sigma^2 = \alpha \sigma_s^2 + (1 - \alpha) \sigma_t^2,$$8 s at $$\mu = \alpha \mu_s + (1 - \alpha) \mu_t, \qquad \sigma^2 = \alpha \sigma_s^2 + (1 - \alpha) \sigma_t^2,$$9 to $$\max_{\pi\in\Pi}\ V_r^{\mathcal{M}_{\text{test}}}(\pi) \quad \text{s.t.}\quad V_c^{\mathcal{M}_{\text{test}}}(\pi)\ge 0,$$0 s at $$\max_{\pi\in\Pi}\ V_r^{\mathcal{M}_{\text{test}}}(\pi) \quad \text{s.t.}\quad V_c^{\mathcal{M}_{\text{test}}}(\pi)\ge 0,$$1, an increase of $$\max_{\pi\in\Pi}\ V_r^{\mathcal{M}_{\text{test}}}(\pi) \quad \text{s.t.}\quad V_c^{\mathcal{M}_{\text{test}}}(\pi)\ge 0,$$2 s or about $$\max_{\pi\in\Pi}\ V_r^{\mathcal{M}_{\text{test}}}(\pi) \quad \text{s.t.}\quad V_c^{\mathcal{M}_{\text{test}}}(\pi)\ge 0,$$3 (Hu et al., 18 May 2025).

GQR bounds its cost by restricting optimization to a small candidate pool and a fixed number of iterations. The paper reports additional latency of $$\max_{\pi\in\Pi}\ V_r^{\mathcal{M}_{\text{test}}}(\pi) \quad \text{s.t.}\quad V_c^{\mathcal{M}_{\text{test}}}(\pi)\ge 0,$$4–$$\max_{\pi\in\Pi}\ V_r^{\mathcal{M}_{\text{test}}}(\pi) \quad \text{s.t.}\quad V_c^{\mathcal{M}_{\text{test}}}(\pi)\ge 0,$$5 ms with single-vector text refiners and $$\max_{\pi\in\Pi}\ V_r^{\mathcal{M}_{\text{test}}}(\pi) \quad \text{s.t.}\quad V_c^{\mathcal{M}_{\text{test}}}(\pi)\ge 0,$$6–$$\max_{\pi\in\Pi}\ V_r^{\mathcal{M}_{\text{test}}}(\pi) \quad \text{s.t.}\quad V_c^{\mathcal{M}_{\text{test}}}(\pi)\ge 0,$$7 ms with multi-vector text refiners, while Colnomic-7B + GQR (Linq) reaches NDCG@5 $$\max_{\pi\in\Pi}\ V_r^{\mathcal{M}_{\text{test}}}(\pi) \quad \text{s.t.}\quad V_c^{\mathcal{M}_{\text{test}}}(\pi)\ge 0,$$8 at $$\max_{\pi\in\Pi}\ V_r^{\mathcal{M}_{\text{test}}}(\pi) \quad \text{s.t.}\quad V_c^{\mathcal{M}_{\text{test}}}(\pi)\ge 0,$$9 ms/query and Colnomic-7B + GQR (Jina text) reaches $\delta \in \mathbb{R}^{1 \times d},$0 at $\delta \in \mathbb{R}^{1 \times d},$1 ms/query; the stronger Llama-Nemo baseline without refinement is reported at $\delta \in \mathbb{R}^{1 \times d},$2 ms/query (Uzan et al., 6 Oct 2025).

Fast T2T exposes the compute–quality trade-off directly through the numbers of consistency sampling steps $\delta \in \mathbb{R}^{1 \times d},$3 and gradient-search iterations $\delta \in \mathbb{R}^{1 \times d},$4. The paper reports that $\delta \in \mathbb{R}^{1 \times d},$5, $\delta \in \mathbb{R}^{1 \times d},$6 already yields strong solutions, while larger $\delta \in \mathbb{R}^{1 \times d},$7 and $\delta \in \mathbb{R}^{1 \times d},$8 improve quality with diminishing returns. This suggests a characteristic TCO pattern: small, explicit test-time budgets are treated as first-class optimization constraints rather than as fixed inference hyperparameters (Li et al., 5 Feb 2025).

Adaptive compute allocation for reasoning LLMs makes this budgeted view explicit. For a fixed dual variable $\delta \in \mathbb{R}^{1 \times d},$9, the oracle action is

$H' = H + \delta,$0

The parameter $H' = H + \delta,$1 functions as a compute price, so the optimizer chooses more expensive budgets only when the marginal accuracy gain justifies their cost (Zhai et al., 16 Apr 2026).

5. Guarantees, optimality, and empirical behavior

The strongest formal guarantees currently appear in settings where the constrained object is simple enough to admit decomposition or concentration analysis. In adaptive compute allocation, the Lagrangian oracle induces an average cost

$H' = H + \delta,$2

and the paper proves that, under deterministic tie-breaking, $H' = H + \delta,$3 is non-increasing in $H' = H + \delta,$4. This monotonicity yields exact budget targeting by binary search and, together with the LP formulation, supports the strong-duality statement

$H' = H + \delta,$5

The same paper gives an imitation-regret bound,

$H' = H + \delta,$6

showing that task-level regret of the learned routing policy is bounded by imitation error times the worst-case per-instance gap (Zhai et al., 16 Apr 2026).

Constrained meta-RL provides a second line of theory. The test-time algorithm mixes a simultaneously feasible safe policy $H' = H + \delta,$7 with candidate policies,

$H' = H + \delta,$8

and updates $H' = H + \delta,$9 only when confidence bounds preserve feasibility. The paper proves safe exploration—every executed $\delta$0 is feasible—and gives a reward-regret upper bound

$\delta$1

along with a matching lower bound of order

$\delta$2

for learning an $\delta$3-optimal feasible policy (Ni et al., 29 Jan 2026).

In other domains, theory is more limited and the evidence is primarily empirical. Projected Coupled Diffusion relies on the fact that projection enforces feasibility by construction, but the paper does not provide new convergence guarantees for the full nonconvex setting with neural scores and coupling costs. TOAD similarly relies on the empirical quality of the learned scorer and reports that too many CEM iterations can overfit to scorer imperfections, a reward-hacking effect rather than a formal optimization guarantee (Luan et al., 14 Aug 2025, Xu et al., 5 Jun 2026).

Empirically, TCO systems often show a characteristic nonuniform gain profile: the greatest benefit appears where the base model is neither trivial nor irrecoverable. Adaptive compute allocation reports the largest gains on intermediate budgets and on “responsive” questions; TOAD reports larger gains for weaker planners or harder evaluation settings such as NAVSIM-v2; TeCo reports larger relative gains on temporally sensitive corruptions and datasets (Zhai et al., 16 Apr 2026, Xu et al., 5 Jun 2026, Yi et al., 2023).

6. Limitations, misconceptions, and research directions

A common misconception is that TCO necessarily means full-model fine-tuning at inference. The surveyed works show otherwise: some optimize only a sample-specific vector, some only a query embedding, some only decoder LoRA parameters, some only a mixture coefficient, and some optimize outputs directly through search rather than updating weights at all (Hu et al., 18 May 2025, Uzan et al., 6 Oct 2025, Zhou et al., 4 Apr 2026, Ni et al., 29 Jan 2026, Xu et al., 5 Jun 2026).

A second misconception is that “constraints” are synonymous with compute budgets. In practice, the constraint set varies widely: temporal smoothness and mixed normalization statistics in video adaptation, feasibility of TSP tours and MIS sets in neural combinatorial optimization, Hamiltonian-cycle or independent-set decoding, velocity limits and obstacle avoidance in trajectory generation, discounted safety constraints in CMDPs, and camera, intrinsic, or depth priors in 3D vision (Yi et al., 2023, Li et al., 5 Feb 2025, Luan et al., 14 Aug 2025, Ni et al., 29 Jan 2026, Zhou et al., 4 Apr 2026).

The main limitations are correspondingly diverse. SLOT has no explicit alignment or safety regularization and depends on prompt quality; GQR adds latency and requires a small in-domain development set for hyperparameter tuning; constrained meta-RL relies on simultaneous Slater’s condition and simulator access during training; TOAD can reward-hack weak scorers and therefore depends on a scorer that generalizes off its training proposal set; TCO for multiview Transformers incurs substantial test-time overhead, rising from about $\delta$4 s per scene at $\delta$5 steps to about $\delta$6 s at $\delta$7 steps on ETH3D in the reported setup (Hu et al., 18 May 2025, Uzan et al., 6 Oct 2025, Ni et al., 29 Jan 2026, Xu et al., 5 Jun 2026, Zhou et al., 4 Apr 2026).

Several design principles recur across the literature. TeCo explicitly identifies exploiting domain structure, balancing adaptation and conservation, combining multiple self-supervised signals, constrained normalization adaptation, and partial efficient adaptation as general principles emerging from its video setting (Yi et al., 2023). Adaptive compute allocation identifies continuous budget spaces, multi-objective and multi-constraint formulations, and stronger robustness under distribution shift as open problems (Zhai et al., 16 Apr 2026). OptPO suggests another direction: treating rollout allocation itself as a sequential hypothesis-testing problem so that every test-time sample serves both evaluation and learning (Wang et al., 2 Dec 2025).

The acronym itself is overloaded. In the HPC paper “Wattlytics,” TCO denotes multi-year Total Cost of Ownership rather than test-time constrained optimization, and the system studies budget, power, performance, and uncertainty trade-offs for GPU clusters (Afzal et al., 9 Apr 2026). Within machine learning, however, the surveyed literature uses TCO to denote inference-time optimization under explicit constraints, and the body of work now spans video robustness, retrieval, language modeling, combinatorial optimization, diffusion sampling, 3D reconstruction, safe reinforcement learning, and autonomous driving (Yi et al., 2023, Uzan et al., 6 Oct 2025, Hu et al., 18 May 2025, Li et al., 5 Feb 2025, Luan et al., 14 Aug 2025, Zhou et al., 4 Apr 2026, Ni et al., 29 Jan 2026, Xu et al., 5 Jun 2026).