Learnable Regression Tokens

• Learnable regression tokens are trainable latent representations optimized to aggregate task-relevant evidence for continuous prediction.
• Architectural variants like RELISH, ViGT, and TokenGS showcase different methods of integrating tokens into regression, such as iterative refinement and cross-modal fusion.
• They act as dedicated latent bottlenecks that replace traditional pooling, improving prediction accuracy and decoupling output structure from input resolution.

Searching arXiv for the cited papers and related context. Learnable regression tokens are trainable latent objects used for continuous prediction tasks rather than textual generation or direct regression from pooled features. In RELISH, this role is played by a learned latent task state that is refined by cross-attention over frozen LLM representations before linear scalar readout; in ViGT, by a randomly initialized transformer token $[REG]$ used as the sole source for temporal boundary prediction; and in TokenGS, by decoder-side learnable Gaussian tokens that each decode into a chunk of 3D Gaussians (Su et al., 1 Apr 2026, Li et al., 2023, Ren et al., 16 Apr 2026). Across these settings, the common pattern is a dedicated latent bottleneck for regression, but the implementation differs substantially by modality, output structure, and placement within the network.

1. Core concept and architectural variants

The term “learnable regression token” denotes a trainable latent representation whose function is to aggregate task-relevant evidence and support direct prediction of a continuous target. In the cited works, this object is not merely a generic embedding. It is explicitly optimized end-to-end for regression and is structurally separated from ordinary input tokens or pooled hidden states.

A compact comparison clarifies the design space:

Work	Latent object	Regression role
RELISH	One learned latent task state $r^{(0)}$	Queries frozen LLM token states and predicts a scalar
ViGT	Learnable token $[REG]$	Aggregates video-query context and predicts temporal boundary
TokenGS	Fixed set of learnable Gaussian tokens	Cross-attends to image features and decodes 3D Gaussians

The three designs differ along at least three axes. First, the latent object may be a single vector or a token set. RELISH uses one learned latent state vector, whereas TokenGS uses a fixed number $N_t$ of decoder tokens. Second, the latent object may reside in a prediction head, in a joint transformer sequence, or in a decoder. Third, the object may produce a scalar, a temporal segment, or a set of 3D primitives. This suggests that “learnable regression token” is best understood as an architectural motif rather than a single canonical mechanism.

2. RELISH and the latent iterative state head

RELISH presents the most explicit formulation of a learnable regression token for LLM-based scalar prediction. It starts from a frozen decoder-only LLM backbone that produces token-level hidden states

$H(x)\in\mathbb{R}^{S\times d}.$

These hidden states may be projected into a smaller head dimension,

$X(x)=H(x)W,\qquad W\in\mathbb{R}^{d\times d_h},$

so that $X(x)\in\mathbb{R}^{S\times d_h}$ serves as the token memory for the regression head (Su et al., 1 Apr 2026).

The central object is a learned latent task state

$r^{(0)}\in\mathbb{R}^{d_h},$

trained end-to-end. The paper characterizes this state as the key “token-like” object: it is not a textual token, but a trainable query vector that plays the role of an internal regression token. At each refinement step $i=1,\dots,L$, the latent state is updated through Transformer-style cross-attention, with the latent state as query and token representations as keys and values: $$\tilde r^{(i)} = LN\big(r^{(i-1)}+ \mathrm{MHA}\!\big(r^{(i-1)},X(x),X(x)\big)\big),$$

$r^{(0)}$0

For head $r^{(0)}$1 out of $r^{(0)}$2,

$r^{(0)}$3

$r^{(0)}$4

The architectural novelty is iterative refinement. Rather than using a single pooling operation, RELISH applies

$r^{(0)}$5

with residual cross-attention and FFN blocks using post-layer normalization. In the authors’ comparison, $r^{(0)}$6 behaves like strong attention pooling, while $r^{(0)}$7 can substantially improve performance. After $r^{(0)}$8 rounds, the final latent state is mapped to a standardized scalar prediction by a linear regressor,

$r^{(0)}$9

Targets are standardized as

$[REG]$0

and training uses Huber loss.

Conceptually, RELISH is a predictive-head method, but it departs from standard “pool then regress” heads. The learned latent state behaves like a learnable regression token or latent prompt/query token in spirit, yet it is not a soft prompt inserted into the LLM input, is not autoregressive, and does not generate numeric text. The latent state lives in the head and attends to frozen LLM outputs.

3. Position within LLM regression methodology

RELISH situates learnable regression tokens within three major LLM regression families: autoregressive decoding, regression-aware inference, and predictive heads (Su et al., 1 Apr 2026). This taxonomy is important because the contribution is not merely a new head, but a redefinition of how information is extracted from frozen LLM representations for continuous targets.

In autoregressive decoding, the model is prompted to generate the target as text and the output is parsed numerically: $[REG]$1 Training uses token-level cross-entropy,

$[REG]$2

RELISH differs by not generating text at all. It directly regresses from hidden states, thereby avoiding numeric-string brittleness while preserving single-pass inference.

In regression-aware inference, methods such as RAIL and RAFT compute decision rules from the output distribution: $[REG]$3 For MSE this becomes a posterior mean approximation over a candidate grid, and RAFT trains with

$[REG]$4

RELISH differs by avoiding multi-sample or candidate-based inference. It remains single-pass at inference while learning a regression-specific readout from internal representations.

In predictive heads, regression is obtained from a pooled representation,

$[REG]$5

RELISH belongs to this family, but replaces rigid pooling or token indexing with an iteratively refined latent query. The functional role of the learnable regression token is therefore to make the extraction function $[REG]$6 adaptive and recurrent rather than fixed.

4. Extensions beyond text regression

ViGT and TokenGS demonstrate that learnable regression tokens are not specific to LLM scalar regression, but also appear in video grounding and feed-forward 3D Gaussian prediction.

In ViGT, the learnable token is denoted $[REG]$7 and corresponds to $[REG]$8. It is randomly initialized, optimized with the full model, and unrelated to video or query at initialization. The model is proposal-free and predicts the target segment $[REG]$9 directly. Video features $N_t$0 are extracted using C3D or I3D, query features $N_t$1 using GloVe, and both modalities are mapped into a joint feature space by a shared Feature Encoder based on the transformer encoder in QANet. Cross-Modal Co-Attention is then applied in both directions. The transformer input is

$N_t$2

followed by a transformer stack with $N_t$3, best feature dimension $N_t$4, and dropout $N_t$5 in multi-head self-attention. The output embedding $N_t$6 is passed to a boundary regression head with 3 fully-connected layers and sigmoid activation to predict normalized center and width $N_t$7. At inference time, only this boundary regression head is used for prediction (Li et al., 2023).

The paper’s interpretation is that $N_t$8 avoids bias toward the original video or query modalities because it is not derived from either one, simultaneously interacts with all video and query features, and retains informative cues from a global viewpoint of contextual aggregation. Rather than regressing directly from fused multimodal features, ViGT uses the token to learn the relation between video and query in a way that is more suitable for boundary regression.

TokenGS moves the idea into 3D reconstruction. Standard feed-forward 3DGS predictors often regress Gaussian means as depths along camera rays and tie output count to image resolution and number of views. TokenGS replaces that design with an encoder-decoder Transformer that uses a fixed number $N_t$9 of learnable 3DGS tokens,

$H(x)\in\mathbb{R}^{S\times d}.$0

which cross-attend to image tokens, self-attend among themselves, pass through an MLP, and produce output token embeddings $H(x)\in\mathbb{R}^{S\times d}.$1. Each output token then regresses multiple Gaussian primitives; in the implementation, each token decodes to 64 Gaussians. The predicted Gaussian set is

$H(x)\in\mathbb{R}^{S\times d}.$2

with mean $H(x)\in\mathbb{R}^{S\times d}.$3, color $H(x)\in\mathbb{R}^{S\times d}.$4, scale $H(x)\in\mathbb{R}^{S\times d}.$5, opacity $H(x)\in\mathbb{R}^{S\times d}.$6, and rotation $H(x)\in\mathbb{R}^{S\times d}.$7. Instead of predicting ray depth, TokenGS directly regresses global 3D mean coordinates $H(x)\in\mathbb{R}^{S\times d}.$8, using

$H(x)\in\mathbb{R}^{S\times d}.$9

for XYZ coordinates (Ren et al., 16 Apr 2026).

The result is that output size becomes a hyperparameter rather than a consequence of input tensor size. This decouples reconstruction capacity from image resolution and number of input views.

5. Objectives, supervision, and empirical behavior

The training objectives differ by task, but all three works optimize the learnable regression token or tokens directly for continuous prediction.

RELISH trains on standardized targets and optimizes Huber loss. Its reported setup uses four decoder-only LLMs—Llama 3.1 8B Instruct, Qwen3 8B, Gemma 3 27B Instruct, and Qwen3 32B—across five regression datasets: STS-B, SICK-R, and WMT 2020 QE on English–Chinese, Russian–English, and Sinhala–English. Reported metrics are Pearson $X(x)=H(x)W,\qquad W\in\mathbb{R}^{d\times d_h},$0, Spearman $X(x)=H(x)W,\qquad W\in\mathbb{R}^{d\times d_h},$1, and range-normalized RMSE. RELISH itself uses $X(x)=H(x)W,\qquad W\in\mathbb{R}^{d\times d_h},$2 attention heads, $X(x)=H(x)W,\qquad W\in\mathbb{R}^{d\times d_h},$3, and $X(x)=H(x)W,\qquad W\in\mathbb{R}^{d\times d_h},$4 refinement steps. Macro-averaged across five datasets, four LLMs, and three runs, it reports Pearson $X(x)=H(x)W,\qquad W\in\mathbb{R}^{d\times d_h},$5, Spearman $X(x)=H(x)W,\qquad W\in\mathbb{R}^{d\times d_h},$6, and NRMSE $X(x)=H(x)W,\qquad W\in\mathbb{R}^{d\times d_h},$7, compared with the best prior baseline RAFT at $X(x)=H(x)W,\qquad W\in\mathbb{R}^{d\times d_h},$8, $X(x)=H(x)W,\qquad W\in\mathbb{R}^{d\times d_h},$9, and NRMSE $X(x)\in\mathbb{R}^{S\times d_h}$0. The paper states improvements of $X(x)\in\mathbb{R}^{S\times d_h}$1 in Pearson, $X(x)\in\mathbb{R}^{S\times d_h}$2 in Spearman, and $X(x)\in\mathbb{R}^{S\times d_h}$3 in NRMSE, while using approximately $X(x)\in\mathbb{R}^{S\times d_h}$4M–$X(x)\in\mathbb{R}^{S\times d_h}$5M trainable parameters, corresponding to $X(x)\in\mathbb{R}^{S\times d_h}$6–$X(x)\in\mathbb{R}^{S\times d_h}$7 overhead, compared with $X(x)\in\mathbb{R}^{S\times d_h}$8M–$X(x)\in\mathbb{R}^{S\times d_h}$9M and $r^{(0)}\in\mathbb{R}^{d_h},$0–$r^{(0)}\in\mathbb{R}^{d_h},$1 for LoRA or RAFT (Su et al., 1 Apr 2026).

ViGT combines regression and auxiliary supervision: $r^{(0)}\in\mathbb{R}^{d_h},$2 The regression loss combines Smooth-$r^{(0)}\in\mathbb{R}^{d_h},$3 and generalized IoU,

$r^{(0)}\in\mathbb{R}^{d_h},$4

with $r^{(0)}\in\mathbb{R}^{d_h},$5 and $r^{(0)}\in\mathbb{R}^{d_h},$6, while the classification head predicts timestamp-wise foreground probabilities and is trained with binary cross-entropy. In the ANet-Captions ablation table, full ViGT reports $r^{(0)}\in\mathbb{R}^{d_h},$7, whereas the variant without the token reports $r^{(0)}\in\mathbb{R}^{d_h},$8. The paper interprets this drop, especially at [email protected] and [email protected], as evidence that the token helps learn more informative cross-modal relations (Li et al., 2023).

TokenGS is trained from rendering supervision only, using

$r^{(0)}\in\mathbb{R}^{d_h},$9

together with a visibility regularizer that addresses the zero-gradient problem for Gaussians outside all supervision frusta. The reported visibility loss removes floaters, improves geometry, and gives about $i=1,\dots,L$0 PSNR gain in an ablation. The paper further reports state-of-the-art feed-forward reconstruction on static and dynamic scenes, competitive or better PSNR/SSIM/LPIPS with fewer Gaussians, and gains from context extension and token tuning across RE10K, DL3DV, and Kubric (Ren et al., 16 Apr 2026).

6. Conceptual distinctions and recurrent misunderstandings

A frequent misunderstanding is to equate learnable regression tokens with soft prompts or autoregressive output tokens. RELISH explicitly distinguishes its latent state from both. The latent state is not prepended into the LLM input, is not used for text generation, and is optimized as a dedicated regression head over frozen token representations (Su et al., 1 Apr 2026).

A second misunderstanding is to assume that learnable regression tokens must always be ordinary discrete tokens in the model input. ViGT uses an actual transformer token $i=1,\dots,L$1 concatenated with encoded query and video features, but RELISH uses a single latent state vector in the head, and TokenGS uses decoder-side scene tokens that are not tied to any input pixel or ray. This suggests that the essential property is not token discreteness, but the presence of a trainable latent carrier for regression signals.

A third misunderstanding is that the token simply duplicates multimodal fusion or pooling. ViGT argues against this directly: the token is unrelated to the video or the query at initialization, simultaneously interacts with all video and query features, and avoids bias toward the original modalities. Its qualitative analysis further reports that $i=1,\dots,L$2 attends progressively to more discriminative frames across transformer layers, whereas Cross-Modal Co-Attention often attends broadly or evenly (Li et al., 2023).

In TokenGS, the distinction is between pixel- or ray-aligned regression and latent-token regression. The paper argues that regressing Gaussian means as depths along rays is too restrictive, couples output count to input sampling density, and is less robust to pose noise and multiview inconsistencies. Learnable Gaussian tokens instead support direct regression of 3D centers in scene space, fixed-capacity decoding, and test-time optimization in token space rather than direct optimization of all Gaussian parameters (Ren et al., 16 Apr 2026).

Taken together, these works suggest a shared principle: learnable regression tokens act as task-specific latent slots that replace rigid pooling, direct multimodal fusion, or geometry tied to sampling coordinates. The precise benefit, however, is domain-dependent. In RELISH the key gain is iterative latent summarization for scalar prediction; in ViGT it is global cross-modal aggregation for temporal boundary localization; in TokenGS it is decoupling primitive prediction from pixels and views while preserving token-space optimization.