Regression via Transformer Classification (RvTC)

• Regression via Transformer Classification (RvTC) is a design pattern that reframes continuous prediction through discrete classification signals.
• It encompasses methods such as bin-based regression, digit-token prediction, and trend-conditioned residual forecasting using Transformer architectures.
• Empirical results in image, molecular, and time series domains demonstrate enhanced optimization, interpretability, and performance.

Searching arXiv for papers on Regression via Transformer-Based Classification (RvTC) and closely related formulations. Regression via Transformer-Based Classification (RvTC) denotes a family of methods in which a Transformer does not treat continuous prediction as a purely direct scalar regression problem, but instead derives the prediction from classification-style outputs, or uses classification signals to guide a continuous regressor. In the literature summarized here, RvTC appears in three closely related forms: bin-based regression in multimodal image assessment, where a continuous score is recovered as the expectation of bin centers; digit-token prediction in sequence modeling, where real-valued properties are encoded as token sequences and learned with cross-entropy; and trend-conditioned residual forecasting in time series, where a classification branch predicts coarse direction and a regression branch predicts deviation magnitude around the latest observation (Jennings et al., 20 Jul 2025, Born et al., 2022, Yao et al., 12 Nov 2025). A complementary theoretical treatment places Transformer-based regression in a distribution regression framework, formalizes an attention operator as an information-preserving embedding, and derives approximation and generalization results that directly translate to RvTC-style estimators (Liu et al., 28 Jun 2026).

1. Conceptual definition and problem framing

RvTC reframes continuous prediction through a classification interface. In the image-based formulation, the target $y \in \mathbb{R}$ is assigned to one of $B$ bins covering $[y_{\min}, y_{\max}]$, the model predicts logits $z \in \mathbb{R}^B$, and the final scalar prediction is the expectation over bin centers,

$\hat{y} = \sum_{b=1}^B p_b\, c_b,$

with $p_b$ given by softmax (Jennings et al., 20 Jul 2025). In the molecular and protein setting, the same principle is pushed to token granularity: a real-valued property is converted into a sequence of digit tokens $t_{v,p}$, and regression is optimized as token-level cross-entropy over the property sequence rather than with MSE or MAE (Born et al., 2022). In CaReTS, the formulation is hybrid: classification is not over the full target space, but over macro-level trend classes, while regression remains continuous over deviations from the latest observation $x_t$ (Yao et al., 12 Nov 2025).

A common misconception is that RvTC is identical to coarse discretization of the target. The published formulations are more heterogeneous. The bin-based image model uses uniform bins and expectation decoding; the Regression Transformer avoids coarse binning by representing numerals digit by digit; and CaReTS uses classification only for coarse direction, explicitly “avoiding quantization of full target space” while leaving magnitude estimation continuous (Jennings et al., 20 Jul 2025, Born et al., 2022, Yao et al., 12 Nov 2025).

The motivation is also domain-dependent. In image assessment, RvTC is positioned against vocabulary-constrained multimodal regression and distributional modeling; in molecular language modeling, it is used to unify regression and controllable generation in one XLNet backbone; in time series forecasting, it is used to disentangle “macro-level trend” from “micro-level deviations” and to improve interpretability (Jennings et al., 20 Jul 2025, Born et al., 2022, Yao et al., 12 Nov 2025).

2. Mathematical formulations

Three formulations recur in the literature.

First, the bin-based estimator used for image-based regression defines uniform bin width

$\Delta = \frac{y_{\max} - y_{\min}}{B},$

bin centers

$$c_b = y_{\min} + (b - 0.5)\Delta,\quad b = 1,\dots,B,$$

hard nearest-bin labels, and cross-entropy loss

$B$0

The scalar estimate is then obtained by expectation over bin centers (Jennings et al., 20 Jul 2025).

Second, the Regression Transformer formulates regression as conditional sequence modeling. Each example is a concatenation $B$1, where $B$2 is the tokenized property and $B$3 is the molecular, protein, or reaction sequence. The nominal-scale regression objective masks only property tokens and predicts them conditioned on text tokens: $B$4 The corresponding loss is the negative log-likelihood over property tokens,

$B$5

The mapping from float to tokens is exact via the regex-based numerical tokenizer, and decoding back to a float is done by reading the predicted digit tokens in order (Born et al., 2022).

Third, CaReTS formulates forecasting as residual prediction around the latest observation $B$6. Its variants differ in how classification and regression are fused. In CaReTS1, a binary trend decision $B$7 gates a non-negative deviation $B$8: $B$9 In CaReTS2, the model predicts direction-specific non-negative deviations $[y_{\min}, y_{\max}]$0 and $[y_{\min}, y_{\max}]$1 and selects one by the predicted sign. In CaReTS3, the fusion is soft,

$[y_{\min}, y_{\max}]$2

and in CaReTS4 trend probabilities are concatenated to the encoder features and a signed deviation $[y_{\min}, y_{\max}]$3 is predicted (Yao et al., 12 Nov 2025).

These formulations show that RvTC is not a single algorithm but a design pattern: a Transformer produces discrete or classification-like intermediates, and a continuous prediction is recovered either by expectation, exact de-tokenization, or trend-conditioned residual fusion.

3. Architectural realizations

The major published instantiations differ in modality, backbone, and coupling between classification and regression.

Setting	Backbone	Classification-to-regression mechanism
Image-based regression	mPLUG-Owl2 with CLIP ViT-L/14, visual abstractor, LLaMA-2-7B	$[y_{\min}, y_{\max}]$4-way linear head, softmax over bins, expectation over bin centers
Molecular/protein/reaction modeling	XLNet	Property tokens predicted with cross-entropy; the same backbone also supports conditional text generation
Time series forecasting	Transformer encoder in CaReTS	Trend classification branch gates, weights, or conditions deviation regression

In the image-based system, the model comprises a CLIP ViT-L/14 vision encoder, a visual abstractor that reduces visual features to 64 visual tokens per image, a LLaMA-2-7B language decoder, and a randomly initialized linear head with $[y_{\min}, y_{\max}]$5 outputs attached to the penultimate hidden state of the final token that aggregates all tokens (Jennings et al., 20 Jul 2025). The binning mechanism is therefore external to the backbone: the MLLM supplies a shared multimodal representation, and the regression estimator is induced by the classification head and expectation decoding.

In the Regression Transformer, the backbone is XLNet with 32 transformer layers, hidden size 256, FFN size 1024, 16 attention heads, dropout 20%, and relative positional encodings. The distinctive representational feature is numerical tokenization. Real-valued properties are converted into digit tokens preserving decimal order, and “numerical encodings” are added to embeddings and relative positional encodings. The float-based encoding is

$[y_{\min}, y_{\max}]$6

and the integer alternative uses sine and cosine terms (Born et al., 2022).

In CaReTS-Transformer variants, a fully connected projection maps raw inputs to model dimension $[y_{\min}, y_{\max}]$7, standard positional encodings are added, and the encoder stack uses $[y_{\min}, y_{\max}]$8 Transformer encoder layers with Multi-Head Self-Attention and a position-wise feedforward network. In experiments, $[y_{\min}, y_{\max}]$9, $z \in \mathbb{R}^B$0, the number of attention heads is 4, the branch heads are two FC layers with 64 hidden units each, and ReLU is the default nonlinearity (Yao et al., 12 Nov 2025).

A broader theoretical abstraction is provided by the distribution regression framework. There, the key representation is the attention operator

$z \in \mathbb{R}^B$1

which maps a probability measure to an RKHS function. A one-block Transformer encoder over distributions is then written as

$z \in \mathbb{R}^B$2

making explicit the view of attention as a distribution-to-function embedding followed by an FNN head (Liu et al., 28 Jun 2026).

4. Optimization, inference, and interpretability

The optimization objectives differ across RvTC variants, but cross-entropy is central in all of them.

In the image-based setting, training uses cross-entropy on hard one-hot bin labels with no label smoothing, ordinal constraints, or distributional losses. Inference computes $z \in \mathbb{R}^B$3 and predicts $z \in \mathbb{R}^B$4. No post-hoc calibration or temperature scaling is used, although the paper notes that temperature scaling could be fitted on a validation set by minimizing NLL (Jennings et al., 20 Jul 2025).

In the Regression Transformer, training proceeds in two phases. First, XLNet is pretrained with permutation language modeling. Second, refinement alternates every 50 steps between a property step that optimizes $z \in \mathbb{R}^B$5 and a generation step that optimizes the conditional generation objective $z \in \mathbb{R}^B$6. A self-consistency extension adds

$z \in \mathbb{R}^B$7

where $z \in \mathbb{R}^B$8 uses greedy-decoded masked text tokens. This term rewards consistency between the primed property and the property predicted for the newly completed sequence (Born et al., 2022).

In CaReTS, the objective is explicitly multi-task. The classification loss is BCE for binary variants or CE for two-class variants. The deviation regression loss is MSE on either absolute deviation or direction-specific deviations, and the output prediction loss is MSE on the final forecast. For architectures (a) and (b), the total loss is weighted by homoscedastic uncertainty, with learnable log-variances $z \in \mathbb{R}^B$9, weights $\hat{y} = \sum_{b=1}^B p_b\, c_b,$0, an additional soft regularizer, and clamping of $\hat{y} = \sum_{b=1}^B p_b\, c_b,$1 to $\hat{y} = \sum_{b=1}^B p_b\, c_b,$2 during training (Yao et al., 12 Nov 2025).

Interpretability enters differently in the three lines of work. CaReTS is explicit that the model disentangles “macro-level trend” from “micro-level deviations,” so a forecast can be narrated in terms of directional probability and deviation magnitude. In the image-based formulation, interpretability is weaker in the forecasting sense, but the bin probabilities provide a discrete score distribution over the label range. In the Regression Transformer, the continuous target is exposed as a token sequence within the same context as the structural sequence, which makes property conditioning and decoding operationally transparent (Yao et al., 12 Nov 2025, Jennings et al., 20 Jul 2025, Born et al., 2022).

5. Empirical results across domains

The empirical literature shows that RvTC is viable across multimodal image assessment, molecular and protein modeling, reaction prediction, and time series forecasting.

In image-based regression, the strongest results are reported on image aesthetic assessment, image quality assessment, and AI-generated image quality assessment. On AVA, vocabulary and generic-prompt models are essentially indistinguishable from image-only training: Q-Align obtains SRCC/PLCC of 0.822/0.817, an image-only model reaches 0.8229/0.8197, and RvTC with only 5 bins reaches 0.8232/0.8183. Increasing bins to 51 improves to 0.8329/0.8314, and full image-only RvTC achieves 0.833/0.831. With challenge titles as data-specific prompts, RvTC+ reaches 0.899/0.901, described as a new state-of-the-art. On IQA datasets, RvTC reports 0.94/0.95 on KonIQ-10k, 0.93/0.93 on SPAQ, and 0.98/0.98 on KADID-10k. On AGIQA-3k, alignment trained and evaluated with true prompts reaches 0.810/0.889, while perceptual quality reaches 0.872/0.916 (Jennings et al., 20 Jul 2025).

The prompt ablations in the same work are central to how RvTC is interpreted in multimodal settings. Generic task prompts such as “How would you rate this image?” do not leverage semantic knowledge, but data-specific prompts do. On AVA, image-only gives 0.833/0.831, challenge ID gives 0.851/0.843, shuffled titles give 0.860/0.851, and true titles give 0.899/0.901. On AGIQA-3k alignment, prompt shuffling at evaluation drops performance from 0.810/0.889 to 0.634/0.702, below image-only evaluation from the prompt-trained model at 0.687/0.826, whereas the perceptual task is largely insensitive to prompt presence or shuffling, with SRCC approximately $\hat{y} = \sum_{b=1}^B p_b\, c_b,$3 (Jennings et al., 20 Jul 2025).

In the Regression Transformer, the synthetic QED dataset shows that PLM-only training achieved regression RMSE below 0.06; the table reports SELFIES+NE per-model mean RMSE 0.0547, PCC 0.971, and perplexity approximately 1.59. Alternating objectives improve regression further; the best SELFIES+NE model achieves RMSE 0.0341 and PCC 0.988 without self-consistency, and RT with alternating objectives achieves MAE 0.017, surpassing SMILES-BERT at MAE 0.020. On MoleculeNet regression, RT reports ESOL 0.71–0.76, FreeSolv 1.19–1.40, and Lipophilicity 0.74–0.76, surpassing RF and XGBoost on all three and matching XLNet with a conventional regression head on Lipophilicity. In constrained molecular optimization for penalized LogP, RT reaches 3.16 improvement at $\hat{y} = \sum_{b=1}^B p_b\, c_b,$4 and 2.21 at $\hat{y} = \sum_{b=1}^B p_b\, c_b,$5, outperforming GCPN and JT-VAE under the reported settings (Born et al., 2022).

The same model extends to proteins and reactions. On the synthetic Boman index, property prediction reaches Spearman $\hat{y} = \sum_{b=1}^B p_b\, c_b,$6, and conditional generation attains Spearman $\hat{y} = \sum_{b=1}^B p_b\, c_b,$7 with 0-Var approximately 0.2%. On TAPE Fluorescence, RT achieves overall Spearman $\hat{y} = \sum_{b=1}^B p_b\, c_b,$8, outperforming the reported TAPE Transformer at 0.68 and UniRep at 0.67. On TAPE Stability, RT reports Spearman $\hat{y} = \sum_{b=1}^B p_b\, c_b,$9, competitive with LSTM 0.69, Transformer 0.73, and UniRep 0.73. For reaction yield prediction, RT reports $p_b$0 of approximately 0.939 on Buchwald-Hartwig and approximately 0.81 on Suzuki, and reaction decoration yields success rates of 47–82% on Suzuki and 42–82% on Buchwald-Hartwig with mean yield improvements of 4.6–14.4% (Born et al., 2022).

In time series forecasting, CaReTS evaluates two real-world tasks with 8,784 hourly observations each: electricity price and import/export power. Using a Transformer encoder, the best unmet power RMSE is reported for CaReTS2-Transformer at $p_b$1, slightly better than TimeXer at $p_b$2. On electricity price, CaReTS2-Transformer reports RMSE $p_b$3 with trend accuracy $p_b$4, while TimeXer has slightly lower RMSE $p_b$5 but lower trend accuracy $p_b$6. Trend accuracy is reported as above 90% across variants and encoders, and multi-task learning improves both RMSE and trend accuracy over single-task learning, with the example of CaReTS2 on unmet power improving RMSE from 0.0704 to 0.0691 and trend accuracy from 0.9060 to 0.9192 (Yao et al., 12 Nov 2025).

6. Theory, misconceptions, and limitations

The theoretical account of Transformer-based distribution regression gives RvTC a formal foundation beyond empirical discretization heuristics. The attention operator

$p_b$7

is proved to be injective and continuous under the stated assumptions, and when $p_b$8 it becomes an isometry onto its image, which the paper interprets as “no information loss” (Liu et al., 28 Jun 2026). This result is used to argue that attention compresses distributions into function representations in an RKHS without loss of information, after which an FNN can learn task-specific features.

The same paper derives an approximation theorem for Gaussian kernels: for Barron-class functionals $p_b$9, there exists a Transformer encoder with $t_{v,p}$0 parameters achieving

$t_{v,p}$1

It also gives a two-stage oracle inequality and, under the stated parameter choice, an expected excess risk rate

$t_{v,p}$2

For RvTC specifically, the paper decomposes regression error into a classification term and a quantization term, with illustrative bounds

$t_{v,p}$3

and

$t_{v,p}$4

This makes explicit the bias–variance tradeoff induced by discretization (Liu et al., 28 Jun 2026).

Several misconceptions are therefore addressed directly by the published record. First, RvTC is not inherently dependent on hand-crafted vocabularies: one motivation of the image-based work is precisely that preset output vocabularies and generic prompts add no value over image-only training, whereas flexible binning and data-specific prompts do (Jennings et al., 20 Jul 2025). Second, RvTC does not always require complex distributional modeling: the image-based study reports that simply increasing $t_{v,p}$5 yields monotonic gains that saturate around $t_{v,p}$6 (Jennings et al., 20 Jul 2025). Third, RvTC is not necessarily restricted to quantized outputs: the Regression Transformer preserves exact numerals through digit tokens, and CaReTS leaves magnitude prediction continuous while classifying only trend (Born et al., 2022, Yao et al., 12 Nov 2025).

The limitations are equally explicit. In pure regression-via-classification, binning introduces bias and resolution limits, and the theory notes that calibration and ordinal design choices matter (Liu et al., 28 Jun 2026). In the Regression Transformer, nominal-scale losses can lag in fine-grained ratio-scale accuracy, as seen in the observation that fine-grained regression within the bright fluorescence mode is harder; label spikes can also trigger degenerate token predictions, which the authors mitigate with small Gaussian noise (Born et al., 2022). In CaReTS, binary trend labels may miss flat or no-change states, anchoring at $t_{v,p}$7 may be fragile under large sudden regime shifts, and longer horizons may require additional mechanisms (Yao et al., 12 Nov 2025). In multimodal image regression, prompt usefulness is task-dependent: semantically relevant prompts help alignment-sensitive tasks, but poor or mismatched prompts can hurt performance, and full-model fine-tuning of a 7B-class MLLM is compute-intensive (Jennings et al., 20 Jul 2025).

Taken together, these works present RvTC as a technical pattern rather than a single canonical model. Its shared premise is that Transformer-generated classification structure—bins, digit tokens, or trend classes—can be exploited to recover or guide continuous prediction, often with favorable optimization behavior, interpretable intermediate structure, and, in some settings, additional capabilities such as controllable generation or semantic multimodal conditioning.