Text-to-Text Regression Paradigm

Updated 3 July 2025

Text-to-text regression is a paradigm that represents both inputs and outputs as text sequences, enabling end-to-end prediction using transformer-based models.
It eliminates traditional feature engineering by converting structured and unstructured data into text, allowing flexible handling of diverse information types.
It applies encoder-decoder architectures and context augmentation to achieve robust regression, uncertainty quantification, and rapid adaptation across domains.

Text-to-text regression is a methodological paradigm in which both inputs and outputs of regression models are represented as sequences or strings of text, and prediction is performed end-to-end by models that natively process text, typically LLMs or transformer-based architectures. This approach subsumes a wide array of modeling settings—from high-dimensional document sentiment analysis to universal regression over arbitrary system configurations, as well as frameworks where context generated by LLMs mediates between predictor and outcome strings. The paradigm eliminates the need for traditional feature engineering, accommodates unstructured, structured, or mixed-format data, and supports both point estimation and full predictive distribution modeling. Recent work demonstrates its technical foundation, practical implementation strategies, and broad applicability across domains such as social science, industry-scale system prediction, and language analysis.

1. Core Principles and Motivations

Text-to-text regression is designed to address several long-standing bottlenecks in classical regression, particularly when applied to high-dimensional or complex text data:

Flexible Input Handling: Text regression models can naturally ingest variable-length, non-tabular, and hierarchically structured data (e.g., raw text, logs, configuration files) without requiring upfront vectorization or canonical schema.
Unified Model Architecture: Modern LMs and encoder-decoder models serve as universal predictors by treating both inputs and numerical targets as token sequences, avoiding regressors or output heads specialized for particular formats. All prediction (including numerics) is performed via next-token prediction, often after serializing numbers using digit, sign, and exponent tokens.
Minimal Feature Engineering: Because all features are passed as text, new variables, fields, or modalities can be incorporated simply by editing the input string, greatly reducing maintenance overhead and brittleness.
Transfer Learning and Generalization: Models trained within this paradigm can share knowledge across tasks and domains, as demonstrated by multi-task approaches that improve performance by leveraging shared structural or semantic representations in the data (2402.14547).
Theoretical and Practical Scalability: By treating both inputs and outputs as text, the paradigm scales to heterogeneous settings and supports tasks where dependencies are not easily encoded in standard tabular form (2506.21718).

2. Model Architectures and Implementation Strategies

Several modeling strategies are employed in the text-to-text regression literature, each tailored to the complexity of the input text and the desired type of prediction.

Encoder-Decoder Architectures: Sequence-to-sequence models (often transformer-based) such as T5 or small custom encoders are trained to conditionally generate serialized numerical outputs from textual descriptions of the input (2506.21718, 2402.14547). Inputs can be long, structured texts (system configs, logs), and outputs may be quantitative (e.g., “72.5” encoded as <+>\<7>\<2>\<5><E-1>).

Span-Extraction as Regression: For tasks requiring selection of a scalar target from a discretized set, models can be trained to extract text “spans” corresponding to the correct value in a bucketed source sequence (1904.09286). This enables regression, classification, and QA to be unified under a single architecture.

Sentence Semantic Regression: In generative modeling, sentence-level semantic vectors can serve as regression targets, bridging the gap between coarse idea “reasoning” and fine-grained surface realization in text generation (2108.02984). Here, SSR-AR and SSR-NonAR architectures operate autoregressively or bidirectionally at the sentence granularity, decoupling high-level semantic planning from token-level realization.

Context Augmentation: In recent inferential paradigms, LLMs generate multiple plausible “contexts” around each predictor string, which serve as mediators in the prediction from predictor text to outcome text (2506.23862). This supports statistical identification, variance estimation, and even causal analysis through negative controls.

Architectural Tuning and Serialization: Empirical ablations show that increasing encoder depth, sequence length, and careful design of numerical tokenization is critical for performance and uncertainty quantification (2506.21718, 2402.14547).

3. Estimation Procedures, Loss Functions, and Inference

Text-to-text regression models use a range of estimation strategies suited to high-dimensional, ordinal, or distributional prediction tasks:

Penalized Likelihood in High Dimensions: Models such as multinomial inverse regression (MNIR) use penalized logistic regression (gamma-lasso) to collapse the predictor space to low-dimensional, sentiment-preserving scores. The gamma-lasso scheme assigns adaptive Laplace priors to each coefficient for stable estimation and variable selection in very high dimensions (1012.2098).
Loss Functions with Ordinal Awareness: For tasks like semantic textual similarity (STS), custom losses such as Translated ReLU and Smooth K2 Loss introduce “buffer zones” where no gradient is applied for near-miss predictions, reflecting the ordinal or progressive nature of the labels and yielding training stability and improved performance (2406.05326).
Uncertainty Quantification: Some models are trained as full conditional density estimators, with numeric targets serialized for sampling, allowing the aggregation of predictive samples (mean, median) and providing valid variance or distribution estimates (2506.21718, 2402.14547).
Iterative and Multi-Stage Inference: In models that aggregate evidence over generated contexts, statistics are estimated repeatedly across splits (“cross-fitting”) to restore uncertainty and achieve higher-order efficiency (2506.23862).
Statistical Validity and Simultaneous Inference: Cluster-robust estimators and post-selection procedures for high-dimensional data enable valid hypothesis testing on thousands of features in text regression, even when perfect model selection is unattainable (1812.09397).

4. Applications and Benchmark Results

Text-to-text regression has been applied to a wide spectrum of domains, with documented empirical advantages:

Application	Method Example	Key Empirical Result
Sentiment analysis, party affiliation	MNIR + gamma-lasso	Out-of-sample RMSE 1.07–10.7, strong interpretability (1012.2098)
Semantic Textual Similarity (STS)	Regression + Smooth K2	Outperforms Sentence-BERT, state-of-the-art Spearman correlations (2406.05326)
Text readability assessment	Seq2seq text-to-text	99.6% pairwise classification accuracy, robust cross-domain (2302.13139)
Industrial performance prediction	RLM text-to-text models	Up to 0.99 rank correlation, 100x lower MSE than tabular models (2506.21718)
Universal scientific regression	OmniPred, multitask LM	Outperforms MLP/tree/GP, robust transfer across black-box tasks (2402.14547)
Dialogue, psycholinguistic analysis	LLMs with context mediation	Reveals syntax/semantics effects, valid statistical inference (2506.23862)

In addition, the paradigm supports rapid cross-domain adaptation: for example, encoder-decoder models can adapt to new tasks in as few as 500 examples (2506.21718), and multi-task pretraining brings notable gains even with highly heterogeneous input spaces (2402.14547).

5. Advantages, Challenges, and Scalability

Advantages:

Full Input and Output Flexibility: Any textual or serializable information can be incorporated, and outputs can be continuous, categorical, or even full distributions.
Minimal Human Bottleneck: No need to design or maintain feature schemas as information needs or system complexity evolve.
Unified Solution for Multiple Downstream Tasks: Text-to-text regression models can subsume classification, regression, ordering, and density estimation tasks in a single system.
Rapid Adaptation and Transfer: Models improve through multi-task learning, can generalize with few-shot learning, and adapt to new data distributions with minimal retraining.
Statistically Valid Inference: Procedures exist for controlling false discovery and supporting simultaneous confidence intervals in high-dimensional settings.

Challenges:

Numeric Representation Artifacts: Poor serialization of numbers can lead to errors or hallucinations; per-digit tokenization and specialized output vocabularies are important.
Computational Resources: While moderate-sized models (60–200M parameters) often suffice, longer sequences and large context windows demand substantial memory.
Theoretical Understanding: While empirical evidence is strong, theoretical guarantees on generalization, especially for in-context learning, are still under investigation (2404.07544).

6. Impact on Broader Research Directions

The paradigm has spurred advances in several related methodologies:

Universal Regressors and Surrogate Modeling: Models trained as text-to-text regressors now serve as surrogates in AutoML, scientific discovery, and complex system simulation, supporting robust uncertainty modeling and density estimation (2402.14547, 2506.21718).
Direct Integration with Statistical Practice: Context augmentation allows LLMs to play an active and interpretable role in semiparametric and causal models, connecting modern language technologies with classical inference and mediation frameworks (2506.23862).
Bridging Classification, Regression, and Generation: Unified architectures permit joint training and direct transfer among QA, classification, regression, and sequence modeling, yielding efficiency in training and deployment (1904.09286, 2005.10433).
Methodological Cross-Fertilization: Custom loss functions and estimation techniques tuned to ordinal or progressive labels (e.g., Smooth K2) may inform new developments in both NLP and statistical regression.

7. Theoretical and Methodological Advances

Recent research has provided:

Identification and Influence-Function Theory: Clear identification conditions, influence-function decompositions, and higher-order efficiency results for estimators using context augmentation and cross-fitting (2506.23862).
Adaptive Penalization in High Dimensions: Advances such as the gamma-lasso enable robust sparse regression in large vocabulary text models, achieving “oracle” properties and unbiasedness for strong features without over-shrinkage (1012.2098).
Regret Analysis in In-Context Learning: Analyses show that with increasing in-context examples, LLMs can approach optimal predictors with sub-linear regret in regression tasks (2404.07544).
Resource-Efficient Inference Procedures: Text-to-text regression offers lower memory requirements and higher efficiency compared to contrastive models in some sequence similarity tasks (2406.05326).

Text-to-text regression, as substantiated by a growing literature, represents a unified, versatile, and scalable approach to regression and statistical inference in diverse text-rich and high-dimensional applications. Its synthesis of LLM flexibility, advanced estimation techniques, and statistical rigor positions it as a foundational paradigm in modern predictive modeling and scientific discovery.