Soft Embeddings in Representation Learning

• Soft embeddings are continuous, trainable representations that capture uncertainty and semantic gradation, enhancing applications like prompt tuning, probabilistic reasoning, and retrieval.
• Techniques such as soft prompt embeddings and attention-based prompt pools have demonstrated measurable performance gains, including improved F1 scores and faster convergence in benchmarks like PERSOMA and SuperPos-Prompt.
• They also facilitate graded supervision and controlled geometric regularization, which improve semantic alignment and inference in multilingual, neuro-symbolic, and federated adaptation tasks.

Soft embeddings are continuous, trainable, probabilistic, or softly regularized embedding representations that replace or augment hard token, label, or state assignments. In recent arXiv literature, the term spans several distinct but related constructions: learnable soft prompt tokens for parameter-efficient adaptation, expected embeddings under output distributions for differentiable generation and latent reasoning, probability distributions over symbolic domains for neuro-symbolic inference, soft-label contrastive embedding spaces for multilingual and dialog-action alignment, and post-processed embedding spaces whose geometry is controlled for semantic retrieval (Hebert et al., 2024, Burdisso et al., 2024, AbdAlmageed, 19 Mar 2026, Zhu et al., 26 Sep 2025, Diera et al., 2024). The shared design principle is that the representation remains continuous enough to preserve uncertainty, semantic gradation, or gradient flow.

1. Conceptual scope

The phrase soft embeddings does not denote a single standardized object. In the cited literature it refers to multiple technical mechanisms, each introduced for a different bottleneck in representation learning, retrieval, reasoning, or control.

Research usage	Embedding object	Reported role
Prompt and adapter tuning	Learnable soft prompt embeddings or task embeddings	Prepend to inputs or inject into activations
Contrastive alignment	Embeddings trained with soft labels	Respect graded semantic similarity
Probabilistic reasoning	Expected token embeddings or per-position symbol distributions	Preserve uncertainty and differentiability
Geometric post-processing	Soft-ZCA-whitened embeddings	Control isotropy for semantic code search
Structural context	Spatio-temporal or vectorized scene embeddings	Encode sensor placement, timing, or soft constraints

Prompt-oriented work uses trainable vectors as surrogate tokens. PERSOMA converts user histories into personalized soft prompt tokens for PaLM 2 XXS; SuperPos-Prompt reparameterizes prompt tokens as superpositions of pretrained vocabulary embeddings; LiSTEN selects soft token values from a learnable prompt pool; SPEAR prepends learnable prompts to quantized time-series embeddings; and SITE shifts task conditioning from prompt space to activation space through soft injection of task embeddings (Hebert et al., 2024, SadraeiJavaeri et al., 2024, Mousavi et al., 24 May 2025, Wei et al., 4 Oct 2025, Park et al., 28 Jul 2025).

A second cluster of work uses soft to mean probabilistic rather than prompt-based. AS2 maintains a probability distribution over symbols at each position, Soft-Di[M]O replaces discrete output tokens with expected embeddings under the generator distribution, SCM forms a probability-weighted average of token embeddings for latent reasoning, and the Soft Graph Transformer embeds prior bit LLRs as soft inputs for MIMO detection (AbdAlmageed, 19 Mar 2026, Zhu et al., 26 Sep 2025, Wang et al., 21 Nov 2025, Hong et al., 16 Sep 2025).

A third usage concerns embedding geometry and alignment. Dialog2Flow introduces a soft supervised contrastive loss that weights negatives by semantic similarity of dialog-action labels; multilingual alignment work distills teacher-measured sentence similarity into cross-lingual soft labels; and Soft-ZCA whitening regulates isotropy in code embeddings rather than enforcing a single hard whitening regime (Burdisso et al., 2024, Park et al., 2024, Diera et al., 2024).

2. Prompt-space and activation-space conditioning

Soft prompt embeddings are the most common contemporary meaning of soft embeddings in LLM adaptation. In PERSOMA, a user history encoder maps each history item $h_i$ to a SentenceT5 embedding $h'_i$, an adapter compresses the set $H'_u$ into personalized prompt tokens $P_u$, and the decoder consumes $[T;P_u;X]$, where $T$ denotes learned task tokens and $X$ the task input. The architecture explores MLP, Transformer, and Perceiver adapters; the Perceiver compresses long histories to a fixed-size prompt. On the MovieLens personalized genre prediction benchmark with 127K users and 14M reviews, PERSOMA MLP reports $F1 = 0.569$ at history size 50, compared with $0.381$ for UEM Large; for 50 history items, the Perceiver variant requires 100 tokens versus 16,000 tokens for text-based methods; and frozen-LLM and LoRA variants report $F1 = 0.541$ and $h'_i$0, compared with $h'_i$1 for full fine-tuning (Hebert et al., 2024).

SuperPos-Prompt changes the parameterization of prompt tokens themselves. Instead of optimizing each prompt token as a free vector, each prompt embedding is represented as

$h'_i$2

where $h'_i$3 contains sampled pretrained token embeddings. This superposition of pretrained vocabulary embeddings yields an average score increase of $h'_i$4 in T5-Small and $h'_i$5 in T5-Base relative to Residual Prompt tuning across GLUE and SuperGLUE, together with faster convergence. The same study reports consistent gains from omitting dropout in the frozen network during prompt tuning (SadraeiJavaeri et al., 2024).

LiSTEN generalizes soft prompts to multitask audio-language adaptation by maintaining a learnable prompt pool of key-value pairs $h'_i$6. A query vector $h'_i$7, formed from mean-pooled audio-text embeddings, selects prompt values dynamically by similarity-based, attention-based, residual-based, or stochastic strategies. The overall loss combines next-token prediction with a strategy-specific key loss, $h'_i$8. The reported dynamic prompt selection configuration achieves best or second-best results on 6 of 8 metrics while using approximately 3.3M trainable parameters (Mousavi et al., 24 May 2025).

SPEAR applies the same prefix principle to time series anomaly detection. A continuous time series $h'_i$9 is quantized to tokens $H'_u$0, embedded as $H'_u$1, concatenated with learnable prompt vectors $H'_u$2, and fed to a frozen LLM as

$H'_u$3

Only the prompt embeddings are updated under binary cross-entropy loss. On MIMIC-IV, SPEAR-BERT reports accuracy $H'_u$4, $H'_u$5, and AUROC $H'_u$6, compared with BERT zero-shot at $H'_u$7, $H'_u$8, and $H'_u$9; prompt modules are also described as very lightweight, with 0.06 MB for BERT’s prompts (Wei et al., 4 Oct 2025).

SITE relocates soft embeddings from the prompt prefix to the network interior. It constructs task embeddings by averaging last-token attention-head activations from few-shot prompts, then blends those task embeddings into head outputs via learnable soft head-selection parameters,

$P_u$0

Across 57 tasks and 12 LLMs, SITE reports improvements of $P_u$1–$P_u$2 over 10-shot ICL while removing in-prompt demonstrations at inference time (Park et al., 28 Jul 2025).

3. Soft labels, contrastive objectives, and embedding geometry

In embedding alignment, soft usually refers to graded supervision rather than trainable prompt tokens. Dialog2Flow learns sentence embeddings for dialog utterances so that utterances cluster by communicative and informative function. Its BERT-based encoder $P_u$3 is followed by an MLP head $P_u$4, and training uses a soft supervised contrastive loss in which the contribution of each negative is weighted by semantic similarity $P_u$5 between action labels. The unified corpus standardizes 20 task-oriented dialog datasets into 18 dialog acts, 524 slot labels, and 3,982 unique action labels. Reported results include 5-shot $P_u$6 and $P_u$7 anisotropy $P_u$8 for the soft model, compared with $P_u$9 and $[T;P_u;X]$0 for the hard variant; workflow graph size also more closely matches annotated ground truth, with node-count difference around $[T;P_u;X]$1 versus $[T;P_u;X]$2–$[T;P_u;X]$3 for baselines (Burdisso et al., 2024).

A closely related idea appears in multilingual sentence alignment. Instead of treating every non-translation pair as an equally negative example, the student model is trained to match the similarity structure of a teacher encoder. Soft labels $[T;P_u;X]$4 are obtained by softmax-normalizing teacher cosine similarities among source sentences, or by averaging source- and target-side similarities. The resulting soft contrastive loss outperforms hard contrastive learning in bitext mining and STS, and the paper reports a Tatoeba score of $[T;P_u;X]$5, compared with $[T;P_u;X]$6 for MSE and $[T;P_u;X]$7 for LaBSE; for en-ko, the reported scores are $[T;P_u;X]$8 for soft contrastive, $[T;P_u;X]$9 for hard contrastive, and $T$0 for MSE (Park et al., 2024).

Embedding geometry introduces another meaning of softness: controlled relaxation of an otherwise aggressive transformation. The code-search study on isotropy observes that pre-trained code LMs such as CodeBERT, CodeT5+, and Code Llama have very low IsoScore values, typically $T$1 to $T$2, and that low isotropy impairs semantic inference. Standard ZCA whitening uses

$T$3

whereas Soft-ZCA introduces an eigenvalue regularizer,

$T$4

This modification prevents very small eigenvalues from being amplified excessively. Reported $T$5MRR gains include $T$6 for CodeBERT on Ruby, $T$7 for CodeBERT on Python, $T$8 for CodeLlama on Ruby, and $T$9 for CodeLlama on low-resource $X$0. The same study explicitly notes that the best IsoScore is not always the highest: base models perform best at IsoScore in $X$1, while fine-tuned models do best near $X$2 (Diera et al., 2024).

A recurrent misconception is that “harder” separation or “more isotropy” is automatically preferable. The reported results in Dialog2Flow, multilingual alignment, and Soft-ZCA indicate the opposite: graded semantic weighting or controlled geometric regularization can outperform all-or-nothing objectives (Burdisso et al., 2024, Park et al., 2024, Diera et al., 2024).

4. Probabilistic embeddings and differentiable latent computation

Several recent systems use soft embeddings as continuous relaxations of discrete symbolic or lexical choices. SCM defines a soft concept vector at each decoding step as a probability-weighted average over vocabulary embeddings,

$X$3

and then adds it to the hidden state,

$X$4

The model is optimized with GRPO-based reinforcement learning. Across five reasoning benchmarks, the paper reports that SCM consistently outperforms CoT, inference-time Soft Thinking, and a GRPO baseline; the average score for DeepSeek-R1-Distill-Qwen-7B is reported as $X$5 for SCM versus $X$6 for GRPO and $X$7 for Soft Thinking (Wang et al., 21 Nov 2025).

Soft-Di[M]O applies an analogous relaxation to one-step discrete image generation. For position $X$8, the generator emits logits whose softmax defines $X$9, and the soft embedding is the expected embedding

$F1 = 0.569$0

Because this representation is differentiable, the generator becomes compatible with GAN-based refinement, differentiable reward fine-tuning, and Test-Time Embedding Optimization. The paper reports a one-step FID of $F1 = 0.569$1 on ImageNet-256 with GAN-based refinement, plus higher GenEval and HPS scores on text-to-image tasks (Zhu et al., 26 Sep 2025).

AS2 pushes the same idea into neuro-symbolic reasoning. Rather than committing to a discrete symbol at each position, it maintains $F1 = 0.569$2, a per-position distribution over a finite symbol domain. Constraint satisfaction is enforced by minimizing the fixed-point residual of a probabilistic lift of the ASP immediate consequence operator $F1 = 0.569$3. For Sudoku-like uniqueness constraints, the lifted operator is

$F1 = 0.569$4

AS2 is explicitly free of conventional positional embeddings and instead uses constraint-group membership embeddings, such as row, column, and box membership in Sudoku. On Visual Sudoku, it reports $F1 = 0.569$5 cell accuracy and $F1 = 0.569$6 constraint satisfaction across 1,000 test boards; on MNIST Addition with $F1 = 0.569$7, digit accuracy is above $F1 = 0.569$8 at all scales (AbdAlmageed, 19 Mar 2026).

In communication-theoretic form, the same relaxation appears in the Soft Graph Transformer for MIMO detection. Here the soft input is a vector of prior bit LLRs $F1 = 0.569$9, embedded as

$0.381$0

The model is described as soft-input-soft-output because it preserves bitwise uncertainty throughout cross-attention message passing and emits posterior LLRs that can be fed back into iterative detection-decoding. The reported result is that SGT closely approaches ML performance and surpasses prior Transformer-based approaches (Hong et al., 16 Sep 2025).

5. Structural and scene-conditioned soft embeddings

In robotics and autonomous driving, soft embeddings are used to encode variable structure, missing modalities, and soft constraint adherence. The differentiable Kalman filtering work on soft robots introduces spatio-temporal embeddings for sensor position and sampling frequency. Positional embeddings use the standard sinusoidal form

$0.381$1

and are combined with raw observations before the sensor model; temporal embeddings encode sampling frequency for the transition model. In the ablation study, DEnKF-PE+TE reports end-effector MAE $0.381$2 mm and quaternion error $0.381$3 under varying positions and sampling rates, compared with $0.381$4 mm and $0.381$5 for DEnKF-Fix. The broader framework reports at least a $0.381$6 reduction in MAE relative to state-of-the-art filtering methods and an average MAE of $0.381$7 mm from ground truth (Liu et al., 2023).

Driving-constraint learning uses vectorized scene embeddings to convert imitation learning into a reward-and-constraint decomposition. The embedding is produced by VectorNet over dynamic and static scene elements, and each candidate trajectory $0.381$8 receives a reward $0.381$9 and a soft constraint score $F1 = 0.541$0. The maximum-entropy selection probability becomes

$F1 = 0.541$1

The constraint network is trained by labeling collisions, out-of-map trajectories, and stuck trajectories as violating constraints, while the trajectory closest to ground truth is labeled compliant. On InD, adding all constraints reduces collision rate from $F1 = 0.541$2 to $F1 = 0.541$3 and outside-road incidents to $F1 = 0.541$4; on TrafficJams, collision rate drops from $F1 = 0.541$5 to $F1 = 0.541$6 and outside-road incidents also drop to $F1 = 0.541$7 (Mobarakeh et al., 2024).

These cases use soft in a distinct sense from prompt tuning: the embedding is not a surrogate token but a continuous encoding of spatial, temporal, or safety-relevant context, and the soft quantity is often a probability of constraint satisfaction rather than a discrete validity flag.

6. Subspace embeddings, compression, and federated adaptation

The mathematical theory of embeddings under nonlinear transformations provides a useful backdrop for the softer constructions above. For a $F1 = 0.541$8-dimensional subspace $F1 = 0.541$9 and an entrywise nonlinearity $h'_i$00, the set of interest is

$h'_i$01

The cited work gives the first low-distortion embeddings for a broad class of nonlinearities. For functions with bounded second derivative and linear asymptotes, additive-error embeddings into

$h'_i$02

dimensions are proved for classes including Sigmoid, SoftPlus, and Gaussian. Under stronger near-linearity conditions near the origin, relative-error embeddings into

$h'_i$03

dimensions are obtained for Tanh, SoftSign, ELU, ArcTan, and SQNL (Gajjar et al., 2020). This result concerns embeddings of sets induced by soft nonlinearities rather than prompt embeddings, but it establishes the dimension-reduction theory underlying continuous nonlinear representations.

Compression-oriented work uses subspace embeddings in yet another sense: a compact reconstruction of token embeddings from a small number of shared factors. In the lightweight adaptation study, a token embedding is written as

$h'_i$04

with modulo-style assignment

$h'_i$05

This replaces $h'_i$06 parameters by $h'_i$07, and the reported compression exceeds $h'_i$08 with up to $h'_i$09 absolute accuracy sacrifice. A reported RoBERTa example reduces 25.7M embedding parameters to 18.9K, a $h'_i$10 reduction; clustered assignment improves performance relative to arbitrary assignment, including $h'_i$11 MNLI accuracy and $h'_i$12 SST-2 versus $h'_i$13 and $h'_i$14 for the 3-SE arbitrary variant (Jaiswal et al., 2023).

Federated adaptation combines soft embeddings with parameter-efficient retrieval training. The proposed encoder freezes a small LLM and inserts a trainable square transformation adapter between token embedding and transformer blocks, so that the adapter learns enhanced soft embeddings for a new corpus. A classifier head then learns a similarity mapping for classifier-as-retriever, and only the adapter and head are trained under federated learning with differential privacy. On SMS Spam, the reported top-1 retrieval accuracies are $h'_i$15 for MIPS with a frozen LLM, $h'_i$16 for Adapter + MIPS, $h'_i$17 for classifier head only, and $h'_i$18 for Adapter + classifier head. Federated training reports speedups up to $h'_i$19 with 3 clients in the adapter-plus-classifier configuration (Fofonjka et al., 20 Sep 2025).

7. Persistence, deletion, and security

A final and conceptually different usage appears in vector-database governance: soft-deleted embeddings. Here soft does not refer to continuity or probabilistic relaxation, but to metadata-only deletion. In HNSW vector databases, soft deletion marks the record as deleted while leaving the raw embedding physically unchanged on disk. The storage-layer analysis confirms this behavior in ChromaDB, FAISS, and Weaviate, and shows that deleted vectors can be recovered by bypassing API access and reading raw index files directly (Chakraborttii et al., 16 Jun 2026).

The security consequences are quantified with Vec2Text inversion and related recovery procedures. On the Wikipedia Biographical Living Persons dataset, the paper reports recovery of $h'_i$20 of exact person names and $h'_i$21 of geographic locations, with ROUGE-L $h'_i$22. On Synthea clinical data, recovery reaches $h'_i$23 for age and gender markers, with ROUGE-L $h'_i$24. For images, top-1 identity recovery on LFW reaches $h'_i$25, while histopathology tissue classification reaches $h'_i$26 (Chakraborttii et al., 16 Jun 2026).

The proposed mitigation, Epoch Key Rotation, encrypts vectors and discards the key upon deletion. The reported effect is a reduction of observed PII recovery to $h'_i$27, with completion time $h'_i$28 ms for 500 deleted vectors, approximately $h'_i$29 ms per record, plus an ECDSA-signed cryptographic proof of deletion (Chakraborttii et al., 16 Jun 2026). Across the broader literature, this security result sharply distinguishes soft as a representational design choice from soft-delete as an administrative operation: the former preserves useful continuity for learning, while the latter preserves the underlying vector unless physical or cryptographic erasure is performed.