Length-Aware Semantics (LAS) Module

• The paper demonstrates that LAS modules significantly enhance neural language models by incorporating explicit length-aware constraints for robust semantic encoding and token-length prediction.
• The methodology integrates contrastive learning with elongation augmentation (LA(SER)³) for length-agnostic document embeddings and employs SE-inspired modulation for token-length regression.
• Empirical results reveal improved information retrieval performance and distributed inference efficiency, with notable gains in nDCG@10 and reduced L1-loss in token length prediction.

Length-Aware Semantics (LAS) Modules are architectural and algorithmic components designed to enhance neural LLMs with explicit handling of text length—either by conferring semantic robustness across length variations (document encoding context) or by enabling precise token length prediction (distributed LLM inference context). Two distinct LAS paradigms have emerged in recent literature: (1) the LA(SER)$^3$ framework for robust, length-agnostic semantic representations in unsupervised contrastive learning (Xiao et al., 2023), and (2) the token-length-sensitive LAS predictor for scalable task offloading in edge-cloud environments (Wu et al., 28 Dec 2025). Both approaches demonstrate that incorporating length-aware design constraints or inductive biases addresses fundamental weaknesses in vanilla LLMs, improving performance in zero-shot information retrieval and distributed scheduling, respectively.

1. Architectural Designs of LAS Modules

LAS modules are instantiated differently depending on the task context. In semantic representation learning (Xiao et al., 2023), the LAS architecture (LA(SER)$^3$) operates by encoding a document $S$ via a Transformer encoder $g(\cdot)$ (e.g., BERT-base, MiniLM), producing parallel anchor ($h = g(S)$) and augmented ($h^+ = g(f(S,m))$) representations. The augmentation $f(S,m)$ implements length-conditioned self-repetition (elongation), ensuring the encoder observes a broad spectrum of input lengths.

Conceptual LAS encoding flow (LA(SER)$^3$):

Input Sentence S ├─> Anchor branch: h = g(S) └─> Augmentation: h^+ = g(f(S, m)), with elongation factor m ↓ Both pooled (mean or [CLS]), compared via contrastive loss (InfoNCE)

In the inference context (Wu et al., 28 Dec 2025), the LAS module consists of a frozen ModernBERT encoder producing a prompt representation $z \in \mathbb{R}^d$, a feature modulation block inspired by Squeeze-and-Excitation mechanisms tailored to amplify dimensions most correlated with output token length, and a regression head that predicts the expected output token count.

Abstract LAS for token-length prediction:

Prompt ↓ tokenization Pretrained ModernBERT Encoder ↓ Last-layer embedding z ∈ ℝᵈ ↓ LAS Feature Modulation (SE block) ↓ Regression head → ŷ (token length estimate)

2. Mathematical Formulation and Theoretical Foundations

LA(SER)$^3$ (Document Semantics Robustness)

Contrastive learning is realized using InfoNCE loss with temperature $\tau$. Given a batch of $N$ anchor representations $\{h_i\}$ and positives $\{h_i^+\}$, the loss for pair $i$ is

$$\ell_i = -\log \frac{\exp(\mathrm{sim}(h_i, h_i^+)/\tau)} {\sum_{j=1}^{N} \exp(\mathrm{sim}(h_i, h_j^+)/\tau) +\sum_{j=1}^{N} \exp(\mathrm{sim}(h_i, h_j)/\tau)}$$

where similarity is $\mathrm{sim}(a, b) = \frac{a^\top b}{\|a\|\|b\|}$.

Elongation augmentation $f(S,m)$ constructs the positive sample as $m$-fold repetition, $[x_1, \ldots, x_n, x_1, \ldots, x_n, \dots]$ (up to $L_{\max}$ tokens), enforcing semantic invariance to artificial document lengthening.

Theoretical analysis establishes that naive contrastive learning inflates similarity scores for elongated inputs by disproportionately increasing attention weights on dominant tokens:

$$\mathrm{Att}(x_i \to x_k) = \frac{\exp(q_i k_k^T / \sqrt{d_k})}{\sum_\ell \exp(q_i k_\ell^T / \sqrt{d_k})}$$

After elongation, dominant tokens ($l_d > l_r$) accrue more attention shift ($G_d > G_r$), degrading isotropy and authentic semantic discrimination. LA(SER)$^3$ counters this effect by training the encoder such that $g(S) \approx g(f(S,m))$ across varied $m$.

LAS for Token Length Prediction (Inference Optimization)

The LAS prediction module prepares the prompt embedding $z$, then applies a length-sensitive modulation:

$$\begin{align*} s &= \text{Pool}_{\mathrm{avg}}(z) + \text{Pool}_{\max}(z) \ e &= \sigma(W_{exp} \mathrm{ReLU}(W_{sq} s)) \ z' &= z \odot e \ \hat{y} &= w^T z' + b \end{align*}$$

Training minimizes $L_1$ loss over predicted and realized output lengths:

$$L_{LAS}(\theta) = \frac{1}{N} \sum_{i=1}^{N} |\hat{y}_i - y_i| + \lambda \|\theta\|_2^2$$

Predicted output $\hat{y}$ is rounded for operational use as $\hat{L} = \lceil \hat{y} \rceil$.

3. Empirical Performance and Evaluations

LA(SER)$^3$ Zero-Shot IR Benchmarks

On BEIR-14 zero-shot information retrieval, LA(SER)$^3$ achieves state-of-the-art unsupervised nDCG@10. With BERT-base and batch=128, nDCG@10 rises to 0.2594, outperforming SimCSE (0.2197), DiffCSE (0.1916), and InfoCSE (0.2509) (Xiao et al., 2023). For MiniLM-L6 trained on MS-MARCO, LA(SER)$^3$-intra-ref yields 0.2087 nDCG@10 across retrieval tasks.

LAS Token Length Estimation and Offloading Impact

The LAS predictor in Argus achieves 91.85 mean absolute error (L1-loss) on held-out test data—lower than both LoRA-tuned ModernBERT (92.07), LSTM-only (107.79), and Transformer-only (106.69), despite having just 0.09M trainable parameters (versus LoRA’s 8.75M) (Wu et al., 28 Dec 2025). In distributed offloading experiments, accurate LAS length prediction yields up to +99.8% reward improvement in resource allocation under high system dynamism, with less over-provisioning or queue congestion.

Model/Method	nDCG@10 (IR)	L1 Loss (Len)
LA(SER)$^3$ (BERT-base,128)	0.2594	–
SimCSE (BERT-base,64)	0.2197	–
LAS (Argus, ModernBERT)	–	91.85
LoRA-tuned ModernBERT	–	92.07
LSTM	–	107.79

4. Training Procedures and Ablation Studies

LA(SER)$^3$ Training Protocol

Corpora include 1M Wikipedia sentences and MS-MARCO passages. Inputs are tokenized and padded/cropped to $L_{\max}=256$. Elongation factor $m$ is uniformly sampled per instance, exposing the encoder to broad document lengths. Hyperparameters: batch size 64/128, learning rate $3\times 10^{-5}$, temperature 0.05, mean-pooling (with ablation for [CLS] pooling). No explicit regularization beyond AdamW weight decay (Xiao et al., 2023).

Ablations indicate that random anchor selection decreases performance (−10.05%), while intra-reference augmentation is more robust to anchor choice (+8.97% first, +5.33% random). Using random elongation (as opposed to fixed or none) increases average nDCG@10 from 0.1263 (none) to 0.1816 (random in $m\in[1,25]$).

LAS Inference Fine-Tuning

LAS predictor is trained on prompt-length pairs collected via a reference LLM. ModernBERT is frozen; only modulation and regression head parameters updated via AdamW (lr=1e-4, batch size 32–128, weight decay 1e-5), typically over 3–5 epochs (Wu et al., 28 Dec 2025). Early stopping is employed on L1 validation loss.

5. Operational Impact and System-Level Integration

By predicting output token length prior to decoding, the LAS module enables distributed LLM inference systems (e.g., Argus (Wu et al., 28 Dec 2025)) to schedule tasks with fine-grained workload awareness. This improves device utilization, reduces queuing delay, and drives higher system-wide Quality-of-Experience. Empirically, incorporating LAS length forecasts doubles reward in offloading scenarios and avoids resource misallocation under bursty, variable prompt loads.

In information retrieval, LA(SER)$^3$’s length-agnostic document embeddings yield robust retrieval across length-diverse documents, resisting semantic degradation from artificial elongation or “length attacks.” This supports deployment in systems facing heterogeneous input lengths without loss of retrieval accuracy (Xiao et al., 2023).

6. Theoretical and Practical Implications

Both LAS implementations demonstrate that length-awareness is a nontrivial inductive bias in neural NLP modeling. The LA(SER)$^3$ results in semantically robust, isotropic, and length-invariant representations, addressing previously unobserved vulnerabilities in vanilla contrastive pretraining. The Argus-style LAS, by focusing representational capacity on dimensions predictive of output length, achieves high accuracy with minimal trainable parameters and enhances distributed inference protocols.

This suggests a plausible implication: future LLM architectures and deployable systems may benefit from modular, plug-in LAS components tailored to either representation or inference time constraints. Selective feature recalibration and augmentation strategies offer principled solutions to length-induced semantic shift and operational inefficiency in large-scale NLP deployments.