IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation

Abstract: Existing sentence representations primarily encode what a sentence says, rather than how it is expressed, even though the latter is important for many applications. In contrast, we develop sentence representations that capture style and dialect, decoupled from semantic content. We call this the task of idiolectal representation learning. We introduce IDIOLEX, a framework for training models that combines supervision from a sentence's provenance with linguistic features of a sentence's content, to learn a continuous representation of each sentence's style and dialect. We evaluate the approach on dialects of both Arabic and Spanish. The learned representations capture meaningful variation and transfer across domains for analysis and classification. We further explore the use of these representations as training objectives for stylistically aligning LLMs. Our results suggest that jointly modeling individual and community-level variation provides a useful perspective for studying idiolect and supports downstream applications requiring sensitivity to stylistic differences, such as developing diverse and accessible LLMs.

• The paper presents a novel framework that decouples stylistic and dialectal cues from semantic content by mapping sentences into a continuous idiolect space.
• It combines relational proximity signals with LLM-extracted linguistic features to achieve state-of-the-art performance in dialect identification and authorship attribution.
• The approach also enables post-training alignment for LLMs, enhancing stylistic fidelity while maintaining translation fluency.

IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation

Motivation and Problem Statement

NLP models and, in particular, LLMs predominantly encode and evaluate sentence meaning, often disregarding how the sentence is expressed—a critical aspect in applications involving personalization, dialectal adaptation, and stylistic alignment. While tasks such as dialect identification (DID) and authorship attribution (AA) utilize style and idiolectal cues, prior approaches are typically task-specific, rely on discrete labels, and do not generalize to continuous, multidimensional variation over style and dialect. The lack of robust, content-independent sentence representations for capturing idiolectal (style/dialect/register) properties restricts both analytical and practical progress, especially in multilingual and user-adaptive settings.

The IDIOLEX Framework

IDIOLEX introduces a representation learning paradigm specifically designed to capture stylistic and dialectal variation, decoupled from semantic content. The central objective is to map sentences into a continuous space—termed idiolectal space—where distances reflect relative proximity in style or dialect, regardless of semantics.

Training Paradigm

IDIOLEX relies on two complementary supervision signals:

• Relational/Proximity Signal: Sentences are organized hierarchically in terms of expected idiolectal similarity: same comment, same author, same dialect community, and different communities. Proximity scores between sentence pairs (0–3) are assigned based on these relations in user-generated corpora (primarily Reddit comments), with the intuition that closer relational ties correspond to greater stylistic/dialectal similarity.
• Feature-Level Signal: LLMs (specifically, GPT-5-mini) are prompted to extract binary linguistic features reflecting dialectal morphology, morphosyntax, lexico-orthographic patterns, and discourse cues (based on dialectological literature for Spanish and Arabic). These features provide weak supervision for the stylistic structure beyond relational labels.

Training Stages

The model architecture, built on top of pre-trained sentence encoders (AraBERT for Arabic and BERTIN for Spanish), is trained in two stages:

1. Proximity-based Pretraining: Margin ranking loss only, maximizing representational closeness for high-proximity pairs.
2. Feature-augmented Training: A subset of the data is annotated with linguistic feature vectors; two auxiliary objectives are combined with the main loss: binary cross-entropy for feature prediction and a supervised contrastive loss weighted by Jaccard feature similarity.

Normalization (mean-centering, L2 normalization), VICReg-style variance and decorrelation regularization, and layerwise attention pooling ensure the idiolectal embedding space is well-structured and resistant to collapse.

Figure 1: The IDIOLEX training framework combines margin ranking over proximity relations with LLM-extracted linguistic features and multiple regularization objectives.

Empirical Results

Dialect Identification and Authorship Attribution

IDIOLEX demonstrates robust performance as a general-purpose idiolectal encoder, surpassing specialized baselines on both dialect identification (DID) and authorship attribution (AA) across Spanish and Arabic datasets, and generalizing effectively in cross-domain scenarios.

On Spanish DID (DSL-ML), IDIOLEX outperforms all neural and non-neural baselines, including the top submission to VarDial 2024 (Sälevä & Palen-Michel). On Arabic DID (MADAR 26), it achieves an F1 of 0.66, approaching the best feature-engineered systems. In Spanish AA (PAN 2019), IDIOLEX models yield a 10% absolute improvement in exact match over the strongest baseline, despite highly challenging open-set, sentence-level conditions.

Figure 2: Classification metrics (F1 and EM) for DID tasks. IDIOLEX consistently outperforms BERT and E5 baselines; lexical ensembling yields minor additional gains.

Additionally, the continuous nature of the idiolectal space provides interpretability: sentences labeled with ambiguous dialectal cues ("both") fall between single-dialect clusters, as reflected in the model's likelihood distribution (see Figure 3).

Figure 3: IDIOLEX's likelihoods for Spanish DID (non-fine-tuned): label assignments smoothly reflect ambiguous or mixed input, supporting nuanced analysis.

Disentanglement from Semantic Content

Correlation analysis reveals that IDIOLEX representations have only a weak positive correlation with semantic similarity (as measured by Multilingual-E5), confirming that stylistic and dialectal information is not confounded with content semantics.

(Figure 4)

Figure 4: Scatter plot of IDIOLEX stylistic similarity vs. Multilingual-E5 semantic similarity; color indicates relational grouping (author, dialect, etc). Pearson $r=0.09$ (Spanish), $r=0.19$ (Arabic) evidences weak coupling.

Crucially, in MADAR-26, where sentences are parallel translations (thus holding content constant), stylistic similarity grades closely with dialectal/geographic proximity, further evidencing dialectal signal capture independent of semantics.

(Figure 5)

Figure 5: On semantically-controlled data, IDIOLEX similarity discriminates dialectal proximity, with higher scores for same-region pairs.

Post-Training Alignment for LLMs

A significant contribution of the work is demonstrating that IDIOLEX can serve as a differentiable, continuous training objective for LLM instruction-tuning. In post-training, the LLM's final hidden states (mean-pooled and projected onto the IDIOLEX space) are aligned with ground-truth reference responses via a cosine similarity term in the loss function, jointly optimized with standard cross-entropy.

On the AMIYA shared task, augmenting SFT with IDIOLEX alignment consistently improves dialectal fidelity (ADI2) without detriment to translation fluency (ChrF++), outperforming systems reliant on discrete dialect labels and reducing the need for dialect-specific fine-tuning. Strongest gains are observed for dialects with ample monolingual data, but generalization is preserved in low-resource scenarios via related-dialect transfer.

Theoretical and Practical Implications

IDIOLEX provides a paradigm shift from discrete, label-centric style models toward continuous, compositionally structured representations. This architecture-agnostic and language-agnostic approach enables nuanced modeling of variation within and across languages, facilitating:

• Improved downstream performance for DID and AA, especially in low-resource or ambiguous cases.
• Content-independent style embeddings suitable for clustering, retrieval, and analytic tasks in sociolinguistics, forensics, and personalization.
• Differentiable objectives for stylistic alignment in LLMs, allowing nuanced control during post-training without rigid label assignment.
• Cross-domain generalization and robustness to topical variation, unattainable with conventional baselines.

Future research may extend IDIOLEX to more languages and registers, integrate more granular speaker meta-data, or investigate dynamic adaptation in conversational settings (e.g., dialogue-level style accommodation, user-level personalization). Integration into RLHF pipelines or other preference-based post-training strategies is another promising direction, as is exploring the impact on fairness and inclusivity in LLM deployment.

Conclusion

IDIOLEX is a generalizable framework for learning continuous, content-independent representations of idiolectal and stylistic variation, combining weak supervision from relational structure and LLM-extracted linguistic features. Empirical results demonstrate state-of-the-art performance on dialect and author classification, effective disentanglement from semantic content, and practical benefits for post-training LLM stylistic alignment. By operationalizing idiolect as a continuous phenomenon, IDIOLEX expands the methodological and practical landscape for style-sensitive NLP.