IDIOLEX: Framework for Idiolectal Embeddings

• IDIOLEX is a framework for idiolectal representation learning that models sentence-level stylistic variation using a unified continuous embedding space.
• It employs provenance-based weak supervision and LLM-generated binary linguistic features to decouple style and dialect from semantic content.
• Evaluated on Arabic and Spanish dialect and authorship tasks, IDIOLEX outperforms baselines and improves LLM post-training alignment.

Searching arXiv for the cited IDIOLEX and related idiolect papers to ground the article in current literature. IDIOLEX is a framework for idiolectal representation learning: the construction of sentence representations that encode how a sentence is expressed rather than primarily what it means. It is designed to model style and dialect, decoupled from semantic content, by combining weak supervision from a sentence’s provenance with explicitly defined linguistic features of its content. In the formulation introduced in "IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation," the target object is a continuous representation of stylistic and dialectal variation that supports analysis, classification, and stylistic alignment of LLMs across Arabic and Spanish (Kantharuban et al., 6 Apr 2026).

1. Conceptual foundation

The term idiolect denotes “the unique and distinctive use of language of an individual,” and in computational authorship research it is treated as the theoretical foundation of authorship attribution (Perifanos et al., 2019). More recent work on short online texts frames idiolects as the personal, individual-level side of language variation, in contrast to sociolects, and argues that they are distinctive yet consistent rather than arbitrary (Zhu et al., 2021).

IDIOLEX adopts a broad notion of idiolectal variation. In this framework, idiolectal variation covers individual style, community/dialect variation, register and orthographic choices, and morphological and syntactic patterns. The objective is not merely to assign a sentence to a discrete author or dialect class, but to represent graded similarity among utterances. This is the sense in which the method is both unified and continuous: it seeks one embedding space for multiple levels of variation, and it treats stylistic relatedness as geometry rather than as a hard label inventory (Kantharuban et al., 6 Apr 2026).

This formulation responds to a limitation in standard sentence embeddings. Common transformer-based representations are typically optimized for semantic similarity and therefore map semantically similar sentences nearby even when their style, dialect, or authorial voice differ substantially. IDIOLEX is explicitly motivated by the claim that stylistic and dialectal signals are present in lexical choice, morphology, syntax, orthography, and discourse markers, but are often overshadowed by topical content in conventional embedding spaces (Kantharuban et al., 6 Apr 2026).

2. Supervision schema and data design

The training data for IDIOLEX consists of Reddit comments from regional language communities, collected from subreddit inception through Dec. 2024, segmented into sentences, filtered by language using fastText, filtered for low-quality examples, and restricted so that batch construction is feasible. Training and evaluation are performed separately for Arabic and Spanish, using regional subreddits for multiple Arabic dialect communities and multiple Spanish varieties (Kantharuban et al., 6 Apr 2026).

A central design choice is the use of provenance-based weak supervision. Each anchor sentence is paired with comparison sentences drawn from four levels of a proximity hierarchy:

• 3 = same comment
• 2 = same author, different comment
• 1 = same subreddit / same dialect community, different author
• 0 = different subreddit / different dialect community

This hierarchy operationalizes a continuum from highly local stylistic relatedness to community-level relatedness and then to contrastive cross-community dissimilarity. A plausible implication is that the framework treats authorial and dialectal signals as partially nested rather than as isolated tasks (Kantharuban et al., 6 Apr 2026).

IDIOLEX supplements provenance with LLM-generated binary linguistic features. The paper uses GPT-5-mini as a feature annotator and prompts it to return a JSON object of binary indicators stating whether a sentence contains certain explicitly defined surface cues. For Spanish, the feature inventory includes items such as explicit subject pronouns, voseo forms, 2nd-person plural endings, diminutive suffixes, clitic doubling, inverted punctuation, repeated punctuation, and orthographic variation. For Arabic, the inventory includes case endings, tanwin, future markers, dialectal negation, dialect-specific interrogatives, pronoun forms, copula patterns, dialectal lexical markers, orthographic variants, repeated punctuation, and laughter tokens. The paper characterizes these as approximate cues, not perfect annotations, but uses them to stabilize and interpret the embedding space (Kantharuban et al., 6 Apr 2026).

3. Architecture and objectives

IDIOLEX builds on a pretrained monolingual sentence encoder: AraBERT v2 for Arabic and BERTIN for Spanish. The representation-learning architecture is intentionally minimal. It uses no task-specific label heads at the representation-learning stage beyond the auxiliary objectives; instead, it derives a sentence embedding by taking hidden states from all transformer layers, applying learned layer-wise attention over layers, averaging token representations, and then mean-centering and L2-normalizing the resulting vector (Kantharuban et al., 6 Apr 2026).

The learned layer aggregation is defined as

$$\tilde{h} = \sum_{l=0}^{L} \alpha_l h^l,\qquad \alpha_l = \frac{\exp(w_l)}{\sum_{k=0}^{L} \exp(w_k)}.$$

The paper further applies VICReg-style constraints—centering, a variance constraint requiring each dimension to have standard deviation above 1, and a decorrelation penalty—to avoid collapse and anisotropy (Kantharuban et al., 6 Apr 2026).

Training proceeds in two stages. In Stage 1, the model uses all available Reddit sentences and only the proximity-based supervision. The objective is a margin ranking loss that encourages the anchor sentence to be closer to more stylistically similar examples than to less similar ones. The margin starts at 0 and is linearly increased to 0.5 during training (Kantharuban et al., 6 Apr 2026).

In Stage 2, performed on the subset of data with LLM-extracted linguistic features, the model adds two auxiliary objectives. First, a two-layer feature prediction head,

$$\text{Linear}(d,2d)\rightarrow \text{ReLU}\rightarrow \text{Linear}(2d,F),$$

predicts the binary feature vector with binary cross-entropy. Second, a supervised contrastive loss is weighted by Jaccard similarity between feature vectors, with temperature $T = 0.07$, so that sentences sharing more dialectal features are pulled closer. The combined feature loss is

$$\mathcal{L}_{\text{feat}} = 0.25\,\mathcal{L}_{\text{BCE}} + \mathcal{L}_{\text{supcon}},$$

and the overall objective is

$$\mathcal{L} = (1-\alpha)\mathcal{L}_{\text{rank}} + \alpha \mathcal{L}_{\text{feat}},$$

with $\alpha = 0.5$ (Kantharuban et al., 6 Apr 2026).

The intended effect of this design is not complete disentanglement of style and semantics; the paper explicitly states that some linguistic cues depend on meaning. Instead, the framework shifts the representation toward linguistic patterning and away from purely topical similarity (Kantharuban et al., 6 Apr 2026).

4. Empirical properties and downstream behavior

IDIOLEX is evaluated through downstream classification, similarity/clustering/retrieval analysis, and LLM post-training / alignment. For dialect identification, the paper uses DSL-ML 2024 for Spanish and MADAR-26 for Arabic. It reports that IDIOLEX outperforms prior neural and shared-task baselines on Spanish and reaches near state-of-the-art performance on Arabic. The paper also emphasizes that even the base IDIOLEX embeddings, combined with only a simple classification head, work well cross-domain, while fine-tuning and lexical ensembling further improve some results (Kantharuban et al., 6 Apr 2026).

For authorship attribution, the paper evaluates on PAN 2019 Spanish open-set authorship attribution and reports that IDIOLEX outperforms baselines by a meaningful margin and generalizes across domains better than comparison systems. This is presented as evidence that the embeddings capture authorial style rather than merely memorizing topic (Kantharuban et al., 6 Apr 2026).

Several analyses are used to characterize what the representations encode. When compared with semantic similarity scores from Multilingual-E5, IDIOLEX similarity shows a positive but weak correlation with semantic similarity: Pearson $r \approx 0.09$ for Spanish and Pearson $r \approx 0.19$ for Arabic. On MADAR parallel sentences, where content is controlled across dialects, the paper finds a clear gradient of same country > same region > different region, with statistically significant differences. This supports the claim that the learned space captures content-independent dialectal structure (Kantharuban et al., 6 Apr 2026).

The paper further notes that, for multi-label Spanish dialect identification, output probabilities reflect nuance: single-label examples get confident probabilities, whereas mixed-dialect examples show broader distributions. This suggests that the continuous geometry of the space is not reducible to a single categorical decision boundary (Kantharuban et al., 6 Apr 2026).

5. IDIOLEX as a post-training objective for LLMs

A distinctive aspect of the framework is its reuse as an alignment objective for LLM post-training. The paper argues that standard supervised fine-tuning on instruction-response pairs optimizes token prediction but does not explicitly enforce dialectal or stylistic fidelity. To address this, it precomputes the IDIOLEX embedding $e_i$ for each ground-truth response, pools the LLM hidden states for the response, projects them into the IDIOLEX space with a learned projection head $g_e$, and adds an embedding-matching term to the usual cross-entropy objective:

$$\text{Linear}(d,2d)\rightarrow \text{ReLU}\rightarrow \text{Linear}(2d,F),$$0

with $$\text{Linear}(d,2d)\rightarrow \text{ReLU}\rightarrow \text{Linear}(2d,F),$$1 (Kantharuban et al., 6 Apr 2026).

This post-training setup is applied to Llama 3.1 8B and Allam 7B using Arabic dialectal data from the AMIYA shared task and instruction-response pairs constructed from MADAR, FLORES-200, SauDial, PALM, HABIBI, EDC, Atlaset, MASC, and JODA. The paper uses both parallel translation pairs and LLM-augmented monolingual data turned into question-answer pairs, yielding about ~39k instruction-answer pairs from monolingual corpora plus ~10k translation pairs (Kantharuban et al., 6 Apr 2026).

Evaluation is reported with ADI2 for dialectalness or adherence and ChrF++ for translation quality. The paper states that adding IDIOLEX to supervised fine-tuning tends to improve dialectal alignment while preserving or improving fluency, especially for Moroccan, Egyptian, and Palestinian Arabic. It also notes a data dependence: dialects with more training data improve more, although lower-resource varieties can still benefit if related dialects are present (Kantharuban et al., 6 Apr 2026).

6. Relation to earlier idiolect modeling and to idiom-centered resources

IDIOLEX belongs to a line of work that treats personal language use as computationally measurable. Earlier author-level embedding approaches learned a vector per social-media user by aggregating all tweets from the same user into a single document and training Paragraph Vectors, specifically PV-DM, because word order matters for stylistic choices. On a Twitter corpus of 4,494 users, about 26,103,963 tweets, and about 325,243,302 words, this line of work reported accuracy consistently above 85% when scaling up to 4,000 users, and near 100% under top-$$\text{Linear}(d,2d)\rightarrow \text{ReLU}\rightarrow \text{Linear}(2d,F),$$2 matching; temporal experiments using $$\text{Linear}(d,2d)\rightarrow \text{ReLU}\rightarrow \text{Linear}(2d,F),$$3 pairs yielded accuracy around 80%, suggesting that idiolect is partly stable over time (Perifanos et al., 2019).

A second line of work studied short online texts with a Siamese transformer based on SBERT/SRoBERTa and authorship verification as a proxy task. On Amazon reviews and Reddit posts, with texts capped at up to 100 tokens, the best model (SRoBERTa) achieved accuracy 82.9%, F1 0.831, AUC 0.909 on Amazon and accuracy 73.0%, F1 0.737, AUC 0.812 on Reddit. Through perturbation and analogy-style probing, this work argued that idiolectal representations are regular, that stylistic shifts are highly aligned, and that individual writers are on average both consistent and distinctive (Zhu et al., 2021).

IDIOLEX differs from both approaches in two main respects. First, it shifts the unit of representation from authors or text pairs to the sentence. Second, it jointly models individual-level and community-level variation with continuous geometry and feature-anchored supervision rather than treating authorship attribution and dialect identification as unrelated discrete classification problems. This suggests a broader representational scope than author-fingerprint models alone (Kantharuban et al., 6 Apr 2026).

The name IDIOLEX can invite confusion with idiom lexicon work because of orthographic proximity to “idio-” terminology. However, the framework is not an idiom dictionary or idiom-sentiment lexicon. By contrast, EPIE is an English Possible Idiomatic Expressions corpus containing 25,206 sentences labeled with lexical instances of 717 idiomatic expressions, including both idiomatic and literal usages, and is explicitly designed for idiom span detection and metaphor or idiom detection (Saxena et al., 2020). CLIX defines a task of Cross-Lingual explanations of Idiomatic eXpressions for 628 English idiomatic expressions with explanations in English, Spanish, and German (Gluck et al., 6 Jan 2025). DualCoTs addresses sentiment lexicon expansion of idioms through EmoIdiomE, a Chinese-English dataset with 7,828 idioms and 741,135 example sentences / passages (Niu et al., 2024). These resources concern idiomatic expressions; IDIOLEX concerns idiolectal and stylistic variation (Kantharuban et al., 6 Apr 2026).

7. Limitations, interpretation, and significance

The paper identifies several limitations. It states that style and semantics are only partially separable, that provenance-based proximity is weak and imperfect, that LLM feature extraction is approximate, that Reddit data introduces dataset bias, and that performance is uneven across varieties, with lower-resource dialects benefiting less. It also includes an ethical caution: modeling dialect does not imply that systems should always generate in a user’s dialect, because user preference and context matter (Kantharuban et al., 6 Apr 2026).

Within these constraints, IDIOLEX advances a specific interpretation of idiolect: not as a single discrete fingerprint, and not as arbitrary noise, but as a continuous, compositional, and reusable representation problem. In this view, stylistic similarity reflects a structured hierarchy—same sentence/comment > same author > same community > different community—while explicit linguistic cues provide an interpretive scaffold for the learned space (Kantharuban et al., 6 Apr 2026).

A plausible implication is that IDIOLEX helps bridge several research traditions that had previously been studied in relative isolation: dialect identification, authorship attribution, style embeddings, and stylistically aligned generation. Its reported performance on external classification datasets, its weak correlation with generic semantic similarity, and its use as a post-training signal for dialectally aligned LLM behavior collectively support the paper’s broader claim that jointly modeling individual and community-level variation offers a useful perspective for studying idiolect and for building diverse and accessible LLMs (Kantharuban et al., 6 Apr 2026).