SLIM-LLMs: Lean Interpretable Models

• SLIM-LLMs are a modeling paradigm that leverages a minimal set of human-interpretable stylometric features for transparent text classification and authorship attribution.
• They integrate LLM-based annotation and engineered feature pipelines to reduce dimensionality while maintaining near state-of-the-art performance.
• This approach facilitates practical applications like academic integrity detection and scientific peer review by enabling efficient rule extraction and clear model explainability.

A Stylometrically Lean Interpretable Model (SLIM-LLM) is a modeling paradigm in which a small set of human-interpretable linguistic or stylistic features, extracted either via LLM annotation or engineered stylometric pipelines, are used as the exclusive or central input to transparent white-box models for tasks such as text classification, authorship attribution, or rule learning. SLIM-LLMs contrast with high-dimensional embedding- or bag-of-words-based representations, enabling end-to-end interpretability, parameter efficiency, and compact rule extraction. The SLIM-LLM framework has emerged both in the context of LLM-assisted feature annotation for research texts and in the stylometric analysis of both human and machine-generated language (Balek et al., 2024, Khalid et al., 4 Aug 2025, Przystalski et al., 1 Jul 2025, Hicke et al., 5 Feb 2025).

1. Definitional Scope and Motivation

The core principle of Stylometrically Lean Interpretable Models is the use of a sharply reduced and semantically meaningful feature set—typically tens rather than hundreds or thousands of dimensions. This lean strategy is motivated by two primary shortcomings observed in conventional text representation approaches:

• Dense embeddings (e.g., BERT, SciBERT, etc.) and large n-gram vectors lack direct interpretability at the feature level, impeding extraction of transparent rules or actionable insights.
• High-dimensionality results in opaque classifiers and hinders downstream rule-mining due to combinatorial explosion in the feature space (Balek et al., 2024).

SLIM-LLMs address these limitations by leveraging either: (a) LLM-guided annotation using prompt-based extraction of high-level document qualities (e.g., methodological rigor, disciplinary assignment) or (b) linguistically motivated stylometric features with automatic selection of the most discriminative subset (Przystalski et al., 1 Jul 2025, Hicke et al., 5 Feb 2025).

2. Feature Engineering and LLM-based Annotation

LLM-Prompted Feature Generation

SLIM-LLM pipelines such as in (Balek et al., 2024) prompt open LLMs (e.g., LLama 2 13B GPTQ-Chat) with deterministic decoding to annotate documents, e.g., abstracts of scientific articles, with values for high-utility features:

• Abstract quality: rigor, novelty, accessibility, reproducibility, grammatical correctness
• Research methodology: 16 categories (e.g., experimental, qualitative, quantitative, etc.)
• Discipline: 41 binary flags corresponding to standardized research fields

The prompt returns a JSON object per document, with all features directly human-interpretable. The total number of features per document is 62, enabling downstream use in white-box models and rule-mining frameworks.

Traditional Stylometric Feature Suites

Stylometrically motivated pipelines (e.g., StyloMetrix, CLARIN-PL) engineer hundreds of interpretable features, including vocabulary size, type-token ratio (TTR), function-word frequency, POS n-grams, dependency-based syntactic bigrams, narrative measures, punctuation and fronting ratios (Przystalski et al., 1 Jul 2025). However, SLIM-LLMs distill only the most discriminative subset for compactness—often no more than 4–10 features suffice for high-accuracy binary discrimination.

3. Dimensionality Reduction and Latent Style Modeling

A significant line of SLIM-LLM research involves latent factor reduction of interpretable, psychologically motivated feature spaces. In (Khalid et al., 4 Aug 2025), Reduced-Rank Ridge Regression (R4) compresses 74 LIWC categories into 24 latent style axes, optimizing the objective

$$\min_{U, V} \ \tfrac{1}{2}\|Y - XUV^\top\|_F^2 + \lambda\sum_{j=1}^m \|U_j\|_2^2$$

where $X$ encodes stylometric vectors and $Y$ is the prediction target. The learned embedding axes correspond to function words, pronouns, informal language, biological terms and other psycholinguistic clusters, which can be visualized by t-SNE or heatmap over genres.

Projection of masked-LLM outputs into this low-rank style space, concatenated with R4-encoded LIWC vectors, enables SLIM-LLM variants that match or marginally exceed full-model BERT performance on sensorial-language prediction with up to 80% parameter reduction.

4. Classifier Formulation and Evaluation

Downstream models for SLIM-LLMs include decision trees, (gradient) boosted trees (e.g., LightGBM), logistic regression, SVMs, and shallow neural networks. White-box classifiers trained exclusively on the selected feature set attain accuracy comparable to deep models:

• In the CORD-19 and M17+ datasets (Balek et al., 2024), gradient-boosted models trained on 62 LLM-generated features achieve 95% of SciBERT (768-dim) performance.
• For stylometric LLM-vs-human discrimination (Przystalski et al., 1 Jul 2025), LightGBM using only 4–8 features obtains 0.98 accuracy (binary, Wikipedia vs GPT-4) and multiclass Matthews correlation coefficient (MCC) up to 0.87.

Validation employs test-time accuracy, F1, and MAE (for ordinal targets), using cross-validation schemes such as group stratification by topic (Przystalski et al., 1 Jul 2025).

5. Feature Selection, Explainability, and Rule Discovery

To ensure only causal or target-associated features are retained, statistical filtering employs the chi-square test of independence for categorical features (significance at $p < 0.05$) and, where appropriate, Pearson’s correlation for ordinal dimensions (Balek et al., 2024). Shapley Additive Explanations (SHAP) identify the most influential features, guiding forward selection until desired accuracy is reached (Przystalski et al., 1 Jul 2025).

Following classifier training, action-oriented rule mining leverages Apriori or its variants:

• Classification rules have the form $\phi \Rightarrow \psi$, with support and confidence computed over the corpus.
• Action rules specify attribute changes associated with improved target outcomes, formally evaluating uplift as the differential probability of the desired outcome following a “treatment” (i.e., an attribute transformation).

Representative minimal rule sets may rely on features such as comparative adjective frequency, function-word counts, and fronting ratios. Example: “If $f_{\text{adj\_comp}} > 0.03$ and $f_{\text{TTR}} > 0.55$, label as Wikipedia” (Przystalski et al., 1 Jul 2025).

6. Cross-Domain Generalization and Parameter Efficiency

A key differentiator of SLIM-LLMs is their transferability:

• The same LLM-generated feature schema applies effectively to disparate domains—e.g., COVID-19 scientific articles and national research evaluation summaries—without corpus-specific tuning (Balek et al., 2024).
• Gradient-boosted SLIM-LLMs reach 95%–100% of full embedding-based model accuracy in tasks as varied as sensorial-word prediction (five genres) and scientific citation/classification, with up to 80% reduction in trainable parameters when factoring transformer weights by the SLIM rank parameter (Khalid et al., 4 Aug 2025).

This suggests SLIM-LLMs admit broad generalization, maintaining interpretability and computational efficiency across distinct registers.

7. Stylometric Insights, Limitations, and Implementation Considerations

Empirical probing (e.g., syntactic ablation, attention-based token scoring) highlights which features drive attribution accuracy in SLIM-LLMs:

• Sequence-level features (n-gram ordering, word-order preservation) are most crucial; shuffling incurs largest accuracy drop.
• Pronoun frequency and gendered terms are persistent stylometric discriminators for authorship.
• Both high-frequency function words and low-frequency, class-specific content words are diagnostically valuable.
• Features such as punctuation frequency and contextual embedding divergence co-vary with model attribution robustness (Hicke et al., 5 Feb 2025).

A practical SLIM-LLM workflow involves:

1. Feature selection (5–10 semantically-rich dimensions, or top stylometric features via SHAP/ablation)
2. Deterministic LLM annotation or pipeline extraction for each document
3. Statistical validation of feature–target association
4. Training of a white-box or hybrid classifier
5. Rule extraction if required for policy/action recommendations

A plausible implication is that the SLIM-LLM strategy enables interpretable AI in domains where explainability and actionable insights are paramount, such as scientific peer review, academic integrity detection, and model auditability.