LLM-Based Classification Scheme

• LLM-based classification schemes are end-to-end pipelines that use pretrained language models to assign structured labels, demonstrating high-precision semantic reasoning.
• They incorporate techniques like structured prompt engineering, taxonomy integration, and hybrid ensembling to optimize performance in multi-label and multi-modal tasks.
• Applied across domains such as e-commerce, law, and social sciences, these schemes enable rapid deployment without extensive domain-specific fine-tuning.

A LLM-based classification scheme refers to any end-to-end pipeline in which a large, pretrained neural LLM serves as the core engine for assigning structured labels—single or multiple, flat or hierarchical—to free-form or structured inputs. LLM-based classification approaches are increasingly utilized across scientific domains, e-commerce, law, social science, and digital libraries owing to their capacity for high-precision semantic reasoning, flexible prompt-based adaptation, and rapid deployment without intensive domain-specific fine-tuning. These schemes incorporate structured prompt engineering, taxonomy-driven output heads, hybrid model ensembling, and innovative techniques for efficiency, transparency, and control.

1. Architectural Paradigms in LLM-Based Classification

LLM-based classification schemes leverage a wide array of paradigms, differentiated along several axes:

• Instruction, In-Context, or Structured Prompting: LLMs can operate with zero-shot or few-shot prompts, sometimes embedding entire taxonomies or multi-stage few-shot demonstrations in their input. For instance, in product taxonomy classification, the prompt concatenates definitions, k-shot demonstrations, a listing of valid categories, and explicit instructions to output the class label as plain text (Gholamian et al., 2024). In political science, the PoliPrompt pipeline involves automatic extraction of rules and dynamic exemplar selection, with prompts optimized through map–reduce analysis of labeled data (Liu et al., 2024).
• Fine-Tuning and Output Mapping: Some architectures fine-tune LLMs for classification, often via adapters or low-rank update modules, mapping class labels to output tokens or phrase spans. The SALSA approach maps each class to a single output token, prompts for a structured response, and restricts output logits to the known label instances for efficiency and calibration (Berdichevsky et al., 26 Oct 2025).
• Taxonomy-Awareness and Consistency Enforcement: Hierarchical or multi-level classification challenges are addressed by taxonomy-refining frameworks (e.g., TaxMorph (Golde et al., 26 Jan 2026)), taxonomy-embedded transitional classifiers (TTC) that enforce consistency across class levels via transition matrices (Chen et al., 12 Jan 2025), or methods that iteratively expand label hierarchies using LLMs in a human-in-the-loop workflow (You et al., 22 Aug 2025).
• Multi-Label and Multi-Modal Extensions: Multi-label tasks are managed via dichotomic prompting—decomposing K-way classification into K binary (yes/no) decisions for each target label, combined with prefix caching for inference efficiency (Langner et al., 5 Nov 2025). Multimodal classification, as in StarWhisper LightCurve (LLM+image/LLM+audio) (Li et al., 2024), exploits sequential or joint fusion of text, image, and audio streams within or before the LLM classifier.
• Randomness and Ensembling: LLMs’ inherent stochasticity can be harnessed by aggregating multiple outputs over random seeds, top-k/p sampling, or prompt ensembles. For legal privileged-document detection, a methodology is described wherein multiple runs are aggregated via a configurable threshold on label frequency, thereby trading off recall and precision according to workflow requirements (Huffman et al., 8 Dec 2025).

2. Taxonomy-Driven and Hierarchical Classification Strategies

LLM-based schemes for hierarchical or taxonomy-driven tasks incorporate explicit knowledge of label structures:

• TaxMorph Framework: Input taxonomies are transformed by LLMs using renaming, splitting, merging, or reordering operations, producing refined structures more aligned with the LLM’s inductive biases, which empirically improve macro-F1 by up to +2.9 pp over human-curated trees (Golde et al., 26 Jan 2026).
• Transitional Classifiers: The TTC layer, agnostic to LLM architecture, injects taxonomy structure by gating logits at each hierarchical level with attention-like masks induced from parent-class scores and predefined binary transition matrices. This mechanism enforces hierarchical consistency for multimodal classification, boosting both exact match and fine-grained accuracy (Chen et al., 12 Jan 2025).
• Semi-Supervised Iterative Expansion: Hierarchical classifiers can be grown via LLM-in-the-loop recursion: high-level class definitions are refined through prompt iteration and joint topic-model alignment, then recursively partitioned, with chain-of-thought prompting inducing labels at each child tier (You et al., 22 Aug 2025).

3. Prompt Engineering, In-Context Learning, and Dynamic Exemplar Selection

Effective prompt design is fundamental to LLM-based classification:

• Structured, Multi-Stage, and Consensus Prompts: Many pipelines utilize multi-stage prompting, e.g., generating candidate keywords, mapping them to subject taxonomies, and re-ranking with relevance scores as in LLMs4Subjects (D'Souza et al., 9 Apr 2025). Automatic prompt optimization is achieved by extracting rules from exemplars and using MMR-based dynamic exemplar selection for query-adaptive, contextually relevant prompting (Liu et al., 2024).
• Chain-of-Thought and Reasoning-Inspired Templates: Certain domains benefit from explicit reasoning steps. In legal article prediction, prompts are crafted to extract conditions, commands, and requirements (syllogism structure), requiring the LLM to map fact segments to legal components, and reasoning through degree-of-match for final label applicability (Chi et al., 26 Sep 2025).
• Mitigating Sequence and Recency Bias: Prompt and exemplar ordering can introduce biases. Validations such as statelessness tests and shuffling of few-shot example order are employed, with drift and recency bias monitored statistically and corrected via prompt iteration (You et al., 22 Aug 2025).

4. Efficient and Robust Multi-Label, Multi-Modal, and Graph-Structured Classification

LLM schemes address efficiency and robustness through multiple mechanisms:

• Dichotomic Prompting with Prefix Caching: Reformulating multi-label tasks as a sequence of independent yes/no queries per label dramatically increases inference efficiency, particularly when combined with caching of expensive prefix computations as implemented in vLLM-like backends (Langner et al., 5 Nov 2025).
• Distillation Pipelines: High-capacity LLMs (teacher) generate multiple pseudo-labels, which are aggregated to fine-tune efficient smaller models (students), thus preserving accuracy on seen dimensions and enabling robust generalization to out-of-distribution labels (Langner et al., 5 Nov 2025). The framework is domain-agnostic, scaling to any problem where labels can be mapped to dichotomic decisions.
• Node Classification in Graphs: LLMs can serve as encoders or reasoners in node classification benchmarks. The integration of LLM text embeddings and structural context via GNNs can yield 5–8% higher accuracy than classic GNNs in semi-supervised, label-sparse scenarios; reasoning-inspired LLMs excel with text-driven homophilic datasets (Wu et al., 2 Feb 2025).
• Product Classification Under Input Perturbations: LLMs using in-context learning and robust prompt design substantially surpass supervised baselines in both clean and adversarial settings, e.g., in product HS code classification GPT-4 maintains macro-F1 of 85.2% after "combined" data attacks, in contrast with a 52.8% drop for DeBERTaV3 (Gholamian et al., 2024).

5. Hybridization, Calibration, and Controllable Output

Integration of LLMs with classical models or explainability modules is increasingly prevalent:

• Hybrid Ensembles and Calibrators: LLM outputs are linearly or adaptively combined with classical ML model scores, yielding statistically significant gains in test accuracy. Calibration by multi-accuracy (post-hoc correction of ML estimator residuals conditional on LLM outputs) further corrects group-wise bias and yields robust thresholds under domain shift (Wu et al., 2024).
• Controllable Classification with Sparse Autoencoders: To address the opacity of LLM embeddings and their tendency to encode undesirable features, sparse autoencoders are trained to disentangle and identify “unintended” latent features, informed by LLM-based semantic judgments. Classification heads are regularized (via $\|\theta^\top W_-\|_1$) to penalize reliance on such features, providing a route to privacy and fairness-improving interventions (Wu et al., 19 Feb 2025).
• Local Explainability Without Perturbation: PLEX introduces a perturbation-free mechanism for word-level importance assessment by aligning Siamese networks to attribution scores from LIME/SHAP, delivering $\approx$92% agreement with classical XAI methods at several orders-of-magnitude lower compute cost (Rahulamathavan et al., 12 Jul 2025).

6. Metrics, Benchmarks, and Empirical Evaluation

Evaluation protocols are tailored to task structure and data scale:

• Standard Metrics: Macro-F1, micro-F1, precision and recall at various cut-offs (e.g., Recall@k for subject tagging), exact match, consistency (for taxonomy-respecting predictions), and cross-entropy loss for multi-way outputs are employed universally (Chen et al., 12 Jan 2025, D'Souza et al., 9 Apr 2025).
• Specialized Metrics: For hierarchical tasks, measures such as Taxonomy Probing Metric (TPM), parent-child similarity (PCS), and embedding-space clusterability are used to quantify both label-separability and alignment with model confusion matrices (Golde et al., 26 Jan 2026).
• Empirical Insights: Off-the-shelf or weakly tuned LLMs consistently achieve competitive performance with strong classical baselines. For example, SALSA achieves 95.9% accuracy on AG’s News vs 95.5% for XLNet and state-of-the-art results on three of seven GLUE test tasks (Berdichevsky et al., 26 Oct 2025). In multi-modal hierarchical classification, TTC integration yields +10–23 points gain in leaf-level accuracy and +15–24 in exact match (Chen et al., 12 Jan 2025). Second-order innovations, such as LLM-refined taxonomies or collaborative LLM+SCM frameworks (Uni-LAP), deliver consistent multi-point absolute F1 gains across diverse domains (Golde et al., 26 Jan 2026, Chi et al., 26 Sep 2025).
• Qualitative Evaluation: Expert judges in structured evaluations (e.g., legal domain, library records) confirm increased relevance and interpretability of LLM-generated labels and explanations compared to both strong transformer baselines and prior LLM variants (Johnson et al., 12 Apr 2025, D'Souza et al., 9 Apr 2025).

7. Practical Recommendations, Limitations, and Future Directions

• Prompt Engineering: Develop prompts tailored for classification structure; leverage full-context taxonomies and explicitly enumerate candidate sets for transparency and output control.
• Taxonomy Integration: Always embed known hierarchies into the output head, whether by explicit transition/gating matrices or LLM-extracted path refinements, to improve label consistency and calibration.
• Model Selection: For scenarios with limited labeled data or high class granularity, combine LLMs with retrieval and distillation pipelines to preserve accuracy and efficiency.
• Randomness and Ensembles: Use randomized outputs in an aggregation strategy (multiple seeds, sampling temperatures, prompt variants) to trade recall for precision and supply calibrated confidence scores, especially critical in compliance-sensitive applications (Huffman et al., 8 Dec 2025).
• Controllability and Explainability: Apply feature disentanglement and regularization (e.g., sparse autoencoders) to improve trustworthiness and avoid reliance on sensitive or spurious features, and employ robust local XAI methods for interpretability at scale.
• Limitations: Challenges include large inference cost and context-length limits of current LLMs, potential output hallucinations, and dependency on prompt/label engineering for datasets with evolving or ambiguous classes. Lack of explicit uncertainty quantification remains a common limitation.

Continued evolution is expected toward hybrid pipelines, richer taxonomy refinement (for DAGs and non-tree graphs), multi-modal/multi-task adaptation, and deeper integration of prompt optimization and label-space consistency constraints (Golde et al., 26 Jan 2026, Chen et al., 12 Jan 2025, Huffman et al., 8 Dec 2025).