Explain the dataset-specific crossover between SetFit and LoRA

Determine why the relative performance of SetFit and LoRA crosses over in opposite directions as the number of labeled examples increases on TREC and AG News, despite class count failing to explain the difference.

Background

SetFit consistently outperforms LoRA on the many-class Banking77 and CLINC150 datasets, but this ordering does not generalize to the few-class datasets. On TREC and AG News, the performance relationship changes with k and the crossover proceeds in opposite directions.

The authors directly test whether class count explains the crossover and reject that explanation: AG News has fewer classes than TREC but exhibits the opposite trend. They therefore identify the phenomenon as dataset-specific and unresolved rather than attributing it to the class-count mechanism used elsewhere in the paper.

References

We report this as a genuinely unresolved, dataset-specific finding rather than force a second class-count story the data does not support.

— Exact Degeneracy Under Balanced k-Shot Sampling:Consequences for Small-Sample Discriminant Analysis on LLM Embeddings  (2609.09860 - Qu, 9 Sep 2026) in Section 4.4, “Trained Baselines: SetFit, LoRA, and In-Context Learning”