Unsupervised ID-based Algorithms for Comparing Models, Task Complexity, and Generated Text

Develop an unsupervised algorithm that leverages intrinsic dimension (ID) estimates of hidden representations in transformer-based large language models to perform (i) model comparison, (ii) task complexity comparison across datasets, and (iii) generated text comparison, operationalizing ID as the core metric for these comparative analyses.

Background

The paper investigates the geometry of decision-making in transformer-based LLMs by estimating intrinsic dimension (ID) across layers and tasks. The authors find consistent hump-shaped ID trends aligned with the emergence of decisiveness and propose that ID can serve as a proxy for representational focus and task commitment.

While they observed some negative correlations between final-layer ID and accuracy within specific model families, these results were not universally robust across all models and datasets. Motivated by these findings, the authors explicitly note that turning ID into a concrete, unsupervised algorithm for comparative purposes—across models, tasks, and generated text—remains unresolved and is a direction for future work.