Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models (2410.13826v2)

Abstract: With models getting stronger, evaluations have grown more complex, testing multiple skills in one benchmark and even in the same instance at once. However, skill-wise performance is obscured when inspecting aggregate accuracy, under-utilizing the rich signal modern benchmarks contain. We propose an automatic approach to recover the underlying skills relevant for any evaluation instance, by way of inspecting model-generated rationales. After validating the relevance of rationale-parsed skills and inferring skills for $46$k instances over $12$ benchmarks, we observe many skills to be common across benchmarks, resulting in the curation of hundreds of skill-slices (i.e. sets of instances testing a common skill). Inspecting accuracy over these slices yields novel insights on model trade-offs: e.g., compared to GPT-4o and Claude 3.5 Sonnet, on average, Gemini 1.5 Pro is $18\%$ more accurate in "computing molar mass", but $19\%$ less accurate in "applying constitutional law", despite the overall accuracies of the three models differing by a mere $0.4\%$. Furthermore, we demonstrate the practical utility of our approach by showing that insights derived from skill slice analysis can generalize to held-out instances: when routing each instance to the model strongest on the relevant skills, we see a $3\%$ accuracy improvement over our $12$ dataset corpus. Our skill-slices and framework open a new avenue in model evaluation, leveraging skill-specific analyses to unlock a more granular and actionable understanding of model capabilities.

• The paper introduces a novel method leveraging model-generated rationales to systematically decode skill-level performance across benchmarks.
• It validates 278 distinct skill-slices using 46,000 examples with 94.1% automated verification and strong human agreement.
• Findings reveal specific trade-offs among models, enabling skill-dependent routing that achieves up to 3.2% accuracy gains.

Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models

This paper explores a novel approach to evaluating foundation models by examining skill-level insights that illuminate fine-grained trade-offs masked by aggregate accuracy measures. As modern benchmarks test multiple abilities simultaneously, it becomes clear that aggregate accuracy can obscure a model's strengths and weaknesses. The presented work introduces a systematic method to decode skill-level performance, allowing for a more granular understanding of model capabilities. This is achieved through automatic inference of skills via rationale-generated analysis, offering an enriched evaluation framework and opening new evaluative directions.

Methodology and Validation

The paper introduces a methodology that leverages model-generated rationales to infer relevant skills for evaluation instances. By parsing rationales—step-by-step solutions provided by powerful models like GPT-4o—the authors identify skills involved at each rationale step. This approach systematically divides evaluation tasks into finer skill-slices, which are collections of instances across benchmarks requiring a common skill. Using 46,000 examples from 12 benchmarks, the paper uncovers 278 distinct skill-slices of substantial size, highlighting shared skills across different tasks and domains.

To validate skill relevancy, the authors employ post-hoc verification and inter-annotator agreement, with a notable 94.1% of skills verified as relevant by automated checks, and a confirmatory human verification aligning closely. Skill granularity, addressed through clustering similar skills, further enhances reliability by ensuring consistency and relevance.

Insights into Models' Skill Proficiency

The skill-slice analysis facilitates nuanced insights into leading models—GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet—revealing significant trade-offs and capability differentials. Despite similar overall accuracy across these models, the paper identifies sizeable differences in specific skills. Gemini 1.5 Pro demonstrates a distinct advantage in mathematics and science skills, while GPT-4o shows greater strength in tasks requiring visual processing like color differentiation and object localization. Meanwhile, Claude 3.5 Sonnet improves notably on legal reasoning over its predecessors.

The authors emphasize the generalizability of skill-slice insights. Through skill-dependent routing of benchmark instances to the most suitable model, they achieve measurable accuracy gains—up to 3.2% across multiple datasets—demonstrating the potential for practical deployment of these insights into model selection strategies.

Probing Questions and Future Implications

The paper also corroborates skill-slice findings using skill-specific probing questions derived from rationale steps. This secondary evaluation, focusing on consistency in model outputs, confirms models’ proficiency on isolated skills. This dual approach ensures robustness, as low accuracy skill-slices generally correlate with high inconsistency in probing, reinforcing the validity of skill-level insights.

Conclusion

This research underscores the importance of moving beyond aggregate model performance metrics to capture nuanced skill-level capabilities. The extraction of skill-specific slices from rationales exemplifies a shift towards more interpretable and actionable evaluation methodologies. Such perspectives are especially invaluable as models increasingly handle multifaceted, real-world problems requiring diverse competencies. By advocating for skill-specific evaluations, this paper suggests promising avenues for more targeted model improvements and better understanding of emergent model properties, paving the way for richer AI development and deployment strategies. As the landscape of foundation models continues to expand, such it embodies a critical step in aligning model assessment with their growing complexity and multifaceted utility.