Characterizing Model-Native Skills

Abstract: Skills are a natural unit for describing what a LLM can do and how its behavior can be changed. However, existing characterizations rely on human-written taxonomies, textual descriptions, or manual profiling pipelines--all external hypotheses about what matters that need not align with the model's internal representations. We argue that when the goal is to intervene on model behavior, skill characterization should be model-native: grounded in the model's own representations rather than imposed through external ontologies. We instantiate this view by recovering a compact orthogonal basis from sequence-level activations. The resulting basis is semantically interpretable but need not correspond to any predefined human ontology; instead, it captures axes of behavioral variation that the model itself organizes around. We validate this characterization on reasoning post-training, using the recovered basis for both SFT data selection and inference-time steering. We develop lightweight proxy interventions to identify which directions are most useful for a given model. Across Llama3-8B and Qwen2.5-3B, selecting data along those directions improves Pass@1 by up to 20% on MATH and 41% on AMC, outperforming data selection based on human-characterized skills. Because the basis lives in activation space, the same directions also serve as steering vectors at inference time, improving Pass@8 by up to 4.8% on MATH--an intervention that human-characterized skills cannot support. We further validate the characterization on safety alignment, where selecting adversarial training data for model-native skill coverage rather than textual diversity yields more sample-efficient learning. These results suggest that recovering skills from the model's own representations, rather than imposing them externally, provides a more effective foundation for intervening on model behavior. Codes are open-sourced.

• The paper introduces AutoSkill, a method that leverages PCA on sequence-level activations to reveal dominant, unsupervised model-native skills.
• The paper demonstrates that aligning interventions with internal activation geometry yields consistent empirical gains in data selection, steering, and safety robustness.
• The paper reports quantitative improvements such as up to 20% absolute Pass@1 gain on MATH-500 and 41% on AMC, validating the framework’s practical utility.

Characterizing Model-Native Skills: A Representational Framework for Data-Driven Skill Discovery and Intervention

Motivation and Limitations of Human-Defined Skill Ontologies

Skill characterization has become central in analysis, intervention, and aligned control of LLMs, particularly in domains such as mathematical reasoning and safety alignment. The dominant paradigm is to annotate or classify skills using human-designed taxonomies, ontologies, or textual capability descriptions. Such external frameworks, while valuable for interpretability and dataset curation, are fundamentally dissociated from the internal representations by which LLMs organize knowledge and support behavior. Consequently, manual skill definitions are often subjective, prone to redundancy, imperfect in coverage, and may fail to meaningfully partition the latent structure that emerges within large models. There is a clear need for frameworks that make skill analysis and downstream data optimization responsive to the model's own representational geometry, not just to external human hypotheses.

Model-Native Skill Characterization: The AutoSkill Method

This paper proposes that, for the purpose of model intervention—whether by data selection, inference-time control, or safety auditing—skill characterization must be "model-native": defined as prominent, often orthogonal axes in the model’s own activation space. The central framework, AutoSkill, instantiates this perspective via a pipeline grounded in principal component analysis (PCA) over sequence-level activations.

Given a pretrained model and a pool of solution traces or behavioral examples, sequence-level activations are formed by pooling hidden states across all layers for each example. This yields a high-dimensional matrix whose principal axes, extracted via PCA, are interpreted as the dominant directions of behavioral variability internal to the model. These model-native skills are unsupervised—no external skill labels or ontologies are imposed—and are both compact (low intrinsic dimensionality) and non-redundant by construction.

Crucially, given these axes, examples can be scored by cosine similarity along each direction, enabling ranking, selection, and grouping with respect to latent skills. The semantic interpretation of these axes is performed post hoc via examination of examples at opposing poles, often assisted by LLM-powered summarization, ensuring human interpretability without constraining the initial discovery.

Figure 1: Schematic comparing subjective, lossy human-defined skill taxonomies to PCA-extracted model-native skills that capture dominant, interpretable axes in latent space with minimal information loss.

Interpretability and Empirical Evidence from Reasoning Models

The interpretability of model-native skill axes is demonstrated by contrasting positive and negative poles (maximally positive and negative scoring examples) for each principal direction. Analysis reveals that directions often correspond to recognizable high-level contrasts, such as "symbolic computation vs. conceptual reasoning" or "elementary numeric calculation vs. advanced analytic argument." Importantly, the precise semantics of each axis depend on the model and trace pool, confirming that the discovered basis adapts to both model and data, not merely surface features.

Figure 2: Layerwise vector norm trends for principal directions show that lower layers weight more strongly in extracted axes.

Figure 3: Correlation between layerwise and global principal directions indicates that major axes are preserved across model depth, supporting interpretability and intervention from different abstraction levels.

Model-Native Intervention: Data Selection and Inference-Time Steering

Model-native skills form the foundation for both SFT data selection and causal intervention at inference.

• Data selection: By applying lightweight proxy scoring—either via pilot fine-tuning on small, axis-selected datasets or by direct inference-time steering—the utility of each principal direction can be efficiently assessed prior to full SFT. Subsequent SFT using data prioritized along “model-useful” axes generates consistent improvements over random sampling or selection based on manual skill labels. Notably, the method achieves up to 20% absolute Pass@1 gain on MATH-500 and 41% on AMC, outperforming STAT, s1.1, and LIMO, as well as approaches based on reward models, difficulty, or diversity heuristics.
• Inference-time steering: Because the axes are in activations, at test time the same vectors serve as steerable directions by bias injection or activation addition (without requiring contrastive supervision). AutoSkill supports seamless test-time control of solution style or approach, enabling local or global behavior modification with measurable impact (Pass@8 improvements up to 4.8% absolute on MATH-500). The empirical response is smooth in steering strength and interpretable by entropy shifts, confirming that interventions track model-native concepts rather than noise.

Application to Safety: Coverage-Optimal Adversarial Data Selection

The same model-native rationale applies in safety: adversarial attacks that are textually diverse may converge in the model’s latent space, leading to redundancy in taxonomy-driven data selection. By sampling adversarial examples using farthest-point coverage in the PCA-projected activation space, AutoSkill ensures maximal coverage of latent adversarial modes.

Compared to random selection, this simple latent coverage objective reliably improves sample efficiency: for budget sizes between 50 and 100, AutoSkill-based selection yields up to 37 more explicit refusals (wildjailbreak), indicating greater robustness after SFT under tight budgets. Notably, surface diversity is not a guarantee of effective coverage; only a model-native latent basis reliably ensures adversarial generalization across model updates and attack families.

Theoretical and Practical Implications

This work effectively demonstrates that the geometry of model-native activation space organizes behavioral primitives—skills—in a manner that is both compact and directly actionable. By removing the dependence on human-imposed ontologies, it frees intervention protocols (data selection, steering, safety, targeted generalization) from the ambiguities and redundancies endemic to natural language and human labeling. The framework is compatible with any architecture where sequence-level activations are accessible and can be deployed in a fully unsupervised fashion, making it highly practical for large-scale model analysis and editing.

Theoretically, this approach provides empirical validation for the linear representation hypothesis: functional concepts are encoded as dominant directions (or subspaces) in model activations. The practical implication is that skill discovery can and should be refocused on model-dependent decompositions—potentially rich, non-linear extensions (e.g., subspace methods or dictionaries) may further improve the expressivity and fine control possible in model steering and data curation.

Future Directions

While PCA provides an effective and interpretable basis, richer unsupervised decompositions (e.g., sparse coding, kernel methods, non-linear ICA) may pick up polysemantic or entangled skills, especially as model size and task complexity scale. There is also a need to systematize and automate the translation between model-native skill axes and actionable domain knowledge for more subjective domains (e.g., open-ended dialogue, multimodal settings). Further, the integration of model-native skill discovery with downstream RL and continual learning pipelines presents open questions regarding curriculum design, long-term transfer, and the anatomy of emergent reasoning in frontier LLMs.

Conclusion

The argument for model-native skills is supported by strong empirical gains across data selection, inference control, and adversarial robustness. By grounding all interventions in the structure emergent within the model's own representations, this framework unifies selection and steering under a geometry-aligned, unsupervised protocol. The implications are immediate for both applied pipeline optimization and theoretical study of representational structure in modern LLMs. Future research in skill discovery and alignment should strongly consider model-native, activation-based methods as the scalable backbone for robust and interpretable intervention.