- The paper presents an empirically-derived taxonomy of SKILL.md files, revealing 13 high-level and 44 lower-level semantic components in agent skills documentation.
- The paper develops an automated detector that found over 99% of SKILL.md files contain at least one skill smell, highlighting prevalent documentation issues.
- The paper shows that persistent documentation quality issues in SKILL.md files can degrade agent performance, urging the need for automated remediation tools.
Empirical Analysis of SKILL.md Authoring Practices in Agent Skills
Introduction and Motivation
This paper presents a comprehensive empirical study on SKILL.md files, which serve as critical metadata and documentation artifacts for Agent Skills used by LLM agents. Unlike code-level skill components, SKILL.md is a human-authored, unstructured, Markdown-based documentation format adopted to describe and register agent skills without requiring LLM retraining. The file includes essential metadata and discretionary information directly authorable by skill developers, leading to significant heterogeneity in its content and structure. Given the increasing adoption of Agent Skills, understanding the quality, consistency, and adherence to authoring guidelines of SKILL.md files is vital for downstream agent selection, maintenance, and usability.
Methodology and Taxonomy Construction
The paper undertakes a systematic qualitative analysis over a curated corpus of 238 publicly-available SKILL.md files, covering diverse agent domains. Through open coding and thematic analysis, it constructs a taxonomy comprised of 13 higher-level and 44 lower-level semantic components, capturing the empirical range and frequency of documentation patterns observed in the wild. This taxonomy provides the first structured lens on SKILL.md composition, revealing common, rare, and missing elements across the ecosystem.
The investigation is deepened by a multivocal literature review encompassing 29 sources—including official documentation, community guidelines, and related empirical studies on documentation and metadata practices in software engineering. This review distills actionable best practices for authoring SKILL.md files, which are operationalized into a catalog of "skill smells": explicit anti-patterns or omissions constituting violations of recommended authoring practice.
Automated Detection and Quantitative Results
Building on the derived taxonomy and best practices, the authors develop and validate an automated detector for skill smells in SKILL.md files. Applying this tool on their dataset produces several noteworthy quantitative findings:
- Prevalence: Over 99% of SKILL.md files exhibit at least one detectable skill smell.
- Persistence: Skill smells, once introduced, are rarely removed over the evolutionary history of a skill, indicating a lack of remediation during maintenance.
- The most frequent smells involve missing critical metadata, inconsistent terminology, poor structuring, and inadequate usage examples.
These results indicate a pervasive quality gap between recommended SKILL.md authoring practices and those observed in practice, with significant implications for the LLM agent tooling ecosystem.
Implications and Theoretical Impact
The strong prevalence and persistence of skill smells have broad implications. Practically, suboptimal SKILL.md documentation can degrade the discoverability, interpretability, and safe composability of Agent Skills, limiting their reusability and integration by downstream agent designers or users. Theoretically, the documentation quality issues uncovered raise concerns regarding the sustainability of the Agent Skill model as skill repositories scale, and suggest parallels to technical debt in code documentation.
This work motivates automated and semi-automated documentation quality assurance as a foundational research and tooling direction for agent ecosystems. The introduction of skill smell detectors constitutes an initial step, but the authors highlight the need for developer-facing remediation tooling, more precise guidelines, and possibly machine learning-based authoring assistance.
Future Directions
Potential developments stemming from this research include:
- Toolchains capable of not only detecting but providing context-aware, actionable suggestions to remediate skill smells.
- Integration of documentation quality metrics into agent evaluation, ranking, and skill recommendation systems.
- Automatic extraction and standardization of semantic components in SKILL.md authoring workflows, potentially leveraging LLMs for draft suggestion and review.
- Longitudinal studies correlating documentation quality with skill adoption, maintenance overhead, and user satisfaction.
Conclusion
This paper delivers the first empirically-grounded taxonomy of SKILL.md components, a rigorously validated catalog of documentation anti-patterns ("skill smells"), and an automated detection framework applied to a large, real-world dataset. The pervasive and persistent nature of skill smells revealed exposes a key challenge in scaling agent skills for LLM ecosystems. Addressing the identified quality gap through automated and sociotechnical intervention promises substantial improvements in the maintainability, accessibility, and utility of agent skill repositories (2607.01456).