Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows

Abstract: Product data scientists often ask LLM-based agents to help with recurring execution tasks such as cleaning data, writing SQL, choosing statistical tests, and formatting results. Reusable skill files are meant to avoid prompting from scratch by packaging guidance for a task family. Expert-written skills can encode high-quality guidance, but writing and maintaining them across many data-science task families creates a manual bottleneck. We ask whether LLM-generated skills offer a useful low-curation alternative: do they improve performance over the task prompt alone? We test this question across four lifecycle stages: data preparation, data extraction, statistical analysis, and reporting, using one generated skill per stage. We find no reliable improvement from full generated skills over No-Skill prompting. We then ask whether any part of the skill is useful by ablating different skill components. The main ablation covers 56 tasks, nine model configurations, and three providers, yielding 7,560 runs. Compared with prompting using the task alone, neither the full generated skill nor any ablated skill variant significantly improves performance; all p-values are at least 0.396, and the total spread across variants is only 1.2 pp. A supplemental token-matched control adds 1,512 runs and finds that Full skills perform similarly to task-irrelevant skill-formatted content. The results caution against using one LLM-generated skill per data-science workflow as a default single-shot prompting strategy.

• The paper demonstrates that LLM-generated skills do not consistently improve data science task performance compared to task-only prompts.
• Ablation of examples and references reveals negligible impact on performance, emphasizing a need for context-specific skill design.
• The findings recommend shifting from broad, flat injection methods to selective, task-specific architectures for robust data-science automation.

Summary and Component-Level Analysis of LLM-Generated Skills in Data Science Workflows

Introduction and Motivation

This work interrogates the utility of low-curation, LLM-generated "skills" as reusable prompt elements for data-science agents performing recurring execution tasks across standard workflow stages: data preparation, data extraction, statistical analysis, and reporting. The study examines whether flat skill injection, generated per workflow family and then prepended to tasks, delivers measurable performance improvement relative to task-prompt-only baselines. The motivation stems from the scalability bottleneck of expert-written skill files, contrasted with the low-friction generation of LLM-based skills.

Experimental Design and Ablation Strategy

The evaluation spans 56 tasks mapped to four data-science workflows, executed on nine model configurations across three providers (OpenAI, Google, Anthropic) including both compact and frontier models and explicit reasoning modes. Each skill instance was generated via Gemini 2.5 Pro in a single, autonomous run, comprising Routing, Core Procedure, Worked Examples, and Reference Notes sections, all injected in a flat markdown wrapper. The ablation deletes Examples and/or References to test section-level impacts while holding Core Procedure and Routing constant.

Supplemental controls include a token-length-matched, domain-irrelevant "office supply" skill to isolate effects of input length and formatting, and a priority directive instructing models to prefer task instructions when they conflict with skill content. Automated deterministic verifiers (DataFrame, SQL, numeric tolerance, JSON schemas) scored outcomes as pass/fail without human intervention.

Results: Aggregate and Component Effects

Across all skill conditions and ablations, no variant reliably outperforms task-only prompts; the maximal spread is 1.2 percentage points with all $p$-values $\geq$ 0.396. The token-matched Length-Control condition performs similarly to Full skills, directly refuting prompt-length as a sole explanation. The prompt-level task-over-skill directive (Full+Priority) partly mitigates skill-task conflicts but does not repair the main failure modes; increases in input token cost ($\sim$4.5×) are observed without gains in pass rates.

No skill component (Examples or References) provides a statistically significant benefit, either individually or interactively (all confidence intervals include zero). Stage-level analysis reveals three distinct failure patterns: redundancy at ceiling for data extraction and reporting, negative effects at moderate baseline in data preparation due to competing generic heuristics, and performance floor for statistical analysis tasks indicating reasoning/computation bottlenecks.

Figure 1: Full-skill effects by task, showing ceiling/floor clustering and near-zero impacts for most tasks.

Diagnostic Patterns Across Lifecycle Stages

High-baseline workflows (SQL generation, JSON reporting) are already well represented in LLM training corpora, resulting in near-ceiling outcomes for task-only prompting with little room for skill-based improvement. For moderately difficult data-preparation tasks, skill injection sometimes reduces accuracy by competing with task-specific requirements, evidenced by deterministic condition-flips where skill heuristics override explicit instructions. Low-baseline statistical analysis tasks reveal that prompt-level knowledge injection cannot surmount inherent capability gaps in multi-step reasoning and numerical reliability.

Figure 2: Skill effects by model group, showing bootstrap confidence intervals centered at zero across compact, frontier, and explicit-reasoning tiers.

Implications for Data-Science Agent Architecture

The findings caution against naive adoption of LLM-generated family-level skills as default single-shot prompt strategies in data-science task automation. Flat injection not only incurs substantial token costs but also introduces risk of harmful conflict in moderately complex procedures. Selective loading of reference material, fine-grained task-specific skill construction, system-level priority mechanisms, and agent architectures incorporating execution validation and feedback appear preferable to monolithic skill-fronted prompting.

The component ablation supports the view that broad skills are brittle for mid-difficulty tasks where procedural nuances matter. The interaction analysis further suggests that skill content cannot be assumed beneficial across workflow regimes or model groups.

Limitations and Future Directions

This evaluation is restricted to single-turn, execution-layer data-science workflows and does not include iterative planning, tool use, or upstream business logic. Only one generated skill per lifecycle stage was tested, absent expert-written task-specific controls. The study cannot rule out small effects (<3pp) or sensitivity to carefully authored skills. Authors recommend subsequent work on task-specific skill tailoring, selective retrieval architectures, schema/tool constraints, and feedback-driven agent orchestration to address the identified bottlenecks.

Conclusion

This component ablation study robustly finds no significant performance improvement from LLM-generated, flat-injected skills for data-science agents over task-prompt-only baselines across 56 tasks and nine model configurations. Token-matched controls establish input length as non-causal, and prompt-level priority directives provide only partial mitigation of conflict-driven errors. Practical agent design must eschew naive skill injection in favor of more nuanced, context-specific, and selectively loaded procedural guidance. The null aggregate and component-level results reinforce the importance of lifecycle-aware, workflow-dependent agent architectures for robust data-science automation.