Why Solve It Twice? Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering

Abstract: ML engineering agents waste compute rediscovering known techniques because every competition is a cold start. We present HASTE, a hierarchical multi-agent system that organizes cross-competition knowledge into three scope tiers (global, domain, and competition-specific), each coupled to a matching agent level. An orchestrator coordinates domain specialists and promotes learning between tiers via LLM-driven abstraction. A controlled ablation provides evidence for scoped loading: holding a 159-skill inventory constant across 8 competitions, tiered loading achieves a 100% medal rate while flat loading reaches only 62.5%, the same medal rate as loading no skills, and consumes 2x the output tokens. On the full MLE-Bench Lite benchmark (22 Kaggle competitions), HASTE reaches a medal rate of 77.3% using Claude Sonnet 4.6 at 12h per competition. In a cold-start run, the system begins with no accumulated skills. In warm-start runs, it reloads skills learned from earlier competitions, using only global and domain-level skills for transfer across competitions. Warm starts use 52% fewer refinement iterations, and the fraction of proposed changes kept by the agent rises from 42% at low inventory to 85% once 50+ skills are available. These results suggest that better knowledge organization can partly substitute for model strength and compute budget in ML-engineering agents.

• The paper introduces HASTE, an ML engineering agent that organizes reusable skills by global, domain, and competition scope and routes each skill to the appropriate workflow stage.
• HASTE achieved a 77.3% medal rate on 22 MLE-Bench Lite competitions with Claude Sonnet 4.6, while warm starts reduced refinement iterations by 52% compared with cold starts.
• The controlled ablation found that tiered loading reached 100% medals on eight competitions and was 2.7× more token-efficient per medal than flat loading, although the single-seed sample limits statistical certainty.

Motivation and problem statement

MLE-Bench evaluates ML engineering agents on Kaggle competitions as independent tasks, and most agents treat them that way, resetting all state between competitions. Techniques proven effective on one task must be rediscovered on the next, a cost typically offset by frontier models or longer compute budgets. Prior cross-task transfer systems—Voyager, ExpeL, Agent K, MLCopilot, ADAS—store knowledge in flat pools or by memory type, which means the agent must still select the right prior from an undifferentiated context window. HASTE (Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering) argues that the organization of accumulated knowledge, not merely its volume, determines whether transfer helps, and provides a controlled ablation supporting this claim.

Architecture

HASTE couples two hierarchies through scoped context loading.

Skill hierarchy: a plain-text filesystem (markdown with YAML frontmatter) organized into three tiers by scope of applicability—global (5 entries, loaded by all specialists), domain (12 NLP, 19 tabular, 15 vision entries, loaded only by the matching specialist), and competition (108 entries across 21 directories, loaded only on re-runs). Within each tier, entries are typed by function: technique entries feed proposal prompts, commitment priors (mined algorithmically from score variance across prototype options) feed the prototype screen, and refinement hints (mined from per-knob acceptance rates) feed optimization stages. Loading is simple directory concatenation; no embedding index is used at the current scale of ~159 entries, a deliberate choice citing theoretical limits of single-vector retrieval (Weller et al., 28 Aug 2025).

Agent hierarchy: an orchestrator handles domain classification, round scheduling, and skill promotion between rounds via LLM-driven abstraction (skip / competition / domain / global / conflict decisions, with conflicts kept and condition-annotated). Three domain specialists run a five-stage pipeline per competition: task profiling, a three-way diverse prototype screen (justified by observed score spreads up to $2.7\times$ between prototypes), adaptive refinement of both winner ($N{=}20$) and runner-up ($N{=}6$) through a linear Exploring → Optimizing → Fine-tuning loop with auto-escalation after two consecutive non-improvements and revert-on-regression, rank-average ensembling accepted only if it beats the best single member, and learning production via paired success/failure reflection.

Notably, the search strategy is intentionally linear rather than tree-based or evolutionary. The authors concede this is plausible only because loaded priors collapse the branching factor, and that a controlled comparison against tree/evolutionary search at fixed knowledge conditions remains future work.

Main results

On MLE-Bench Lite (22 competitions) with Claude Sonnet 4.6 under a 12-hour budget—half the dominant leaderboard budget—HASTE achieves a 77.3% medal rate (17/22: 10 gold, 2 silver, 5 bronze; above-median rate 86.4%). This places it in the top public performance band while being the only non-frontier-model agent at or above 77%, and one of only two top-band agents running at 12h. The authors appropriately caveat that public leaderboard numbers carry per-task noise (reported SD ≈ 4.4) and that the headline figure is a single campaign result, not a multi-seed estimate.

The transfer evidence is the paper's core contribution:

• A cold-start single pass reaches only 40.9%; reloading global and domain skills (competition skills excluded, so no same-dataset leakage) lifts the same system to 77.3%, flipping 8 of 13 previously failed competitions to medal.
• Warm-start runs need 7.8 vs. 16.3 refinement iterations to reach their best score, a 52% reduction.
• The hit rate (fraction of proposed changes kept rather than reverted) rises from 42% at 0–15 skills to 85% at 50+ skills, indicating the store functions as a structured prior over code edits.
• Refinement beats the prototype winner in 92% of runs (23/25); in 3 cases the runner-up branch wins, validating the second hedge.

Per-domain performance is uneven: NLP 100%, vision 80%, tabular 40%, audio gold—the failures (taxi-fare, dog-breed) plausibly require approaches absent from the store.

The controlled ablation: organization matters more than volume

The most consequential experiment holds the 159-skill inventory, model, pipeline, and 11-iteration budget constant across 8 competitions, varying only the loading function:

Condition	Medal rate	Mean score	Output tokens	Tokens/medal
Tiered	100% (8/8)	0.949	2.27M	284K
Flat (all 159)	62.5% (5/8)	0.910	3.78M	756K
Empty	62.5% (5/8)	0.893	1.86M	371K

The striking claim is that flat loading performs no better than loading no skills at all, while consuming $2\times$ empty's output tokens—tiered is 2.7× more token-efficient per medal. Flat runs more experiments (75 vs. 60) with a slightly higher execution success rate, yet gains nothing: the extra compute is poorly directed. Log analysis attributes this to signal dilution, context-budget displacement (~145K characters crowding out reasoning), and overconfident model selection (the flat agent repeatedly attempted DeBERTa-v3-large and hit OOM where the empty agent scored higher with simpler models). Gaps concentrate on harder tasks: on mlsp-2013-birds, tiered scores 0.964 (gold) versus 0.860/0.832; on random-acts-of-pizza, 0.798 (silver) versus 0.599/0.481.

The authors are candid about statistical weakness: with $N=8$ and a single seed, bootstrap CIs just exclude zero (mean difference +0.040, CI [+0.001, +0.093]) and Wilcoxon/Fisher tests fall short of significance ($p = 0.08$–$0.11$). They also correctly scope the claim: the flat condition confounds organization with prompt-length budget, so the supported conclusion is that scoped loading beats full-load and empty baselines on this inventory—not a general theorem about hierarchy. A fully flat end-to-end pipeline is not tested.

Limitations and open questions

The principal limitation is single-seed evaluation throughout; multi-seed replication at the full 75-competition benchmark is explicitly deferred. The ablation isolates the loading function but leaves other engineering components (prototype screen, runner-up branch, ensemble policy, failure taxonomy, auto-escalation) un-ablated. Whether linear refinement genuinely matches tree or evolutionary search when priors are held fixed is unresolved. Finally, the character-cap-based loading will presumably require embedding-based retrieval as inventories scale beyond ~159 entries, and its interaction with scoping is untested.

Conclusion

HASTE demonstrates that hierarchical, scope-tiered organization of plain-text skills—with LLM-driven promotion between tiers—can substitute substantially for model strength and compute budget in ML engineering agents, reaching competitive MLE-Bench Lite performance with a non-frontier model at half the standard time budget. The fixed-inventory ablation is the key evidentiary piece: it shows accumulation without scoping yields no medal-rate gain over a cold start at doubled token cost. The findings rest on small-N, single-seed evidence, but the consistent direction of effect across medal rate, mean score, and token efficiency makes scoped loading a credible mechanism warranting replication.