TroVE: Training-Free Code Toolbox

• TroVE is a training-free method for programmatically solving tasks by automatically inducing, storing, pruning, and reusing higher-level Python functions.
• It employs three prompting modes—SKIP, CREATE, and IMPORT—to balance direct generation and tool creation for efficient code execution.
• Evaluation on the MATH benchmark reveals marginal gains largely due to selection improvements rather than the standalone benefit of reusable tool libraries.

TroVE is a training-free method for programmatic task solving with code-generating LLMs that induces, stores, prunes, and reuses higher-level Python functions as a shared toolbox across examples (Wang et al., 2024). In its original formulation, TroVE was motivated by the observation that mathematicians and other problem solvers routinely reuse lemmas, formulas, and compact helper procedures rather than reconstructing every solution from primitives. TroVE operationalizes that idea through three prompting modes—SKIP, CREATE, and IMPORT—and selects among sampled candidate programs by execution agreement and simplicity (Wang et al., 2024). A later compute-matched re-evaluation on the MATH benchmark argued that TroVE’s apparent advantage on that benchmark is largely explained by sampling budget and selection effects rather than by reusable tool libraries per se, reducing its net benefit over a compute-matched primitive baseline to approximately $1\%$ and finding that difference not statistically significant over five seeds (Sesterhenn et al., 16 Jul 2025).

1. Conceptual basis and problem setting

TroVE addresses programmatic task solving, where a LLM writes a program that, when executed in an environment, produces the answer (Wang et al., 2024). The central claim of the original method is that solutions written only with primitive functions—such as Python built-ins, numpy, sympy, or pandas depending on the dataset—tend to be verbose, tedious, and error-prone, whereas higher-level composed functions can encapsulate reusable logic and simplify both execution and verification (Wang et al., 2024). TroVE therefore seeks to induce these functions automatically, without additional supervision, parameter updates, or expert-designed tool libraries.

Formally, the original paper denotes programmatic solution writing as $P_{LM}(q, e, f)$, where a LLM solves a query $q$ grounded in an environment $e$ by composing functions $F = \{f_1,\dots,f_n\}$ (Wang et al., 2024). In TroVE, the function set is not fixed: it evolves over time as the model creates new helper functions and later reuses them. The toolbox is intended to remain both verifiable and efficient through execution agreement-based selection and periodic trimming (Wang et al., 2024).

The MATH benchmark became a focal setting for later analysis because it stresses structured procedural reasoning, symbolic manipulation, and function orchestration in code, which are precisely the conditions under which reusable abstractions should, in principle, confer benefits (Sesterhenn et al., 16 Jul 2025). The compute-matched re-evaluation therefore treated MATH as a critical test of TroVE’s stated rationale rather than as a peripheral benchmark (Sesterhenn et al., 16 Jul 2025).

2. Modes, workflow, and toolbox management

TroVE processes examples online in a streaming fashion while maintaining a shared toolbox $F$ over time (Wang et al., 2024). For each example, it defines three interaction modes.

SKIP is direct generation using only primitive capabilities, with no learned toolbox reference (Sesterhenn et al., 16 Jul 2025). On MATH, SKIP is identical to the PRIMITIVE baseline prompt (Sesterhenn et al., 16 Jul 2025).

CREATE instructs the model to define a new helper function that abstracts reusable logic, use it to solve the current task, and add it to the toolbox (Sesterhenn et al., 16 Jul 2025). In the original formulation, each CREATE response contains both a solution and any function it creates, and the selected function is appended to the shared library (Wang et al., 2024).

IMPORT exposes the current toolbox in the prompt and asks the model to invoke existing functions appropriately (Sesterhenn et al., 16 Jul 2025). Tools are represented as Python functions, stored as code strings, and made available in prompt context for subsequent tasks, either in a task-local or session-local toolbox depending on the implementation setting (Sesterhenn et al., 16 Jul 2025).

The original TroVE paper used a default of $K=5$ responses per mode, producing $3K$ candidates per example (Wang et al., 2024). The MATH re-evaluation instead used a fixed total budget $K=15$ samples per task and, under TroVE, split this budget equally across SKIP, CREATE, and IMPORT, yielding $K/3$ candidate programs per mode (Sesterhenn et al., 16 Jul 2025). This compute split can be written as

$P_{LM}(q, e, f)$0

with $P_{LM}(q, e, f)$1; under TroVE, $P_{LM}(q, e, f)$2, while under PRIMITIVE, $P_{LM}(q, e, f)$3 (Sesterhenn et al., 16 Jul 2025).

Toolbox maintenance is integral to the method. The original paper periodically trims functions that have been used less than a threshold $P_{LM}(q, e, f)$4, with

$P_{LM}(q, e, f)$5

where $P_{LM}(q, e, f)$6 and $P_{LM}(q, e, f)$7 is the number of processed examples; trimming is performed every 200 steps in the default setup (Wang et al., 2024). To preserve solvability, examples that depended on trimmed functions are regenerated under IMPORT and SKIP modes, affecting fewer than $P_{LM}(q, e, f)$8 of the dataset in the reported experiments (Wang et al., 2024).

3. Selection mechanism and compute allocation

TroVE’s defining verification mechanism is execution agreement followed by a simplicity preference (Wang et al., 2024). In the original method, all sampled candidate programs are executed; non-runnable programs are discarded; the most frequent execution outcome is chosen; and if multiple programs yield that winning answer, the least complex program is preferred, with deterministic tie-breaking if needed (Wang et al., 2024).

The MATH re-evaluation formalized this selector as follows. Let successful candidates be indexed by $P_{LM}(q, e, f)$9, each producing answer $q$0. Let $q$1 denote the frequency of answer $q$2, and let $q$3 be a complexity proxy such as operation count. TroVE then chooses

$q$4

followed by

$q$5

This combines self-consistency with a shortest-program tie-break (Sesterhenn et al., 16 Jul 2025). The same paper also writes the standard accuracy definition as

$q$6

A central technical finding of the re-evaluation was that the released implementation did not actually perform this selector over the full pooled ensemble. Instead, it first selected the best candidate within each mode and then selected among the three per-mode winners, thereby reducing the effective ensemble size and lowering the chance that the global plurality answer would win (Sesterhenn et al., 16 Jul 2025). Correcting this from a two-stage to a one-stage selector over all $q$7 candidates yielded a $q$8 absolute accuracy gain on the MATH subset, improving reproduced TroVE from $q$9 to $e$0 mean accuracy over five seeds (Sesterhenn et al., 16 Jul 2025).

The same re-evaluation emphasized that compute matching must include the total number of candidate generations and executions, tokens per sample, prompts, decoding, and runtime settings (Sesterhenn et al., 16 Jul 2025). Its matched setup used Code Llama, temperature $e$1, top_p $e$2, max_new_tokens $e$3, a $e$4 s execution timeout, and five random seeds, reporting mean $e$5 standard deviation (Sesterhenn et al., 16 Jul 2025).

4. Original empirical claims and broader benchmark results

In its original report, TroVE was evaluated on 11 datasets spanning math, table question answering, and visual reasoning, using CodeLLaMa2-7b-Instruct as the primary model and GPT-4 for comparison to prior tool-making methods (Wang et al., 2024). The method was described as yielding simpler solutions with higher accuracy than baselines while using $e$6 smaller toolboxes (Wang et al., 2024). On WTQ, a human verification study with 100 examples and 6 evaluators found that TroVE improved detection accuracy by about $e$7 over Primitive and reduced verification time by $e$8, with average verification accuracy $e$9 versus $F = \{f_1,\dots,f_n\}$0 and mean times $F = \{f_1,\dots,f_n\}$1 s versus $F = \{f_1,\dots,f_n\}$2 s (Wang et al., 2024).

For MATH in the original paper, TroVE used seven subjects and reported improvements over Primitive ranging from $F = \{f_1,\dots,f_n\}$3 with CodeLLaMa2-7b-Instruct (Wang et al., 2024). More generally, on overlapping datasets with GPT-4-based prior methods, the paper reported that TroVE achieved, for example, $F = \{f_1,\dots,f_n\}$4 accuracy with 16 tools on MATH_algebra, $F = \{f_1,\dots,f_n\}$5 with 38 tools on TabMWP, and $F = \{f_1,\dots,f_n\}$6 with 8 tools on GQA, comparing favorably to CREATOR, CRAFT, and LATM while using much smaller toolboxes (Wang et al., 2024).

The original paper also stressed that induced functions varied by dataset. For MATH, TroVE often imported advanced libraries such as sympy or defined functions like calculate_remainder; for table question answering it induced composed data-manipulation functions such as get_match_after_condition; for visual reasoning it composed modules such as locate_objects and crop_region into higher-level functions like get_image_region (Wang et al., 2024). This was presented as evidence that the toolbox adapted to domain-specific functional structure rather than merely accumulating generic code fragments.

At the same time, the original paper explicitly acknowledged limitations. TroVE depends on the ability of the underlying code LLM to produce executable programs; its verification mechanism covers execution success and answer matching rather than rich property tests or learned verifiers; and its pipeline incurs additional computational overhead by sampling $F = \{f_1,\dots,f_n\}$7 responses per example and maintaining the toolbox (Wang et al., 2024).

5. Compute-matched re-evaluation on MATH

The later study "A Compute-Matched Re-Evaluation of TroVE on MATH" revisited these claims under a matched evaluate-time budget on the 3,201-problem MATH subset used by Wang et al. (2024), reporting per-category and overall accuracy across algebra, number theory, counting/probability, geometry, intermediate algebra, prealgebra, and precalculus (Sesterhenn et al., 16 Jul 2025).

Its headline results reframed the original comparison. Wang et al. had reported TroVE at $F = \{f_1,\dots,f_n\}$8 versus PRIMITIVE at $F = \{f_1,\dots,f_n\}$9, but the re-evaluation observed that PRIMITIVE had effectively been run with $F$0, whereas TroVE used many samples (Sesterhenn et al., 16 Jul 2025). Under $F$1 with the two-stage selector, reproduced TroVE reached $F$2 ($F$3); after correcting the selector to one-stage pooling, TroVE reached $F$4 ($F$5); and compute-matched PRIMITIVE at $F$6 reached $F$7 ($F$8) (Sesterhenn et al., 16 Jul 2025). The resulting TroVE advantage over PRIMITIVE therefore shrank to approximately $F$9 and was not statistically significant over five seeds (Sesterhenn et al., 16 Jul 2025).

Per-category differences were also limited. The re-evaluation reported that absolute differences were at most $K=5$0, with PRIMITIVE slightly better in algebra, geometry, intermediate algebra, number, and prealgebra, and corrected TroVE slightly better in counting and precalculus (Sesterhenn et al., 16 Jul 2025). This weakens any interpretation that toolbox reuse yields a broad, category-level advantage on MATH.

The study further examined larger budgets by aggregating across seeds up to 75 samples. Under majority vote, TroVE only slightly outperformed PRIMITIVE by at most $K=5$1; under oracle selection the gap was at most $K=5$2; and both curves plateaued, indicating diminishing returns from additional sampling without better selection (Sesterhenn et al., 16 Jul 2025).

6. Mechanistic interpretation, controversy, and implications

A central controversy around TroVE concerns the source of its gains: reusable tool libraries versus sampling and selection. The re-evaluation directly addressed this by decomposing mode contributions and comparing oracle versus majority-vote settings (Sesterhenn et al., 16 Jul 2025).

Under an oracle selector with pass@5, CREATE solved the largest fraction of unique tasks on the MATH aggregate, at $K=5$3, followed by SKIP at $K=5$4 and IMPORT at $K=5$5 (Sesterhenn et al., 16 Jul 2025). TroVE also produced, on average, $K=5$6 more distinct candidate answers per task than PRIMITIVE at $K=5$7, including $K=5$8 in prealgebra and $K=5$9 in precalculus (Sesterhenn et al., 16 Jul 2025). These findings indicate that prompt diversity broadens the hypothesis pool.

However, IMPORT contributed the smallest unique coverage, and the paper noted prior analysis finding that tools were often trivial or rarely reused (Sesterhenn et al., 16 Jul 2025). It therefore concluded that the gains attributable specifically to tool reuse appear limited relative to CREATE and SKIP. This suggests that, at least on MATH, TroVE’s residual advantage arises more from diverse prompting and corrected ensemble selection than from the sustained reuse of a non-trivial toolbox (Sesterhenn et al., 16 Jul 2025).

The oracle analysis strengthened that interpretation. With an oracle selector, both TroVE and PRIMITIVE gained about $3K$0 absolute accuracy at $3K$1 relative to majority voting (Sesterhenn et al., 16 Jul 2025). The paper therefore argued that selection quality dominates toolbox diversity on this benchmark, and recommended investment in stronger selectors, including agreement-plus-complexity with learned scoring, semantic verification, or checker-guided selection (Sesterhenn et al., 16 Jul 2025).

The same study also identified several limitations. Its reuse analysis emphasized unique mode contributions rather than direct measurement of reuse frequency or tool triviality; MATH’s numeric-answer format and relatively short-horizon reasoning may underrepresent domains where toolboxes are more valuable, such as long-horizon agentic workflows or formal theorem proving; and differences of roughly $3K$2 at $3K$3 over five seeds were not statistically compelling (Sesterhenn et al., 16 Jul 2025). A plausible implication is that TroVE’s underlying idea may remain more relevant in settings with stronger cross-task recurrence, more complex retrieval demands, or richer verification signals than those available on MATH.

7. Reproducibility, relation to adjacent work, and current standing

The original TroVE implementation is available through its official GitHub repository and was described as training-free, Python-based, and compatible with CodeLLaMa2-7b-Instruct as well as GPT-4 in comparative experiments (Wang et al., 2024). Its standard hyperparameters included temperature $3K$4, top-p $3K$5, a 512-token generation cap, and $3K$6 in-context examples; toolbox trimming occurred every 200 steps (Wang et al., 2024). The MATH re-evaluation used the official TroVE GitHub under a CC-BY-SA-4.0 license, with Code Llama as backend and five seeds per method and category (Sesterhenn et al., 16 Jul 2025).

In relation to adjacent work, the re-evaluation situated itself alongside findings that removing IMPORT did not substantially harm performance and that compute-matched baselines can erase apparent toolbox gains (Sesterhenn et al., 16 Jul 2025). The original TroVE paper, by contrast, had positioned the method against domain-specific library learning and prior tool-making pipelines by emphasizing its training-free nature, execution-agreement selection, and small induced toolboxes (Wang et al., 2024). These two perspectives are not logically incompatible: one emphasizes broad cross-dataset empirical utility, while the other narrows the claim on MATH specifically by showing that much of the earlier gap was due to unmatched compute and a suboptimal selector.

The current standing of TroVE is therefore mixed but technically well-defined. As an architecture, it denotes a three-mode, toolbox-inducing, execution-agreement-based framework for programmatic task solving (Wang et al., 2024). As a result on MATH, its strongest supported interpretation is narrower: correcting the selector materially improves reproduced performance, but under matched compute the net advantage over a repeated-sampling primitive baseline is marginal, and the main bottleneck appears to be selection rather than toolbox construction (Sesterhenn et al., 16 Jul 2025).