Ablation of schema structure and explicitness

Determine the separate contributions of typed field decomposition, explicit specification of required fields, structured extraction, and deterministic serialization to the observed reliability improvement over direct native-syntax generation for HEPToolBench tasks.

Background

The controlled comparison changes several aspects of the model-facing interface simultaneously. The structured condition provides typed field decomposition, explicit required-field descriptions, recoverable structured-output extraction, and deterministic serialization, whereas the native condition asks the model to produce final MadGraph, run-card, or workflow syntax directly.

Because these components are not independently varied, the reported experiment cannot identify which component produces the observed improvement. The authors specifically identify an ablation in which a native-syntax prompt carries the same explicit list of required fields as a concrete way to separate the effect of output format from the effect of added prompt explicitness.

References

The structured prompts also state the required information more explicitly and restrict the space of possible responses, and the present experiment does not separate these effects; isolating the format effect from this added explicitness would require a further ablation, such as a native-syntax prompt carrying the same explicit list of required fields, which we leave to future work.

— HEPToolBench 1.2: Testing How Reliably Language Models Can Drive Particle Physics Software  (2608.28232 - Singh et al., 28 Aug 2026) in Section 5.1, “Implications for HEP software interfaces”