Determine whether IR node-count stabilization predicts downstream tool outputs

Determine whether stabilization of the inferred intermediate representation’s node count corresponds to stabilization of outputs from downstream tools such as linters, coverage analyzers, or mutation-analysis systems.

Background

The stabilization analysis measures only the inferred IR’s node count across repeated runs. No downstream linter finding, coverage value, or mutation score is computed, so the analysis cannot establish that a number of repeated inferences is sufficient for a real consumer of the IR.

Resolving this problem would require evaluating downstream tools directly and testing whether their outputs track the measured node-count signal. Until then, the reported stabilization behavior is only a proxy result.

References

We do not measure a downstream tool. We measure the IR's node count across runs, as a model-agnostic stand-in for any quantity a consumer might derive from the IR. No linter finding, coverage value, or mutation score is computed. Conclusions here transfer to a real tool only insofar as its output tracks node count, which we have not established.

— How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure  (2609.30074 - Sarkar, 24 Sep 2026) in Section 6, subsection “How many reruns a consumer needs”; Appendix, Section “Stabilisation in full, and the recovery-campaign completion counts,” paragraph “The signal is a proxy”