Establish whether reproducibility rankings transfer between annotated and unannotated prompt regimes

Establish whether the ordering of models by prompt-structure reproducibility transfers from the annotated transcription regime to the genuinely unannotated recovery regime, using a sample large enough to support a meaningful comparison.

Background

The study compares a predominantly annotated corpus, in which models copy explicitly marked structure, with an unannotated real-world corpus requiring structural recovery. Only four models occur in both regimes, and the observed ordering is broadly preserved but statistically inconclusive.

The unresolved issue is whether reproducibility performance is general across transcription and recovery tasks or depends on the regime, corpus, and model panel. The authors explicitly limit their conclusion to a level difference rather than a claim about transfer.

References

On those four the ordering is broadly preserved (Spearman $\rho=0.80$) but at $n=4$ this is not significant ($p=0.20$), and two of the four change rank. We had expected the ranking to scramble between regimes and it does not visibly do so; we simply cannot test it at this sample size.

— How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure  (2609.30074 - Sarkar, 24 Sep 2026) in Section 4, paragraph “What we cannot conclude”