Understanding generalization behavior of interval-tuned Irace configurations across sample sizes

Determine the reasons underlying the observed size-dependent performance crossover of Irace-tuned (p2, p3) parameter pairs for 3-dimensional Kronecker point sets (with p1 = 1/n), where configurations tuned on specific n-intervals (e.g., INTERVAL_5 and INTERVAL_9) generalize well up to roughly 1000–1200 points before being surpassed by the IRACE_1500 configuration; characterize how the parameter-to-discrepancy landscape changes with n and explain why certain training intervals yield superior cross-n performance.

Background

The authors tune Kronecker parameters with Irace on disjoint intervals of n to study generalization across sample sizes. Multiple interval-specific configurations perform similarly for small n, but as n grows, certain configurations (notably INTERVAL_5 and INTERVAL_9) remain competitive up to about 1000–1200 points before a different configuration (IRACE_1500) clearly dominates.

This behavior occurs in a highly multimodal parameter landscape and suggests nontrivial dependence of the optimal parameters on the sample size. The authors explicitly state they lack a full understanding of the phenomenon, highlighting a need to analyze how optimal Kronecker parameters evolve with n and what structural properties govern generalization across sizes.

References

Figure \ref{fig:irace_chunck_complete} illustrates how the lines are closely packed when n is small, suggesting that most configurations perform similarly for smaller sizes. Performance differences become more noticeable as the number of points increases. For example, INTERVAL_5 and INTERVAL_9 consistently outperform the majority of other interval configurations and are competitive with IRACE_1500 up to about 1000–1200 points; beyond this range, a clear improvement emerges. We do not have a full understanding of why this happens.

— Finding Low Star Discrepancy 3D Kronecker Point Sets Using Algorithm Configuration Techniques  (2604.00786 - Abderrahim et al., 1 Apr 2026) in Subsection "Assesing Parameter tuning on different n intervals"