fastml: Guarded Resampling Workflows for Safer Automated Machine Learning in R

Abstract: Preprocessing leakage arises when scaling, imputation, or other data-dependent transformations are estimated before resampling, inflating apparent performance while remaining hard to detect. We present fastml, an R package that provides a single-call interface for leakage-aware machine learning through guarded resampling, where preprocessing is re-estimated inside each resample and applied to the corresponding assessment data. The package supports grouped and time-ordered resampling, blocks high-risk configurations, audits recipes for external dependencies, and includes sandboxed execution and integrated model explanation. We evaluate fastml with a Monte Carlo simulation contrasting global and fold-local normalization, a usability comparison with tidymodels under matched specifications, and survival benchmarks across datasets of different sizes. The simulation demonstrates that global preprocessing substantially inflates apparent performance relative to guarded resampling. fastml matched held-out performance obtained with tidymodels while reducing workflow orchestration, and it supported consistent benchmarking of multiple survival model classes through a unified interface.

• The paper introduces a guarded resampling workflow that restricts data-dependent preprocessing to each cross-validation fold, effectively eliminating leakage risks.
• fastml implements hard-guard mechanisms and audit features that automatically block misconfigurations and enforce safe data handling across various resampling strategies.
• Empirical simulations demonstrate that fastml significantly reduces performance inflation, ensuring accurate evaluation for classification, regression, and survival analysis tasks.

fastml: Guarded Resampling Workflows for Safer Automated Machine Learning in R

Motivation and Problem Statement

The prevalence of preprocessing leakage in machine learning pipelines—where data-dependent transformations (e.g., scaling, imputation) are estimated using the entire dataset before train/test splitting or cross-validation—constitutes a critical source of performance inflation and irreproducibility. In the R ecosystem, while frameworks such as caret, tidymodels, and mlr3 offer substantial flexibility, correct and leakage-free evaluation typically depends on the user structuring and coordinating complex pipelines. These frameworks often make leakage avoidance possible but do not enforce it, leaving pipelines vulnerable to subtle but consequential flaws. Empirical assessments have demonstrated that such methodological faults propagate optimistic and irreproducible results in a variety of applied domains.

Guarded Resampling: Design and Implementation

fastml introduces a safety-centric execution paradigm that decouples the user from manual assembly of leakage-award pipelines by default. The core design is the "guarded resampling" workflow, in which any data-dependent preprocessing is estimated exclusively within each cross-validation (CV) fold’s analysis split, and the resultant transformation is applied only to both the fold-specific analysis and assessment splits (i.e., test data). All preprocessing estimation and model fitting is strictly performed within the confines of the resampling design—whether group-based, time-based, or standard random folds—thereby ensuring explicit train/evaluation separation throughout.

A set of hard-guard mechanisms are implemented to block pathological or high-risk configurations. Notably, fastml:

• Detects "full-analysis" splits (i.e., no holdout data) and aborts with a diagnostic.
• Audits and rejects user-supplied recipe steps that reference external environments or embed global data, to prevent hidden dependencies and undetected leakage.
• Surfaces configuration errors for grouped and time-ordered resampling, warning and refusing to proceed if requisite data structure (e.g., group/block labels) is absent.

The package architecture encodes these mechanisms at the resampling engine level (not as an optional user choice), making fold-local parameter estimation non-optional whenever supported. While this cannot categorically prevent all forms of upstream leakage (e.g., already-preprocessed input), it eliminates most pipeline-level leakage within the supported execution paths.

Empirical Demonstration: Simulation and Numerical Findings

A Monte Carlo simulation is presented which quantifies the effect of global (leaky) vs. fold-local (guarded) preprocessing in a prototypical batch-effect scenario. In this experiment, site-wise normalization performed prior to grouped CV induces substantial optimism in ROC AUC: the leaky workflow yields a mean AUC of 0.809 ($\mathrm{SD}=0.018$), compared to 0.651 ($\mathrm{SD}=0.046$) under guarded resampling—a mean inflation of 0.158 (95% CI: [0.149, 0.167]) per run. This result provides numerical evidence that improper preprocessing order can yield spurious claims of model performance even for unsupervised steps and in settings with group- or batch-structured data. In all simulation runs, the guarded resampling paradigm eliminated this inflation.

Interface, Automation, and Software Architecture

fastml exposes a single-call user interface in R, encapsulating data splitting, preprocessing, model fitting, evaluation, and optional hyperparameter tuning. A broad set of model families are supported for classification, regression, and survival; preprocessing is modular and customizable, but all data-dependent estimation steps are tightly coupled to the annotated analysis splits by default.

The package returns a unified S3 object aggregating models, preprocessing artifacts, predictions, evaluation results, and, optionally, the exact data partitions used. This fully auditable result object supports prediction, summary, serialization, and visual diagnostics, and enables comprehensive post hoc analysis of workflow integrity.

Security and auditability are addressed at the recipe and execution environment level. Audit mode flags and logs potential environment or I/O dependencies during execution; preventive guards statically reject structures that appear to encode unsafe patterns. While these features do not constitute comprehensive runtime sandboxes, they represent the strongest routine safety measures currently found in R-based AutoML libraries.

Survival Analysis and Model Generality

A distinguishing contribution of fastml is end-to-end support for survival learning, with native guarded resampling for Cox-type, parametric, ensemble-based, and boosting-based survival learners. Key technical points:

• Survival targets use proper right-censoring structures, and the package supports engines requiring interval-censored representations (e.g., XGBoost AFT).
• Survival resampling is implemented in a leakage-robust manner, with metric computation consistently standardized (e.g., Harrell’s C-index) across engines, and interface-level constraints prevent method/resampling mismatches.
• Fastml’s interface allows direct comparison and benchmarking of classical baselines (e.g., Cox PH, Weibull AFT) against penalized, non-linear, and boosting methods across a range of data regimes.

Numerical survival benchmarks over datasets of size $N=228$ (lung), $N=2,982$ (Rotterdam breast cancer), and $N=7,874$ (flchain) demonstrate that fastml achieves concordant or superior results to baseline classical approaches, providing mean C indexes (e.g., 0.641 on lung, 0.695 on Rotterdam, 0.793 on flchain for penalized Cox) and enabling unified workflow management for methodologically rigorous survival analysis.

Comparison to Existing Frameworks

Architecturally, fastml sits between the low-level flexibility (and user responsibility) of tidymodels/mlr3 and the high-level automation of h2o. Unlike other frameworks, fastml enforces guarded resampling by default, reduces workflow orchestration burden, performs structural and environment audits, and makes preprocessing safety opt-out only via deliberate user action. Survival modeling is natively supported with both standard and custom distributions (e.g., piecewise exponential with arbitrary cut-points), and extension points for exploration (fastexplore) and explanation (fastexplain) are architecturally isolated from model training to avoid indirect leakage.

Implications and Future Directions

From a methodological perspective, fastml enforces best practices for evaluation integrity, shifting the baseline expectation from user-determined discipline to software-enforced correctness. The architecture reduces spurious findings, especially in biomedical and multi-site studies prone to subtle cross-fold contamination. In practical terms, workflow assembly, resampling configuration, and auditability are unified at the interface, reducing analysis code surface and opportunity for human error.

Several limitations remain. Leakage prevention is pipeline-scoped: trusted user inputs are still required. Audit and security guards are not exhaustive; runtime side channels and external process manipulations are beyond R-level mitigation. Survival resampling support is more constrained than for standard prediction tasks, and deployment object portability is contingent on R/package versioning and engine implementations. Stacking and super learner model combinations are identified as an area for future expansion, with strong emphasis on maintaining explicit leakage constraints.

Further directions include integrated and transparent model stacking, improved deployment ergonomics, and more sophisticated audit instrumentation at the recipe and engine layer. These developments would maintain the emphasis on explicit, reproducible, and leakage-resistant evaluation—principles that are essential as pipelines scale in complexity and are increasingly applied in sensitive and high-stakes inferential contexts.

Conclusion

fastml defines a paradigm for AutoML in R that prioritizes default correctness and auditability through guarded resampling workflows, structural and behavioral code audits, and comprehensive support for classification, regression, and survival learning. Strong numerical findings demonstrate that common preprocessing misorderings can cause severe inflation in apparent model performance, and that fold-local preprocessing, enforced by fastml’s architecture, eliminates this major source of bias. The interface supports efficient benchmarking, model explanation, and extensibility without sacrificing methodological rigor. This positions fastml as a robust, safety-forward foundation for reproducible, honest, and extensible AutoML in R (2604.05225).