ease.ml/ci: CI for ML Models

• ease.ml/ci is a continuous integration system designed to integrate machine learning models into reproducible pipelines with strict (ε, δ) guarantees.
• It overcomes traditional CI limitations by addressing stochastic model properties, adaptivity challenges, and high labeling costs through principled statistical methods.
• The system utilizes a specialized DSL and variance-based optimizations to compute sample sizes and reduce labeling efforts, ensuring reliable test outcomes.

ease.ml/ci is a continuous integration (CI) system specifically designed to integrate ML models into reproducible engineering pipelines while providing rigorous statistical guarantees on integration conditions. Unlike conventional CI engines, which treat ML models as black-box artifacts and are not equipped to verify stochastic properties such as accuracy or model stability, ease.ml/ci introduces a principled framework that addresses the unique challenges of ML development—most notably, the high labeling cost required for statistically valid evaluation and the adaptivity arising from iterative model tuning (Renggli et al., 2019).

1. Motivation and System Objectives

The main objective of ease.ml/ci is to bring the discipline and automation of software CI to the development life cycle of ML models, comprising design, implementation, tuning, testing, and deployment. Classical CI approaches, such as measuring software correctness via fixed test cases, are inadequate for ML systems where model quality is inherently probabilistic and evolves across commits. Naive application of statistical bounds (e.g., applying Hoeffding’s inequality for accuracy estimation at each commit) results in label requirements that are prohibitive for nontrivial error tolerances—on the order of tens of thousands of labels for error tolerance $\epsilon = 0.01$ and risk $\delta = 10^{-4}$ per test (requiring approximately $63,000$ labels per test for such settings). Furthermore, adaptivity—where developers respond to pass/fail feedback between commits—forces stricter statistical union bounds, inflating the required sample size. ease.ml/ci is engineered to allow users to specify customizable, high-level integration conditions on model quality with rigorous $(\epsilon, \delta)$ guarantees, all while keeping human labeling costs practical (a few thousand labels per month in production settings) (Renggli et al., 2019).

2. System Architecture and End-to-End Workflow

The ease.ml/ci system comprises four main workflow steps:

1. Script and Condition Specification: Users write an extended CI script (compatible with formats such as .travis.yml) that specifies:
   • The script command, which runs both the new and the previous model and collects per-example statistics.
   • A logical integration condition $F(n,o,d)$, where $n$ denotes the accuracy of the new model, $o$ the accuracy of the old model, and $d$ the fraction of predictions that differ between models.
   • Statistical guarantees: reliability requirement $1-\delta$ and error tolerance $\epsilon$.
   • Operational parameters: mode (either false-positive-free or false-negative-free), adaptivity (choices include none, full, or firstChange), and a maximum number of testing steps $\delta = 10^{-4}$0.
2. Sample Size Determination: The system’s sample-size estimator parses the specified conditions and policy choices to compute the minimum required labeled test set size $\delta = 10^{-4}$1.
3. Commit Evaluation Loop: For each developer commit, the CI engine executes the designated script, gathers empirical statistics $\delta = 10^{-4}$2 on the $\delta = 10^{-4}$3 labeled examples (and any requisite unlabeled pool), and applies the logical condition $\delta = 10^{-4}$4 using appropriately computed confidence intervals.
4. Outcome Reporting and Testset Management: A PASS/FAIL verdict is returned (possibly hidden from the developer if adaptivity is disabled). The step budget $\delta = 10^{-4}$5 is decremented. Upon exhaustion of the step budget, or immediately after a decisive PASS (depending on mode), the system notifies the user to obtain a new labeled test set.

A summary of key workflow configuration fields is displayed below.

Field	Purpose	Options/Format
script	Test command for running models and stats	Shell/Python command
condition	Logical formula $\delta = 10^{-4}$6 over accuracy metrics	Textual DSL
reliability	Required reliability, $\delta = 10^{-4}$7	e.g., 0.9999
mode	Error type to avoid: fp-free or fn-free	fp-free, fn-free
adaptivity	Developer feedback visibility	full, none, firstChange
steps	Number of workflow steps before dataset refresh	Integer

3. Domain-Specific Language for Integration Conditions

ease.ml/ci introduces a concise domain-specific language (DSL) enabling users to specify probabilistic integration conditions over test statistics:

• Variables: $\delta = 10^{-4}$8, $\delta = 10^{-4}$9, and $63,000$0 represent, respectively, new-model accuracy, old-model accuracy, and the fraction of differing predictions, each in $63,000$1.
• Expressions: Support arithmetic combinations and tolerance specifications, e.g., $63,000$2.
• Logical Formulas: Conditions may be singular or conjunctive, such as $63,000$3.
• Configuration Fields: DSL scripts include script, textual condition, reliability, mode, adaptivity, and steps.

This formalism allows precise, testable specifications for model integration, with guarantees tied directly to the user’s reliability and tolerance requirements.

4. Statistical Guarantees, Adaptivity, and Modes

Statistical rigor in ease.ml/ci is established through $63,000$4-style guarantees:

• Error Tolerance ($63,000$5): Half-width of the confidence interval on the aggregate metric or condition.
• Failure Probability ($63,000$6): Probability that PASS/FAIL is incorrect for any type-I or type-II error over all $63,000$7 test steps.

The reporting modes define how ambiguity is resolved:

• False-Positive-Free (fp-free): Ambiguous results default to FAIL; any returned PASS implies the true condition holds with probability at least $63,000$8.
• False-Negative-Free (fn-free): Ambiguous results default to PASS; any FAIL is guaranteed to be a violation with probability at least $63,000$9.

Adaptivity has a direct bearing on sample complexity:

• Non-Adaptive: No developer feedback; a union bound over $(\epsilon, \delta)$0 models suffices, with sample size growing as $(\epsilon, \delta)$1.
• Fully Adaptive: Feedback per commit permits potential overfitting; a union bound over all $(\epsilon, \delta)$2 histories is required, increasing sample size proportional to $(\epsilon, \delta)$3.
• Hybrid (firstChange): Adaptivity ceases after first decisive outcome; sample complexity again scales as $(\epsilon, \delta)$4 but mandates immediate data refresh on PASS.

5. Sample Complexity and Optimizations

The baseline approach for ensuring statistical validity applies Hoeffding’s inequality to each logical or arithmetic clause, allocating the $(\epsilon, \delta)$5 and $(\epsilon, \delta)$6 tolerances across variables and test steps. However, this strategy is label-inefficient for realistic settings. Sample sizes for a single clause with tolerance $(\epsilon, \delta)$7, $(\epsilon, \delta)$8, and $(\epsilon, \delta)$9 are approximately $F(n,o,d)$0 (non-adaptive) to $F(n,o,d)$1 (fully adaptive), rendering it impractical for fine-grained conditions.

ease.ml/ci includes optimizations that, particularly in fine-tuning phases with small $F(n,o,d)$2 (few prediction differences), yield up to two orders of magnitude reduction in label requirements:

• Difference-based Optimization: When CI conditions include both $F(n,o,d)$3 and $F(n,o,d)$4, the engine first filters on $F(n,o,d)$5 using unlabeled data. If $F(n,o,d)$6 is acceptably small, Bennett’s inequality is used to test $F(n,o,d)$7 within tighter variance bounds, significantly reducing $F(n,o,d)$8 as compared to Hoeffding’s baseline.
• Active Labeling: When only $F(n,o,d)$9 fraction of examples differ, only entries with differing predictions between old and new models need to be labeled for evaluating $n$0, effectively reducing human labeling cost to $n$1 per commit.
• Implicit Variance Bound Optimization: Even absent an explicit $n$2-clause, ease.ml/ci estimates $n$3 from an unlabeled pool, and for small $n$4, applies Bennett’s inequality to $n$5.
• Tight Numerical Bounds: For pure Bernoulli queries, inversion of the Binomial CDF enables more precise sample size estimation than reliance on asymptotic bounds.

6. Example Usage and Case Studies

Example CI scripts illustrate the interplay of configuration options:

• Non-Adaptive, False-Negative-Free:

$o$4 This setting requires approximately $n$6 labels. One labeling phase supports 32 commits, after which a new test set is requested.

• Fully Adaptive, False-Positive-Free, Difference-Based:

$o$5 With $n$7, active labeling reduces labeling requirements to approximately 670 labels per commit, which is feasible for production workflows.

In a real-world case study on the SemEval-2019 Task 3 emotion-classification challenge, difference-based and active labeling optimizations reduced the required test set size from an initial estimate of 44,000 labels (Hoeffding-based) to only $n$8 labels with a tolerance of $n$9, $o$0. All evaluation modes in this case study selected the correct “best” model from the development sequence (Renggli et al., 2019).

7. Limitations and Future Directions

Support in ease.ml/ci is presently focused on accuracy-based metrics. Extending to metrics such as $o$1, AUC, and others requires adapting alternative concentration inequalities (e.g., McDiarmid’s) and sensitivity analyses. Ratio and order statistics, including ratios like $o$2, ranks, and top-$o$3 accuracies, pose further challenges due to the lack of simple direct concentration bounds. Additional areas for development include detection of concept drift (online monitoring for changes in a fixed model), incorporation of privacy guarantees (e.g., differential privacy approaches to limit adaptivity-induced overfitting), and improved management of unlabeled data pools for efficient active labeling and sample stationarity (Renggli et al., 2019).

In summary, ease.ml/ci marks the first CI system that comprehensively treats ML models as first-class entities, combines a succinct DSL for expressing testable probabilistic conditions, provides automated sample complexity management and test set refresh strategies, and leverages variance-based and active labeling optimizations to make continuous integration of ML practical under strict statistical guarantees.