OMEGA: Optimizing Machine Learning by Evaluating Generated Algorithms

Abstract: In order to automate AI research we introduce a full, end-to-end framework, OMEGA: Optimizing Machine learning by Evaluating Generated Algorithms, that starts at idea generation and ends with executable code. Our system combines structured meta-prompt engineering with executable code generation to create new ML classifiers. The OMEGA framework has been utilized to generate several novel algorithms that outperform scikit-learn baselines across a robust selection of 20 benchmark datasets (infinity-bench). You can access models discussed in this paper and more in the python package: pip install omega-models.

• The paper presents an end-to-end system that autonomously generates and benchmarks novel ML classifiers via LLM-based prompt engineering.
• It introduces recursive self-healing for error correction in code generation, enabling robust and reproducible model synthesis.
• Empirical results show models like MetaSynthesisClassifier and DirectionalForest consistently outperform traditional scikit-learn algorithms.

OMEGA: Optimizing Machine Learning by Evaluating Generated Algorithms

Introduction and Motivation

OMEGA presents a comprehensive end-to-end system for autonomous ML algorithm discovery, bridging prompt-driven hypothesis generation to executable and benchmarked ML code. Unlike prior AutoML and NAS frameworks that optimize within the constraints of established architectures, OMEGA delegates the generation, validation, and evaluation of entirely new classifier logic to LLMs. By targeting the under-explored problem of algorithm synthesis rather than parameter tuning, it repositions LLMs as functional agents in computational science pipelines.

Figure 1: OMEGA framework: prompts (human/LLM-generated) produce ML code, which undergoes auto-validation and benchmarking on diverse datasets for standardized evaluation.

OMEGA Framework and Methodology

OMEGA’s architecture incorporates structured prompt engineering with LLM-based code generation, self-healing feedback for robust code validation, and systematic evaluation on a large curated benchmark, "infinity-bench". Model ideas originate either from direct human prompts or are autonomously generated by LLMs, each prompt yielding a candidate estimator. The framework enforces strict scikit-learn API compliance (requiring .fit() and .predict()), facilitating plug-and-play integration with standard workflows.

A critical innovation is the recursive error correction ("self-healing"): generated code encountering execution failures automatically loops stacktraces back as context for further repair cycles, reducing human intervention and fostering reproducibility.

For objective model selection, OMEGA utilizes min-max normalized accuracy across twenty OpenML/scikit-learn datasets, aggregating per-dataset ranks to produce a robust comparative score. This yields an empirical manifold for evaluating the generalization of generated estimators relative to strong baselines.

Empirical Results and Model Analysis

OMEGA synthesizes a suite of novel classifiers—some prompted by humans, others by LLMs—automatically ranked against canonical scikit-learn models. The results on infinity-bench demonstrate that several OMEGA-generated classifiers consistently outperform RandomForest, MLPClassifier, and other established baselines.

Figure 2: Comparative bar chart of top OMEGA-generated models versus scikit-learn baselines, measured by mean normalized min-max score.

Significantly, the MetaSynthesisClassifier and DirectionalForest—representing meta-learning and directionality-based feature engineering, respectively—achieve high aggregate performance, surpassing most traditional ensemble models.

MetaSynthesisClassifier

This model formalizes stacked generalization by synthesizing predictions from heterogeneous base estimators (including logistic regression, random forest, decision tree) via an additional meta-learner. Training employs out-of-fold predictions to construct meta-features, reducing data leakage. The meta-estimator is then trained on these synthesized features to optimize a secondary mapping to the ground truth.

Figure 3: Dataset-wise accuracy for MetaSynthesisClassifier compared to leading scikit-learn baselines, highlighting consistent overperformance or parity across the benchmark.

Empirical results show the MetaSynthesisClassifier achieves robust generalization, particularly due to its ability to adaptively weight base learners’ contributions across manifold complexities.

DirectionalForest

The DirectionalForest augments classical random forest logic by pre-computing a "directionality" vector for features, oriented by aggregate class-wise mean deviations. Each input is rescaled by this vector prior to tree induction, enforcing consistent alignment of the feature space. Ensembles of such directional trees are then aggregated via majority vote.

Figure 4: Dataset-level evaluation of DirectionalForest versus scikit-learn benchmarks, illustrating advantage on several low-bias, high-variance classification tasks.

This approach reduces variance in high-dimensional regimes and provides measurable gains on structurally diverse datasets.

Comparative Study of LLMs and Self-Improvement Experiments

A systematic ablation across four large code-generating LLMs—Anthropic Claude Sonnet 4.5, OpenAI GPT-4.1 mini, Google Gemini 2.5 Flash, xAI grok-code-fast-1—reveals Gemini 2.5 Flash outperforms on the majority of prompts in terms of benchmarked model accuracy, although overall spread between LLMs narrows with high-quality prompt engineering. These observations suggest that iterative prompt optimization, rather than subsequent code tweaking, is the critical determinant for generating effective classifiers.

OMEGA further implements recursive self-improvement, contrasting model quality improvements from prompt iteration versus direct code mutation. Prompt optimization yields superior average performance gains across multiple LLMs, reinforcing the value of prompt-centric frameworks for generative ML research.

Figure 5: Average model performance improvements achieved via prompt versus code iteration loops, demonstrating stronger gains for prompt-based refinement across all major LLMs.

Practical and Theoretical Implications

OMEGA operationalizes LLMs as closed-loop research agents, capable of autonomously generating, repairing, and benchmarking executable ML logic. The adoption of robust self-healing and standardized evaluation protocols positions the framework as a reproducible testbed for large-scale algorithmic search, with potential to extend beyond tabular classification into unsupervised, structured, or even neural model synthesis domains as LLM context and reasoning capacity increase.

Practically, the deployment of OMEGA-generated models via a pip-installable package ("omega-models") can democratize adoption by practitioners and lower the technical barrier for incorporating novel state-of-the-art estimators into real-world pipelines.

Theoretically, OMEGA reinforces the paradigm shift from parameter-centric to framework- and prompt-centric automation in ML, with the implication that future algorithmic progress may rely more on advances in LLM reasoning, prompt design, and autonomous benchmarking than on handcrafted architectural design.

Conclusion

OMEGA demonstrates a functional realization of executable ML algorithm discovery by LLMs, leveraging prompt-driven synthesis, self-healing validation, and robust comparative benchmarking. The system produces classifiers that outperform widely used scikit-learn models, validates the utility of prompt optimization, and serves as a stepping stone for fully automated ML research. Future directions will likely involve scaling framework applicability to broader domains and coupling deeper integration with emerging LLM capabilities.

Reference: "OMEGA: Optimizing Machine Learning by Evaluating Generated Algorithms" (2604.26211)