Residual-Space Evolutionary Optimization via Flow-based Generative Models

Abstract: Data editing with generative methods typically requires differentiable objectives and gradient-based search. However, these assumptions break down in flow-based settings, where edits are performed through forward and backward integration and often involve non-differentiable or black-box objectives. We introduce residual-space evolutionary optimization, a model-agnostic framework that addresses this gap by combining flow-based generative editing with evolutionary algorithms. Building on the observation that conditional flow matching (CFM) can disentangle condition-controlled factors from instance-specific residuals, our framework directly operates in residual space and separates two complementary search regimes: self-pollination performs local exploitation through feature-preserving residual refinement, and cross-pollination promotes broader exploration by recombining residuals across heterogeneous samples. As a proof of concept, we validate on MorphoMNIST, a benchmark dataset for counterfactual generation, and on crystal data, demonstrating that this exploration--exploitation decomposition provides a useful mechanism for balancing target alignment, instance preservation, and diversity, and extends beyond images to real-world scientific domains.

• The paper introduces a residual-space evolutionary optimization method that decouples semantic control from instance-specific residuals using conditional flow matching.
• It leverages both self-pollination and cross-pollination in residual space to achieve >99.9% target validity and enhanced diversity in image and crystal design tasks.
• The method is model-agnostic, offering a framework adaptable to various domains, including counterfactual explanations and scientific materials optimization.

Residual-Space Evolutionary Optimization with Flow-based Generative Models

Introduction and Motivation

Residual-space evolutionary optimization (RSEO) introduces a robust, model-agnostic framework for controlled data editing in generative modeling contexts with black-box or non-differentiable objectives, targeting domains where standard gradient-based methods are inapplicable due to the architecture of flow-based generative models. The core insight is that conditional flow matching (CFM) decouples condition-controlled semantics from instance-specific residuals, thus providing a manipulable genomic residual space. This genomic abstraction enables the application of evolutionary search—specifically, mutation (self-pollination) and crossover (cross-pollination)—directly on instance-specific residuals while using fixed conditional generative flows for semantic imposition.

Methodology

The principle mechanism is a two-stage procedure built on frozen conditional flows: first, data is lifted into a residual state devoid of condition-specific information via reverse-time integration through a learned conditional flow. Second, candidate residuals are decoded under target semantics by forward integration (landing). Evolutionary operations are executed in this residual space, effectively transforming the search domain from pixels or standard latents to a disentangled representation where only instance variation is mutated.

Self-pollination is realized by perturbing a single residual via additive isotropic Gaussian noise or feature swapping, constituting local exploration (exploitation) around an existing instance—appropriate for objectives demanding high instance fidelity (e.g., counterfactual explanations).

Cross-pollination extends search globally by recombining residuals (via linear or dimension-wise crossover) from a diverse set of source samples, enabling enhanced coverage and diversity in the search space—especially relevant for discovering rare or maximally-featured instances under a specific target condition.

Selection is operationalized through custom, potentially black-box, fitness functions that can incorporate target validity, instance similarity, feature value, and population diversity. The evolutionary cycle proceeds independently of gradient information from the generative or scoring models, making it well-suited for intractable or empirical objective assessment.

Empirical Results

Image Domain (MorphoMNIST)

RSEO was instantiated on MorphoMNIST using a VAE-based encoder/decoder, class-conditional CFM, and a digit classifier.

• Self-pollination achieved >99.9% target validity, improving source similarity by ~3% over leap-only editing, with consistent improvements across all target classes. This indicates that residual mutation enables fine local refinement without sacrificing target alignment.
• Cross-pollination with diverse residuals maintained perfect validity and increased both the maximal morphological feature value (e.g., digit thickness) and population diversity compared to homogeneous (target-class-only) populations. Improvements were observed both in mean and top-percentile feature scores, and in pairwise image diversity metrics.

Scientific Domain (Crystal Structures)

The approach was extended to WyCryst, with residual editing targeting maximal band gap materials under target crystal system constraints.

• Diverse cross-pollination significantly increased latent-space diversity (Δ=+1.2 Euclidean mean pairwise distance), at a slight cost to maximal band gap (Δ=-0.08 eV), reflecting a trade-off between exploration and feature optimization in highly heterogeneous data. Validity remained perfect across generations and crystal systems, showing the framework's robustness in expanding scientific search domains beyond image data.

Theoretical Implications

This work demonstrates that leveraging residual structure in flow-based models decouples feature control from instance-specificity, permitting effective application of classical genetic algorithm constructs in the generative editing context. The approach is inherently model-agnostic and post hoc, requiring only an existing conditional flow and optionally pretrained scoring functions. Thus, it generalizes to any domain where an editable latent or residual representation is available, including molecular and materials optimization, allowing empirical, constraint-driven oracles to be seamlessly incorporated.

Diverse cross-pollination's performance suggests that residual recombination can effectively traverse non-convex, disjoint search spaces by mixing instance features across heterogeneous data to achieve novel semantic targets. However, the exploitation-exploration decomposition highlights and quantifies inherent trade-offs in evolutionary latent search, particularly as the heterogeneity of domains increases.

Practical Implications and Future Directions

Practically, RSEO provides a lightweight optimization layer for any flow-based editor, facilitating the use of black-box or empirical evaluations (e.g., physical simulations, user studies) within generative pipelines. This is particularly valuable in physical sciences and engineering, where differentiable surrogate models are often unavailable or unreliable.

Several key avenues remain open:

• Systematic ablation studies on mutation/crossover regimes, selection heuristics, and population hyperparameters are required for robust deployment in high-dimensional or highly-complex generative domains.
• Integration with more sophisticated selection (e.g., multi-objective, Pareto-front) or crossover designs (e.g., semantic-aware, dynamic adaptation) could further mitigate the exploration-exploitation tension observed in scientific data.
• Explicit incorporation of physical validity or stability checks (e.g., via external simulation) during candidate selection may enable RSEO to serve as a driver for guided scientific discovery beyond proof-of-concept scenarios.

Conclusion

Residual-space evolutionary optimization with flow-based generative editors establishes a flexible, model-agnostic bridge between classical EA methods and contemporary generative ML pipelines, removing the restriction of differentiable or transparent objectives. The proposed method empirically balances fidelity, target alignment, and search diversity in both image and scientific data domains, thus providing a promising paradigm for generative editing and inverse design in black-box settings. Expansion to broader scientific optimization problems and a deeper investigation of domain-specific genotype abstractions represent compelling directions for future research.

Reference: "Residual-Space Evolutionary Optimization via Flow-based Generative Models" (2606.20084)