Graybox Machine-Learning Framework

• Graybox machine-learning frameworks are hybrid models that combine fixed physics-based or analytic components with flexible learned maps to capture unknown processes.
• They separate known deterministic layers from unknown mappings, enabling improved accuracy and interpretability in quantum control, epidemiological calibration, and noise characterization.
• Experimental implementations show superior performance over purely whitebox or blackbox models, achieving lower mean squared errors and higher fidelity in various applications.

Graybox machine-learning frameworks are hybrid modeling strategies that combine a mechanistic, analytic, or physics-based component with a learned component, thereby occupying an intermediate position between whitebox models and blackbox models. Across quantum control, Bayesian optimization, structured probabilistic inference, evolutionary optimization, software testing, and interpretability systems, the common pattern is to expose internal computational or physical structure—such as Hamiltonians, unitaries, compartment trajectories, subfunction decompositions, or interpretable additive terms—while using machine learning or probabilistic surrogates for the unknown map that is difficult to specify a priori (Youssry et al., 2022, Niu et al., 2024, Astudillo et al., 2022).

1. Concept and terminology

In the control-engineering sense adopted in experimental quantum system identification, a graybox model merges an abstract mathematical structure, such as a neural network, with physical laws. In that usage, a whitebox model is a fully mechanistic model derived from a known physical parameterization, a blackbox model maps inputs directly to outputs with no explicit physics inside, and a graybox model places a learned map upstream of fixed, differentiable physics layers that enforce constraints such as Hermiticity, unitary evolution, and Born’s rule (Youssry et al., 2022).

In Bayesian optimization, grey-box methods are defined more generally as methods that leverage access to the internal computational structure of objective-function or constraint evaluation. The key distinction is not merely the presence of prior knowledge about a function, but explicit use of internal structure such as composite objectives, constituents, or fidelity controls inside the surrogate-and-acquisition loop (Astudillo et al., 2022). In nested-function optimization, a grey-box objective is a factorable composition of white-box and black-box elementary functions, represented through an augmented state and a sequence of intermediate variables (Xu et al., 2023).

Related literature uses neighboring terms with stricter transparency requirements. In program synthesis, a “glass-box” loss is a scoring program whose source code is available to the synthesizer, rather than an opaque oracle (Christakopoulou et al., 2017). In interpretability systems, “glassbox models” are intrinsically interpretable models, whereas “blackbox explainability” denotes post-hoc explanations of opaque predictors (Nori et al., 2019). This suggests that graybox denotes partial access to internal structure rather than complete transparency.

2. Structural pattern of graybox frameworks

A recurring architectural principle is explicit separation of known and unknown structure. The known structure is encoded as deterministic layers, equations, or graph relations; the unknown structure is represented by a flexible learned map. In the quantum-control formulation, the neural component maps control voltages to a candidate Hamiltonian, after which fixed physics layers impose Hermiticity, compute $U(\mathbf{V}) = e^{-iH(\mathbf{V})T}$, evolve basis states, and return Born-rule probabilities $$P_{j\to k}(\mathbf{V}) = |\langle k|U(\mathbf{V})|j\rangle|^2$$ (Youssry et al., 2022). In epidemiological calibration, Gaussian processes are placed on simulator outputs $\mathbf{y}(x)\approx \eta(x)$, while the calibration loss is kept as a known composite function $f(x)=g(\mathbf{y}(x))$, and dependencies among compartment trajectories are encoded as a function network (Niu et al., 2024). In probabilistic quantum characterization, a Bayesian neural network predicts the parameters of a learned effective operator $\hat W_O(\Theta)$, while the ideal unitary evolution remains analytic (Pathumsoot et al., 29 Sep 2025).

Setting	Learned component	Fixed structure
Quantum system identification	NN map from controls to Hamiltonian entries	Hermiticity, unitary evolution, Born’s rule (Youssry et al., 2022)
Epidemiological calibration	GP surrogate on compartment outputs	SIQR ODE structure, composite loss, function network (Niu et al., 2024)
Probabilistic quantum characterization	BNN for effective observable or noise operator	Ideal Hamiltonian, ideal unitary, measurement model (Pathumsoot et al., 29 Sep 2025)

This architecture is often summarized by the principle “use ML for the unknown map, not for the physics itself.” In the quantum literature, the unknown map may be $\mathbf{V}\mapsto H$, $\vec\theta\mapsto V_O$, or $(\tau,\phi,f_B,\vec\chi)\mapsto \hat V_Z$; the fixed part may be Schrödinger evolution, matrix exponentials, measurement rules, or tomographic reconstruction (Youssry et al., 2022, Mayevsky et al., 16 Jun 2025, Youssry et al., 24 Jan 2026).

A second recurring principle is latent-variable exposure. Graybox models are valued not only because they fit observables, but because they expose internal quantities unavailable to a purely supervised blackbox. In the photonic qutrit experiment, the graybox provides access to $H(\mathbf{V})$, $U(\mathbf{V})$, and evolved states even though only measurement probabilities are used for training (Youssry et al., 2022). In qudit control under realistic noise, the learned objects are observable-specific noise operators $$P_{j\to k}(\mathbf{V}) = |\langle k|U(\mathbf{V})|j\rangle|^2$$0, and a local analytic expansion $$P_{j\to k}(\mathbf{V}) = |\langle k|U(\mathbf{V})|j\rangle|^2$$1 is introduced as an interpretability mechanism (Mayevsky et al., 16 Jun 2025).

3. Learning, inference, and decision mechanisms

Training objectives in graybox frameworks depend on the observable layer at which supervision is available. In quantum system identification with a fixed-time, closed qutrit device, the training loss is mean squared error between predicted and measured probability vectors, optimized with Adam; because the complete computation from controls to probabilities is differentiable, gradients are obtained by automatic differentiation through complex matrix exponentials and linear algebra operations (Youssry et al., 2022). In qudit noise characterization, the training target is again an MSE between predicted and simulated expectation values, while control is performed later by minimizing a gate cost

$$P_{j\to k}(\mathbf{V}) = |\langle k|U(\mathbf{V})|j\rangle|^2$$2

over pulse parameters (Mayevsky et al., 16 Jun 2025). In noisy-qubit control, the graybox is used as a differentiable emulator inside a gradient-based optimal-control loop, with gate infidelity

$$P_{j\to k}(\mathbf{V}) = |\langle k|U(\mathbf{V})|j\rangle|^2$$3

as the objective (Cantone et al., 18 Jul 2025).

Probabilistic graybox models replace point estimation with posterior inference over the learned component. In probabilistic quantum characterization, a Bayesian neural network is used for the blackbox part, with prior $$P_{j\to k}(\mathbf{V}) = |\langle k|U(\mathbf{V})|j\rangle|^2$$4, Bernoulli or binomial likelihoods for binary measurement data, and a variational posterior $$P_{j\to k}(\mathbf{V}) = |\langle k|U(\mathbf{V})|j\rangle|^2$$5 learned by maximizing the ELBO

$$P_{j\to k}(\mathbf{V}) = |\langle k|U(\mathbf{V})|j\rangle|^2$$6

(Pathumsoot et al., 29 Sep 2025). In Bayesian quantum sensing, the trained graybox furnishes the likelihood $$P_{j\to k}(\mathbf{V}) = |\langle k|U(\mathbf{V})|j\rangle|^2$$7 used in Bayesian posterior updates for the unknown Larmor frequency or magnetic field (Youssry et al., 24 Jan 2026).

Graybox Bayesian optimization follows a different but structurally analogous logic. For composite objectives $$P_{j\to k}(\mathbf{V}) = |\langle k|U(\mathbf{V})|j\rangle|^2$$8, the surrogate is placed on $$P_{j\to k}(\mathbf{V}) = |\langle k|U(\mathbf{V})|j\rangle|^2$$9, not on the scalar $\mathbf{y}(x)\approx \eta(x)$0. Expected improvement for composite functions is written as

$\mathbf{y}(x)\approx \eta(x)$1

with Monte Carlo and reparameterization used for optimization (Astudillo et al., 2022). In epidemiological calibration, this becomes a GP surrogate over SIQR compartment trajectories, a known negative-MSE functional $\mathbf{y}(x)\approx \eta(x)$2, and a Knowledge Gradient acquisition defined on the output space rather than on the scalar loss (Niu et al., 2024). In structured Gaussian-process inference, the same gray-box principle appears in variational form: the framework exploits GP priors and linear-chain likelihood structure, yet does not require model-specific derivations of the structured likelihood, and estimates the ELBO using expectations over low-dimensional Gaussians together with control variates and SAGA-style stochastic optimization (Galliani et al., 2016).

4. Quantum realizations

Quantum control and characterization provide the most explicit realizations of graybox machine-learning frameworks. In experimental qutrit system identification on a three-mode lithium-niobate integrated photonic chip controlled by four electrodes, the graybox model outperformed the whitebox model while preserving access to Hamiltonians and unitaries. The reported training and testing MSEs were $\mathbf{y}(x)\approx \eta(x)$3 and $\mathbf{y}(x)\approx \eta(x)$4 for graybox, compared with $\mathbf{y}(x)\approx \eta(x)$5 and $\mathbf{y}(x)\approx \eta(x)$6 for the whitebox model. For output-distribution control on 1000 random targets, average fidelity between experimentally realized distributions and targets was $\mathbf{y}(x)\approx \eta(x)$7 for graybox, $\mathbf{y}(x)\approx \eta(x)$8 for whitebox, and $\mathbf{y}(x)\approx \eta(x)$9 for blackbox; for gate control on 1000 random unitaries, average gate fidelity was $f(x)=g(\mathbf{y}(x))$0 for graybox and $f(x)=g(\mathbf{y}(x))$1 for whitebox (Youssry et al., 2022).

The same design pattern extends from closed, time-independent systems to open and time-dependent ones. In arbitrary-dimensional qudit control, the whitebox path computes exact noiseless Hamiltonian evolution without the rotating-wave approximation, while the blackbox path predicts observable-specific noise operators $f(x)=g(\mathbf{y}(x))$2 through GRUs and dense layers constrained to produce Hermitian operators with bounded spectra. The framework was used for both global $f(x)=g(\mathbf{y}(x))$3 operations and two-level subspace gates, and a local analytic expansion

$f(x)=g(\mathbf{y}(x))$4

was introduced to interpret how noise responds to control perturbations (Mayevsky et al., 16 Jun 2025). In the qutrit example, closed-system global gate infidelities were reported below $f(x)=g(\mathbf{y}(x))$5, while weak-noise and strong-noise infidelities were in the ranges $f(x)=g(\mathbf{y}(x))$6–$f(x)=g(\mathbf{y}(x))$7 and $f(x)=g(\mathbf{y}(x))$8–$f(x)=g(\mathbf{y}(x))$9, respectively (Mayevsky et al., 16 Jun 2025).

For a noisy single qubit with Markovian and non-Markovian dynamics, a graybox framework combining physics-informed equations with a lightweight transformer neural network learned an effective operator that predicts observables accurately under random-telegraph and Ornstein–Uhlenbeck noise. Using the model as a dynamics emulator, gradient-based optimal control produced fidelities above $\hat W_O(\Theta)$0 for the lowest considered coupling and above $\hat W_O(\Theta)$1 for the highest (Cantone et al., 18 Jul 2025). In Bayesian quantum sensing on a single-spin solid-state sensor, a graybox model trained on roughly 10,000 prior experimental datapoints yielded several orders of magnitude improvement in mean squared error over the corresponding physics-only model when estimating a static magnetic field in a Bayesian loop (Youssry et al., 24 Jan 2026).

Recent work has also emphasized uncertainty quantification and finite-shot effects. Probabilistic graybox characterization with Bayesian neural networks uses binary measurement outcomes directly for inference and reports that the probabilistic model outperforms the original graybox by up to $\hat W_O(\Theta)$2 times in capturing the distribution of observed data (Pathumsoot et al., 29 Sep 2025). In superconducting-qubit calibration with finite-shot data, the decomposition of expected MSE loss shows that finite-shot estimation of expectation values is the main contribution to the minimum achievable expected MSE loss, and the expected loss is shown to be an upper bound on the expected absolute error of average gate fidelity between exact value and model prediction (Pathumsoot et al., 18 Aug 2025).

5. Grey-box Bayesian optimization and structured optimization

Grey-box optimization frameworks recast objective evaluation itself as a structured computation. In the tutorial literature on grey-box Bayesian optimization, the canonical formulation is a composite function

$\hat W_O(\Theta)$3

or, in multi-fidelity settings, a target-fidelity objective $\hat W_O(\Theta)$4. The surrogate is built on the internal function $\hat W_O(\Theta)$5, not directly on $\hat W_O(\Theta)$6, and the acquisition operates over enriched action spaces such as $\hat W_O(\Theta)$7 or constituent indices (Astudillo et al., 2022). This formulation covers composite objectives, constituent evaluations, and multi-fidelity optimization in a unified way.

Epidemiological calibration makes this concrete. In SIQR model calibration, the graybox BO scheme places Gaussian processes on compartment trajectories rather than on the scalar loss, uses the negative MSE

$\hat W_O(\Theta)$8

and encodes epidemiological dependencies through a function network over $\hat W_O(\Theta)$9 outputs. A decoupled acquisition introduces a binary vector $\mathbf{V}\mapsto H$0 indicating which GP components to update, normalizing information gain by $\mathbf{V}\mapsto H$1 (Niu et al., 2024). The reported experiments show that graybox variants improve calibration performance measured by the logarithm of mean square errors and achieve faster performance convergence in terms of BO iterations on synthetic and real COVID-19 datasets (Niu et al., 2024).

A more general theory appears in optimization of nested grey-box functions. There the objective is written as a chain of intermediate variables $\mathbf{V}\mapsto H$2 and elementary functions $\mathbf{V}\mapsto H$3, some white-box and some black-box, with

$\mathbf{V}\mapsto H$4

An optimism-driven algorithm maintains GP confidence bounds for black-box components, solves an auxiliary optimization problem over decision variables and intermediate variables, and achieves regret bounds of the same order as standard black-box BO up to multiplicative constants depending on downstream Lipschitz constants (Xu et al., 2023). This makes explicit that structural exploitation need not degrade asymptotic BO guarantees.

In evolutionary optimization, the same gray-box principle is operationalized through partial evaluations. The GOMEA library defines a gray-box objective as

$\mathbf{V}\mapsto H$5

where each subfunction depends only on a subset of variables and partial evaluation updates a fitness buffer by recomputing only the affected subfunctions after a local modification (Bouter et al., 2023). Together with linkage models such as Family Of Subsets, linkage trees, and the Gene-pool Optimal Mixing operator, this yields strong performance in settings where limited domain knowledge about the subfunction structure is available (Bouter et al., 2023).

6. Broader variants, misconceptions, and limitations

Graybox is not a single model family; it is a design pattern. In software testing, Vulseye is a stateful directed graybox fuzzer that combines static analysis, pattern matching, backward analysis over contract state, and runtime feedback from code space and state space. Its fitness integrates CodeDistance, StateDistance, branch coverage, and state-dependence feedback, and the reported evaluation shows superior effectiveness and efficiency over state-of-the-art fuzzers on smart contracts (Liang et al., 2024). In program synthesis, glass-box optimization exposes the scoring function as source code, and learning conditions the search policy on the structure of that scoring program rather than on input–output examples alone (Christakopoulou et al., 2017). In interactive optimization, a glass-box human-in-the-loop framework exposes the internal state of an ant-colony optimization process through a Human-Interaction-Matrix and a Human-Impact-Factor, so that a user can intervene during search rather than merely before or after it (Holzinger et al., 2017).

A common misconception is to equate graybox with interpretability alone. InterpretML makes a sharper distinction: glassbox models are intrinsically interpretable models such as linear models, rule lists, generalized additive models, and Explainable Boosting Machines, while blackbox explainability refers to tools such as Partial Dependence, LIME, and SHAP that explain existing opaque models (Nori et al., 2019). A plausible implication is that graybox frameworks may include interpretability, but their defining property is the explicit use of partially known internal structure, not simply the availability of explanations.

The limitations are similarly domain dependent but structurally recurrent. Overly rigid whitebox assumptions can lead to systematic error when the real device violates those assumptions, as in photonic quantum control where tridiagonality, reality, and linear voltage dependence were not adequate for the effective Hamiltonian (Youssry et al., 2022). Pure blackbox surrogates can match data but do not expose latent physical quantities such as $\mathbf{V}\mapsto H$6, $\mathbf{V}\mapsto H$7, or $\mathbf{V}\mapsto H$8, and therefore cannot support tasks such as gate-level fidelity optimization or Hamiltonian learning (Youssry et al., 2022, Mayevsky et al., 16 Jun 2025). In grey-box Bayesian optimization, function-network acquisitions can be noisier and harder to optimize because stochastic parent outputs enter the Monte Carlo estimator, and GP surrogates retain the usual scaling issues in the number of training points and the dimensionality of outputs (Niu et al., 2024, Galliani et al., 2016). In quantum settings, scaling to larger systems increases neural-network size, dataset requirements, and measurement overhead, while finite-shot estimation can dominate the minimum achievable expected MSE loss in experimental calibration (Youssry et al., 2022, Mayevsky et al., 16 Jun 2025, Pathumsoot et al., 18 Aug 2025).

The most stable formulation across these literatures is therefore not a specific architecture but a methodological rule: identify what is governed by trusted equations, graph structure, or constraints; encode that part explicitly; learn only the residual map; and keep the learned representation coupled to the mechanistic layer so that latent variables, uncertainty, or optimization-relevant internal states remain accessible (Youssry et al., 2022, Niu et al., 2024, Astudillo et al., 2022).