Graybox Approach in Optimization

• Graybox approach is a method that integrates partial internal information with black-box techniques to preserve known structural invariants.
• It applies across domains such as Bayesian optimization, quantum control, and fuzzing, balancing analytical models with data-driven learning.
• By exploiting available internal clues, it improves data efficiency, interpretability, and targeted exploration over traditional methods.

Graybox approach denotes a family of methodologies positioned between black-box and white-box analysis. In black-box settings, only outputs, fitness values, or externally visible behavior are used; in white-box settings, full internal structure, equations, or state machines are assumed available. Graybox methods exploit partial internal information without requiring full disclosure or full analytic tractability. The literature uses both spellings, “graybox” and “grey-box.” Across Bayesian optimization, quantum characterization and control, fuzzing, software testing, network management, and cyber-physical modeling, the common pattern is to expose internal constituents, known dynamics, dependency structure, coverage signals, or limited device capabilities, while leaving the unknown portion to learning, search, or lightweight instrumentation (Astudillo et al., 2022, Gasse et al., 6 Jun 2026, Youssry et al., 2022).

1. Conceptual position and recurring design pattern

Several works define the graybox position explicitly as an intermediate point on a spectrum. In Bayesian optimization, grey-box methods treat objective computation as “partially observable and even modifiable,” using internal computational structure while preserving the sequential uncertainty-aware logic of BO (Astudillo et al., 2022). In combinatorial optimization, gray-box optimization makes “some problem-specific information available to the algorithm while still relying on fitness information as the main guide to an optimum,” thereby capturing a continuum between fully black-box algorithms and tailored algorithms (Gasse et al., 6 Jun 2026). In quantum characterization, graybox modeling combines a whitebox part, consisting of known system dynamics, with a blackbox part, consisting of unknown transformations learned from data (Pathumsoot et al., 29 Sep 2025).

This middle position is not merely terminological. In some domains, the exposed structure is a known loss function, fidelity parameter, or constituent simulator output; in others, it is bytecode-level field access, coverage traces, tool invocation sequences, or middlebox APIs. TopoMan gives a particularly strict formulation: vendors are not expected to expose the “Full state machine (FSM),” but a limited capability interface is sufficient for discovery and verification (Nagendra et al., 2016). The same logic appears in model-based settings, where the system evolution is retained from first principles and only the “missing physics” is learned (Cantone et al., 18 Jul 2025).

Family	Exposed structure	Graybox consequence
BO and sequential optimization	composite functions, constituent evaluations, fidelities, admissible parameter sets	structure-aware acquisition and lower-confidence optimization
Quantum modeling	ideal evolution, measurement layers, known Hamiltonian structure	learning of effective noise or observable transformations
Testing and fuzzing	event dependencies, coverage, tool traces, database fetches	guided mutation, targeted seeds, behavior-aware exploration
Networked and cyber-physical systems	probe APIs, circuit structure, aerodynamic and motor models	topology discovery, path verification, voltage and range prediction

A recurring consequence is that graybox methods preserve structural invariants that black-box surrogates can violate. One gray-box optimization paper notes that if the true objective is nonnegative because of loss/model composition, a structure-agnostic black-box surrogate may still predict negative values, whereas the gray-box formulation respects the known structure (Baumgärtner et al., 16 Jun 2026).

2. Hybrid modeling of partially known dynamics

In quantum system identification and control, the graybox approach is formulated as a hybrid physics-plus-learning model. For a closed $N$-dimensional system, one implementation learns the control-to-Hamiltonian map with a neural network, then applies fixed quantum-mechanical layers enforcing Hermiticity, unitary evolution, and Born-rule measurement: $$H = \frac{A + A^\dagger}{2}, \qquad U = e^{-iHT}, \qquad P_{j\to k} = |\langle k|U|j\rangle|^2.$$ Because only probabilities are used for training, yet the model outputs $H(\mathbf V)$ and $U$, the approach can recover physically meaningful quantities that were not directly measured (Youssry et al., 2022).

A closely related characterization formalism writes the predicted expectation value as

$$\mathbb{E}[\hat{O}]_{\rho_0} = \mathrm{Tr}\!\left[ \hat{W}_O(\mathbf{x})\, \hat{U}_0(\mathbf{x})\rho_0 \hat{U}_0^\dagger(\mathbf{x}) \right],$$

where $\hat{U}_0(\mathbf{x})$ is known ideal evolution and $\hat{W}_O(\mathbf{x})$ is an unknown observable transformation learned by a model. In the deterministic statistical graybox model, the blackbox is a DNN/MLP; in the probabilistic extension, it is replaced by a Bayesian Neural Network trained with Stochastic Variational Inference and Trace Mean Field ELBO, which provides a posterior predictive distribution rather than only point predictions (Pathumsoot et al., 29 Sep 2025).

The same whitebox-plus-blackbox decomposition appears in experimental sensing. In a single nitrogen-vacancy center Ramsey sensor, the graybox model retains the ideal Ramsey unitary

$$U_{\text{Ramsey}}(\tau,\phi,f_B)=R_X(\pi/2)\,R_Z(\theta)\,R_X(\pi/2), \qquad \theta = 2\pi f_B\tau + \phi,$$

while a neural network learns an effective noise operator $\hat V_Z(T)$. Using roughly $10{,}000$ training datapoints, the reported result is a several-orders-of-magnitude reduction in mean squared error relative to the corresponding physics-only model (Youssry et al., 24 Jan 2026). On superconducting-qubit experiments, the same philosophy is combined with finite-shot analysis. The total Hamiltonian is decomposed as

$$H = \frac{A + A^\dagger}{2}, \qquad U = e^{-iHT}, \qquad P_{j\to k} = |\langle k|U|j\rangle|^2.$$0

and the learned object is an effective observable map $$H = \frac{A + A^\dagger}{2}, \qquad U = e^{-iHT}, \qquad P_{j\to k} = |\langle k|U|j\rangle|^2.$$1. For the single-qubit Pauli-tomography setup studied there, the expected MSE floor is bounded below by

$$H = \frac{A + A^\dagger}{2}, \qquad U = e^{-iHT}, \qquad P_{j\to k} = |\langle k|U|j\rangle|^2.$$2

showing that finite-shot estimation dominates the minimum achievable expected MSE (Pathumsoot et al., 18 Aug 2025).

Graybox control of open quantum systems extends the same template to non-Markovian dynamics. For a noisy qubit driven by Gaussian pulses, the whitebox part computes controlled evolution and process fidelities, while a lightweight Transformer learns an effective operator $$H = \frac{A + A^\dagger}{2}, \qquad U = e^{-iHT}, \qquad P_{j\to k} = |\langle k|U|j\rangle|^2.$$3. The model benchmarks both random telegraph noise and Ornstein-Uhlenbeck noise, achieves low prediction errors across coupling regimes, and is then used as a differentiable emulator for gradient-based gate optimization; fidelities are reported above $$H = \frac{A + A^\dagger}{2}, \qquad U = e^{-iHT}, \qquad P_{j\to k} = |\langle k|U|j\rangle|^2.$$4 for the lowest considered coupling and above $$H = \frac{A + A^\dagger}{2}, \qquad U = e^{-iHT}, \qquad P_{j\to k} = |\langle k|U|j\rangle|^2.$$5 for the highest (Cantone et al., 18 Jul 2025).

Taken together, these results show that graybox modeling is especially suited to settings in which the measurable outputs are limited but the governing transformations are partially known. In such cases, the approach can remain interpretable, support uncertainty quantification, and enable open-loop optimization over latent physical quantities (Youssry et al., 2022, Pathumsoot et al., 29 Sep 2025, Pathumsoot et al., 18 Aug 2025).

3. Structure-exploiting optimization and learning

A major theoretical line of work treats graybox methodology as structured optimization over partially exposed objective computation. The tutorial literature centers on composite objectives

$$H = \frac{A + A^\dagger}{2}, \qquad U = e^{-iHT}, \qquad P_{j\to k} = |\langle k|U|j\rangle|^2.$$6

where $$H = \frac{A + A^\dagger}{2}, \qquad U = e^{-iHT}, \qquad P_{j\to k} = |\langle k|U|j\rangle|^2.$$7 is expensive and vector-valued while $$H = \frac{A + A^\dagger}{2}, \qquad U = e^{-iHT}, \qquad P_{j\to k} = |\langle k|U|j\rangle|^2.$$8 is known and cheap. In this setting, graybox BO models $$H = \frac{A + A^\dagger}{2}, \qquad U = e^{-iHT}, \qquad P_{j\to k} = |\langle k|U|j\rangle|^2.$$9 directly, rather than the scalar $H(\mathbf V)$0, and can also exploit constituent evaluations and multi-fidelity controls. For multi-fidelity BO, the target is $H(\mathbf V)$1, with a fidelity parameter $H(\mathbf V)$2 and evaluation cost $H(\mathbf V)$3; the key decision is therefore not only where to evaluate, but also at which fidelity (Astudillo et al., 2022).

Epidemiological model calibration provides an explicit graybox BO instantiation. The calibration objective is written as a composition of a known loss $H(\mathbf V)$4 and expensive simulator outputs $H(\mathbf V)$5, with Gaussian-process surrogates placed on the compartment trajectories rather than directly on the scalar loss. The method further introduces a function network $H(\mathbf V)$6 to encode epidemiological dependencies and a decoupled decision-making scheme to exploit incomplete observations. On synthetic SIQR data and real COVID-19 data, the graybox variants improve calibration performance measured by the logarithm of mean square errors and reach faster convergence in terms of BO iterations (Niu et al., 2024).

Sequential gray-box optimization generalizes linear and contextual bandits by assuming a known loss $H(\mathbf V)$7, a known parametric model $H(\mathbf V)$8, and an unknown parameter $H(\mathbf V)$9. The true objective is

$U$0

The proposed method uses optimism in the face of uncertainty by minimizing a lower confidence bound: $U$1 The resulting regret bounds are of $U$2-type in general and $U$3-type for finite action sets (Baumgärtner et al., 16 Jun 2026).

In combinatorial optimization, gray-box operators are used to break plateaus that hinder unbiased search. For vertex coloring, the literature distinguishes black-box optimization, problem-aware gray-box optimization, and instance-aware gray-box optimization. Starting from a random $U$4-coloring, RLS can find a proper $U$5-coloring of a bipartite graph in expected time $U$6. By contrast, when starting from a proper $U$7-coloring, the $U$8 EA cannot find such a coloring without additional plateau guidance. Gray-box mutation operators that prefer less-used colors and exploit neighborhood feasibility substantially improve runtime; on complete bipartite graphs, gray-box RLS obtains $U$9 expected time (Gasse et al., 6 Jun 2026).

Graybox learning of timed languages follows the same structural principle in active automata inference. Instead of learning the full region automaton, the learner is given the semantics of consistent region words and infers only the regular structure over these known symbolic objects. The key identity is

$$\mathbb{E}[\hat{O}]_{\rho_0} = \mathrm{Tr}\!\left[ \hat{W}_O(\mathbf{x})\, \hat{U}_0(\mathbf{x})\rho_0 \hat{U}_0^\dagger(\mathbf{x}) \right],$$0

which allows the method to avoid learning already known clock semantics. In the idealized version, the learned DERA can have the minimum number of states; in the practical implementation, completeness is checked instead of strong-completeness, and a small consistent DFA is constructed heuristically (Majumdar et al., 2024).

4. Testing, fuzzing, and vulnerability discovery

Software testing and fuzzing have produced some of the clearest operational definitions of grey-box behavior. In GUI testing, the classic black-box model is the event-flow graph (EFG), which captures executable event sequences. The grey-box extension introduces an event-dependency graph (EDG) derived from bytecode analysis of event handlers. Weighted EDG edges encode shared read-write dependencies,

$$\mathbb{E}[\hat{O}]_{\rho_0} = \mathrm{Tr}\!\left[ \hat{W}_O(\mathbf{x})\, \hat{U}_0(\mathbf{x})\rho_0 \hat{U}_0^\dagger(\mathbf{x}) \right],$$1

and abstract dependency paths are then mapped back to executable sequences through the EFG. On four open-source applications, the EDG-based method reduced the number of test cases while still achieving at least the same coverage, and it detected two new bugs (Arlt et al., 2012).

Greybox fuzzing classically relies on lightweight instrumentation and random mutation. One major extension augments this with input learning from execution-derived costs. The fuzzer instruments branch-flipping costs and learns new inputs by fitting a line through two observed $$\mathbb{E}[\hat{O}]_{\rho_0} = \mathrm{Tr}\!\left[ \hat{W}_O(\mathbf{x})\, \hat{U}_0(\mathbf{x})\rho_0 \hat{U}_0^\dagger(\mathbf{x}) \right],$$2 pairs, then solving for cost $$\mathbb{E}[\hat{O}]_{\rho_0} = \mathrm{Tr}\!\left[ \hat{W}_O(\mathbf{x})\, \hat{U}_0(\mathbf{x})\rho_0 \hat{U}_0^\dagger(\mathbf{x}) \right],$$3. This retains the lightweight nature of greybox fuzzing while adding a semantic feedback loop. On $$\mathbb{E}[\hat{O}]_{\rho_0} = \mathrm{Tr}\!\left[ \hat{W}_O(\mathbf{x})\, \hat{U}_0(\mathbf{x})\rho_0 \hat{U}_0^\dagger(\mathbf{x}) \right],$$4 real-world Solidity smart-contract benchmarks, the reported outcome is up to $$\mathbb{E}[\hat{O}]_{\rho_0} = \mathrm{Tr}\!\left[ \hat{W}_O(\mathbf{x})\, \hat{U}_0(\mathbf{x})\rho_0 \hat{U}_0^\dagger(\mathbf{x}) \right],$$5X path coverage, up to $$\mathbb{E}[\hat{O}]_{\rho_0} = \mathrm{Tr}\!\left[ \hat{W}_O(\mathbf{x})\, \hat{U}_0(\mathbf{x})\rho_0 \hat{U}_0^\dagger(\mathbf{x}) \right],$$6 more bugs, and often orders-of-magnitude faster discovery (Wüstholz et al., 2018).

Another line of work recasts AFL’s energy scheduling as a contextual bandit. The state is a $$\mathbb{E}[\hat{O}]_{\rho_0} = \mathrm{Tr}\!\left[ \hat{W}_O(\mathbf{x})\, \hat{U}_0(\mathbf{x})\rho_0 \hat{U}_0^\dagger(\mathbf{x}) \right],$$7-byte substring of the current test case, the action is a multiplier from $$\mathbb{E}[\hat{O}]_{\rho_0} = \mathrm{Tr}\!\left[ \hat{W}_O(\mathbf{x})\, \hat{U}_0(\mathbf{x})\rho_0 \hat{U}_0^\dagger(\mathbf{x}) \right],$$8, and the reward is the ratio of interesting test cases to total generated test cases. The learned policy uses a one-layer LSTM with $$\mathbb{E}[\hat{O}]_{\rho_0} = \mathrm{Tr}\!\left[ \hat{W}_O(\mathbf{x})\, \hat{U}_0(\mathbf{x})\rho_0 \hat{U}_0^\dagger(\mathbf{x}) \right],$$9 recurrent units, $\hat{U}_0(\mathbf{x})$0-greedy exploration with $\hat{U}_0(\mathbf{x})$1, and a policy-gradient loss

$\hat{U}_0(\mathbf{x})$2

This replaces AFL’s hand-designed heuristic for random-fuzzing energy allocation, although the reported implementation remains worse than AFL without the deterministic phase ($\hat{U}_0(\mathbf{x})$3) (Patil et al., 2018).

Directed hardware fuzzing pushes the same idea toward native structural guidance. PROFUZZ constructs a hypergraph of the hardware design, uses ATPG to generate target-specific activation patterns, merges non-conflicting patterns to maximize don’t-care values, and then runs a target-coverage-guided fuzzing loop. The framework operates directly on RTL or gate-level netlists, supports target submodules for cross-module verification, and is reported to scale $\hat{U}_0(\mathbf{x})$4 better than DirectFuzz, improve coverage by $\hat{U}_0(\mathbf{x})$5, and run $\hat{U}_0(\mathbf{x})$6 faster (Saravanan et al., 25 Sep 2025).

Graybox vulnerability discovery can also be centered on system boundaries rather than source instrumentation. For stored and context-sensitive XSS, a scanner cooperates with the database, intercepts string fetches at the database–application boundary, reinjects payloads there, and identifies transformed payloads in the response by regex and browser-context analysis. Evaluated on eight mature web applications, the method found $\hat{U}_0(\mathbf{x})$7 incorrect sanitization instances, of which $\hat{U}_0(\mathbf{x})$8 were confirmed exploitable through the normal user interface and $\hat{U}_0(\mathbf{x})$9 more were exploitable only with direct database write access; the reported false-positive count on correct sanitization was zero (Steinhauser et al., 2020).

Agentic systems introduce a newer grey-box testing target. VeriGrey instruments the sequence of tool invocations as a coverage-like feedback signal, then mutates prompts through context bridging so that the injection task becomes a necessary step of the legitimate task. On AgentDojo with a GPT-4.1 back-end, the method achieves $\hat{W}_O(\mathbf{x})$0 additional efficacy in finding indirect prompt injection vulnerabilities over a black-box baseline. In OpenClaw, mutated malicious skills achieve $\hat{W}_O(\mathbf{x})$1 success on Kimi-K2.5 and $\hat{W}_O(\mathbf{x})$2 success on Opus 4.6 (Zhang et al., 18 Mar 2026).

Adversarial machine learning has used the same label in robustness evaluation and training. “Gray-box Adversarial Training” introduces gray-box adversarial attacks and a training variant that uses intermediate versions of the models to seed the adversaries, motivated by shortcomings of existing evaluation policy and degenerate minima in fast adversarial training (S. et al., 2018).

5. Networked and cyber-physical embodiments

Graybox approaches are not confined to learning or testing; they also appear in systems engineering, where limited exposure of internals is often the only deployable compromise. TopoMan addresses topology discovery and path verification in networks containing heterogeneous middleboxes. Rather than requiring full whitebox disclosure, it assumes that middleboxes expose only the minimum necessary capabilities: device metadata, probe initiation and forwarding hooks, and secure reporting of traversal information. The framework combines a Topology Manager with MB Agents, probe-based discovery, and APIs such as DEVICE-CAPABILITIES, PROBE-INIT, PROBE-UPDATE, UPDATE-OUTINTERFACE, RESOLVE-PROBEID, and HEARTBEAT. In a Cisco small-scale enterprise network with $\hat{W}_O(\mathbf{x})$3 nodes, topology discovery time is reported under $\hat{W}_O(\mathbf{x})$4 seconds, or about $\hat{W}_O(\mathbf{x})$5 seconds with NaCl-based security enabled; edge-device heuristics reduce probes by about $\hat{W}_O(\mathbf{x})$6 to $\hat{W}_O(\mathbf{x})$7 (Nagendra et al., 2016).

Cyber-physical performance estimation offers a different form of graybox hybridization. For multicopter range, endurance, and optimal speed estimation, the overall model combines a blade-element-momentum aerodynamic model, an electric-motor model, and a graybox battery model. The battery component uses a one-time-constant Thevenin equivalent with open-circuit source voltage $\hat{W}_O(\mathbf{x})$8, ohmic resistance $\hat{W}_O(\mathbf{x})$9, and polarization branch $$U_{\text{Ramsey}}(\tau,\phi,f_B)=R_X(\pi/2)\,R_Z(\theta)\,R_X(\pi/2), \qquad \theta = 2\pi f_B\tau + \phi,$$0: $$U_{\text{Ramsey}}(\tau,\phi,f_B)=R_X(\pi/2)\,R_Z(\theta)\,R_X(\pi/2), \qquad \theta = 2\pi f_B\tau + \phi,$$1 Because multicopters operate under non-constant discharge rates, the model predicts terminal voltage from power demand rather than assuming fixed-rate discharge. Across real-world experiments, the reported battery-voltage accuracy is $$U_{\text{Ramsey}}(\tau,\phi,f_B)=R_X(\pi/2)\,R_Z(\theta)\,R_X(\pi/2), \qquad \theta = 2\pi f_B\tau + \phi,$$2 mV RMSE per cell, or $$U_{\text{Ramsey}}(\tau,\phi,f_B)=R_X(\pi/2)\,R_Z(\theta)\,R_X(\pi/2), \qquad \theta = 2\pi f_B\tau + \phi,$$3 relative error; flight validation gives about $$U_{\text{Ramsey}}(\tau,\phi,f_B)=R_X(\pi/2)\,R_Z(\theta)\,R_X(\pi/2), \qquad \theta = 2\pi f_B\tau + \phi,$$4 mV RMSE, and the simplified pen-and-paper estimator predicts commercial drone specifications typically within about $$U_{\text{Ramsey}}(\tau,\phi,f_B)=R_X(\pi/2)\,R_Z(\theta)\,R_X(\pi/2), \qquad \theta = 2\pi f_B\tau + \phi,$$5 (Bauersfeld et al., 2021).

These examples show that graybox methodology can be implemented either through limited control interfaces, as in network discovery, or through hybrid physics-plus-empiricism, as in battery and propulsion modeling. This suggests that graybox is better understood as a systems-design principle than as a single algorithmic template (Nagendra et al., 2016, Bauersfeld et al., 2021).

6. Advantages, misconceptions, and limits

Across the cited literature, several advantages recur. Graybox methods often improve data efficiency because they do not attempt to relearn known structure. In BO, the use of constituents or fidelities can yield substantially better decisions than standard black-box BO (Astudillo et al., 2022). In quantum control, graybox models are more realistic than restrictive white-box fits and more interpretable than black-box predictors because they retain access to Hamiltonians, unitaries, or effective operators (Youssry et al., 2022). In sequential optimization, known loss and parameter structure produce tighter confidence reasoning and better regret guarantees than structure-agnostic approaches (Baumgärtner et al., 16 Jun 2026). In security testing, grey-box feedback exposes rare behaviors that purely black-box mutation fails to reach (Steinhauser et al., 2020, Zhang et al., 18 Mar 2026).

A common misconception is to equate graybox with partial code access alone. The surveyed work is broader. In some cases, the exposed information is not code but intermediate simulator outputs, admissible parameter sets, tool traces, database fetches, or device capability APIs. Another misconception is that graybox is a weak approximation to whitebox. Several papers instead treat it as the appropriate abstraction when full internal disclosure is impossible, undesirable, or analytically unhelpful; TopoMan rejects full FSM disclosure, and composite-objective BO explicitly assumes that only part of the evaluation process is usefully observable (Nagendra et al., 2016, Astudillo et al., 2022).

The limitations are equally consistent. Structure-aware acquisition can be harder to optimize than black-box acquisition, and decoupled choices can become combinatorial (Niu et al., 2024). Exact minimization or strong-completeness guarantees in timed-language learning are computationally costly, with practical implementations relying on heuristics (Majumdar et al., 2024). Some grey-box testing models remain incomplete: the XSS scanner treats JavaScript strings as leaf nodes, which can cause false negatives, and it depends on database cooperation and disabled caching for soundness (Steinhauser et al., 2020). In experimental quantum calibration, AGF-trained models can overfit, and device drift can break the predicted uncertainty envelope (Pathumsoot et al., 18 Aug 2025). In fuzzing, learning-based scheduling may still fail to beat stronger baselines, as with the contextual-bandit AFL modification relative to AFL $$U_{\text{Ramsey}}(\tau,\phi,f_B)=R_X(\pi/2)\,R_Z(\theta)\,R_X(\pi/2), \qquad \theta = 2\pi f_B\tau + \phi,$$6 (Patil et al., 2018).

The broad significance of the graybox approach is therefore not universal superiority over black-box or white-box methods, but a disciplined exploitation of partial structure. Where the unknown component is large enough to defeat purely analytic modeling, yet the known component is too valuable to ignore, graybox methods provide a principled compromise. In the surveyed literature, that compromise repeatedly yields physically meaningful latent variables, more targeted exploration, and better alignment between the optimization or testing procedure and the real structure of the system under study (Gasse et al., 6 Jun 2026, Youssry et al., 2022).