SPY Benchmark: Diverse Domain Evaluations
- SPY Benchmark is a context-dependent label defining experimental testbeds across gravitational-wave signal classification, optimal execution in finance, and synthetic excess-growth-rate generation.
- It employs tailored data protocols, domain-specific augmentations, and precise evaluation metrics to expose model shortcomings and validate methodological enhancements.
- Each implementation uses explicit operational criteria—such as recall in signal detection, exposure-cost trade-offs in execution, and statistical goodness-of-fit measures in time-series generation—to assess performance.
Searching arXiv for the cited papers and related uses of “SPY Benchmark” to ground the article in current sources. “SPY Benchmark” is a context-dependent label used in multiple arXiv works for materially different benchmark constructions rather than a single canonical resource. In the cited literature, it denotes at least three distinct artifacts: a Gravity Spy-derived image-classification benchmark for distinguishing compact-binary-coalescence signals from LIGO/Virgo glitches (Jarov et al., 2023), an SPY intraday optimal-execution backtest for Multi-Trajectory Physics-Informed Neural Networks solving HJB equations with a hard-zero terminal inventory constraint (Valin, 14 Dec 2025), and a benchmarking framework for synthetic SPY excess growth-rate generation based on a hybrid hidden Markov model with a Poisson-driven jump-duration mechanism (Alswaidan et al., 10 Mar 2026). The commonality is evaluative rather than methodological: each usage defines a domain-specific testbed, data protocol, and metric suite.
1. Terminological scope and main usages
In the cited works, the phrase “SPY Benchmark” does not name a single standardized dataset or leaderboard. Instead, it functions as a local designation for a benchmarked experimental framework within a paper’s own domain. This suggests that the term should always be interpreted with explicit reference to the surrounding methodology and data source.
| Usage | Core data | Primary task |
|---|---|---|
| Gravity Spy benchmark | Gravity Spy q-scan images plus simulated CBC injections | Signal-vs-glitch classification |
| SPY intraday execution benchmark | NASDAQ TotalView–ITCH SPY mid-price ticks | Optimal execution under HJB control |
| SPY synthetic-path benchmark | SPY excess growth-rate time series | Statistical fidelity of synthetic path generation |
The first usage is only indirectly tied to the string “SPY”: it is a benchmark built from Gravity Spy, not from the SPY ETF. The latter two usages concern financial time series for the SPY instrument itself. A plausible implication is that the acronym-like visual similarity between “Gravity Spy” and “SPY” can produce ambiguity unless the benchmark’s research area is stated explicitly.
2. Gravity Spy-derived signal-versus-glitch benchmark
In gravitational-wave data analysis, the benchmark summarized from "A new method to distinguish gravitational-wave signals from detector noise transients with Gravity Spy" repurposes Gravity Spy from glitch taxonomy to signal-versus-glitch discrimination (Jarov et al., 2023). The base dataset is the original Gravity Spy training set, consisting of 9,631 single-detector q-scan images divided into 22 glitch classes plus 60 examples of simulated GW “chirps.” Typical glitch classes include Blip, Low-Frequency Blip, Scattered Light, Koi Fish, Scratchy, Tomte, and Violin Mode.
To enrich the dataset with astrophysical signals, simulated compact-binary-coalescence waveforms are injected into real LIGO noise using PyCBC injections. The injections span total mass , , and spins . Each injection is converted into the same four-duration q-scan images used for real glitches. The resulting benchmark is then organized into three balanced binary or multi-class training sets of approximately 750 images each, with approximately 150 per class, by selecting only the glitch classes most often confused with CBC signals in each mass range and adding equal numbers of simulated GW examples.
The three benchmark variants are defined as follows. GSpySVG_LM covers low-mass CBCs from $3$ to and includes the classes Blip, LF Blip, Scratchy, No-Glitch, and “GW.” GSpySVG_HM covers $50$ to and includes Blip, LF Blip, Koi Fish, Tomte, and “GW.” GSpySVG_EHM addresses the most difficult regime, $250$ to , using Mercator-rescaled q-scans of LF Blip and Mercator-rescaled q-scans of “GW.”
The CNN architecture retains the Gravity Spy backbone: four time-frequency q-scans of durations 2 s, 1 s, 0.5 s, and 0.25 s are concatenated as channels and passed through convolutional layers with ReLU activations and pooling, followed by one or two fully connected layers and a softmax output. The main restructuring consists of four components. First, the task is divided by mass range rather than handled by a single 22-class glitch-plus-chirp model. Second, time-translation augmentation is introduced by duplicating each training image four times with a random time shift s. Third, the extremely high-mass regime uses a Mercator-type vertical stretch of the q-scan to enhance high-frequency detail. Fourth, the training objective uses binary cross-entropy for two-class CNNs or categorical cross-entropy for multi-class models, optimized with standard stochastic gradient descent or Adam.
Evaluation uses accuracy, precision, recall, and 0 score in the standard notation 1. On simulated-signal classification, GSpySVG_LM achieves 98% recall on 500 test CBCs, with 491 of 500 classified correctly and 9 misclassified as glitches. GSpySVG_HM achieves 96% recall on 500 test CBCs, with 481 of 500 correct and 19 misclassified. GSpySVG_EHM achieves 96% recall, with 48 of 50 correct after Mercator projection. On glitch classification, both GSpySVG_LM and GSpySVG_HM correctly label 99% of the 200 test glitch images in their respective sets, while GSpySVG_EHM labels 100% of test LF Blip glitches correctly.
The application to public O3b alerts makes the benchmark operational rather than merely synthetic. For each public O3b event, single-detector strain data are cut around the nominal merger time, four q-scans are produced, and the pipeline-reported mass estimate routes the event to GSpySVG_LM, GSpySVG_HM, or GSpySVG_EHM. The model’s highest-confidence label is then compared to the event’s eventual catalog status. Of the 21 publicly reported O3b events, 11 were later confirmed astrophysical and 10 were retracted. The combined GSpySVG_LM+HM system correctly flags 100% of the 11 confirmed events as “GW” or “No-Glitch” and 75% of the 10 retracted candidates as glitches. No human relabeling of q-scan images is required; the ground truth is taken from the LIGO–Virgo event catalog.
The benchmark’s significance lies in its reframing of Gravity Spy from pure glitch taxonomy to targeted candidate validation. The paper’s summary states that splitting the task by mass range, targeting only the small subset of glitch classes that mimic each signal morphology, adding time-shift augmentation, and applying a Mercator projection in the highest-mass regime together produce a 2 accurate, single-detector, image-based signal-vs-glitch classifier.
3. SPY intraday optimal-execution benchmark
In quantitative finance, the “SPY Benchmark” in "Multi-Trajectory Physics-Informed Neural Networks for HJB Equations with Hard-Zero Terminal Inventory: Optimal Execution on Synthetic & SPY Data" is a backtest for intraday liquidation on SPY mid-price data (Valin, 14 Dec 2025). The data source is NASDAQ TotalView–ITCH SPY mid-price ticks sampled from 09:45–15:45 ET, excluding the first and last 15 minutes, on seven trading days from February 10 to 19, 2025. The mid-price series is interpolated to a 5-second frequency, and each day is partitioned into three non-overlapping 2-hour windows: W1 from 09:45–11:45, W2 from 11:45–13:45, and W3 from 13:45–15:45.
The preprocessing and simulation assumptions are explicit. Time is normalized to trading-day units, with 6.5 h set equal to 1, so each 2-hour window has 3. Inventory is scaled to 4. Price is clipped to 5, described as approximately 6 around the observed range of \$\mathrm{SNR} \in [3,35]$7613.2. Realized volatility per window is computed from log-returns and annualized; averaged across windows, this gives $\mathrm{SNR} \in [3,35]$8, or approximately 6% annualized. The permanent-impact parameter is set to $\mathrm{SNR} \in [3,35]$9. The market-impact model used in simulation is the Gatheral–Schied model with unaffected price dynamics $\chi \in [-0.95,0.95]$0, transaction price $\chi \in [-0.95,0.95]$1, temporary impact scaled to one, permanent impact subsumed in the quadratic penalty $\chi \in [-0.95,0.95]$2, no fees, no order-book microstructure, and continuous-time control $\chi \in [-0.95,0.95]$3.
The control problem is posed through a value function $\chi \in [-0.95,0.95]$4 with $\chi \in [-0.95,0.95]$5 and the hard-zero terminal constraint $\chi \in [-0.95,0.95]$6, expressed by $\chi \in [-0.95,0.95]$7 if $\chi \in [-0.95,0.95]$8 and $\chi \in [-0.95,0.95]$9 otherwise. In the risk-neutral case $3$0, the state is $3$1 and the HJB equation is
$3$2
subject to $3$3 for all $3$4 and the stated terminal jump condition. In the risk-averse case $3$5, the state is $3$6 and the equation becomes
$3$7
with boundary $3$8 for all $3$9 and the same hard terminal condition. The optimal feedback control is recovered analytically as
$50\,M_\odot$0
Two neural formulations are benchmarked. The vanilla PINN is an MLP with tanh activations, using inputs $50\,M_\odot$1 or $50\,M_\odot$2 and typical widths of 500 on synthetic problems or 32 on SPY. Its losses are the HJB residual $50\,M_\odot$3, an internal-condition loss $50\,M_\odot$4, a symmetry loss $50\,M_\odot$5, and a soft terminal penalty $50\,M_\odot$6, combined as a weighted sum with DWA-style EMA balancing. The MT-PINN adds a rollout-based trajectory loss $50\,M_\odot$7 defined by unrolling Euler inventory dynamics over batches of initial states and horizons and penalizing terminal inventory by a piecewise function $50\,M_\odot$8 for $50\,M_\odot$9 and $50$0 otherwise. This loss is backpropagated through the unrolled dynamics by backpropagation-through-time. The MT-PINN also uses a $50$1-curriculum, starting from $50$2 and warm-starting successive stages up to a target $50$3.
For SPY, the implementation uses three hidden tanh layers of width 32, approximately 2k parameters, AdamW with learning rate $50$4, no dropout, and no batch normalization. Collocation uses approximately 20,000 PDE points, approximately 2,000 internal-condition points, and approximately 200 terminal-condition points. The multi-trajectory batch uses $50$5 and $50$6, giving approximately $50$7 states, with $50$8 horizons on the grid $50$9 and $300\,M_\odot$0 Euler steps. The SPY curriculum considers $300\,M_\odot$1 with stages at $300\,M_\odot$2, with Phase A at $300\,M_\odot$3 for 20k epochs and Phase B at 5k epochs per stage. Reported hardware is a single TPU v6e, with a typical run time of approximately 2–4 minutes per experiment.
Results are reported over 21 two-hour windows using exposure, defined as mean$300\,M_\odot$4, and implementation-shortfall cost in basis points. TWAP yields exposure $300\,M_\odot$5 and cost $300\,M_\odot$6 bps with standard deviations $300\,M_\odot$7 and $300\,M_\odot$8. MT-PINN at $300\,M_\odot$9 gives $250$0 and $250$1 bps. MT-PINN at $250$2 gives $250$3 and $250$4 bps, and at $250$5 gives $250$6 and $250$7 bps. The paper characterizes this as a risk–cost frontier in which raising $250$8 front-loads trades, reducing exposure while increasing cost. In strictly rising SPY windows, $250$9 MT-PINN and TWAP tie for lowest cost; in falling windows, risk-averse MT-PINN with $350\,M_\odot$0 outperforms TWAP. Trajectory plots show smooth controls and tight enforcement of $350\,M_\odot$1, with mean final $350\,M_\odot$2 at tolerance $350\,M_\odot$3, and the hard-zero enforcement is summarized as achieving $350\,M_\odot$4 in more than 95% of cases.
Within this benchmark, the main methodological point is not merely that PINNs solve an HJB, but that the rollout-based trajectory loss makes the terminal inventory constraint operational along simulated execution paths. The paper states that at $350\,M_\odot$5 the method exactly reproduces TWAP in expectation, whereas for $350\,M_\odot$6 it yields a clear exposure–cost trade-off that is advantageous in downtrends.
4. SPY excess-growth-rate synthetic-generation benchmark
In time-series generation and risk modeling, the SPY benchmark in "Hybrid Hidden Markov Model for Modeling Equity Excess Growth Rate Dynamics: A Discrete-State Approach with Jump-Diffusion" is a validation framework for synthetic SPY excess growth-rate paths (Alswaidan et al., 10 Mar 2026). The benchmark is designed to reproduce three empirical features simultaneously: a heavy-tailed marginal distribution, negligible linear autocorrelation, and persistent volatility clustering.
The model begins by discretizing the continuous excess growth-rate series $350\,M_\odot$7 through a Laplace-quantile state partition. A two-parameter Laplace distribution $350\,M_\odot$8 is fitted by maximum likelihood, with $350\,M_\odot$9 equal to the sample median and $\delta t \in [-0.1,0.1]$0 equal to the mean absolute deviation from that median. Hidden states are then defined by equal-probability quantile thresholds
$\delta t \in [-0.1,0.1]$1
with $\delta t \in [-0.1,0.1]$2 and $\delta t \in [-0.1,0.1]$3, and observations are assigned to states according to the interval containing $\delta t \in [-0.1,0.1]$4.
Given the discrete state sequence, the model estimates the transition matrix $\delta t \in [-0.1,0.1]$5 by direct transition counting rather than Baum–Welch EM. The stationary distribution satisfies $\delta t \in [-0.1,0.1]$6. To enforce realistic dwell times in extreme states, the model adds a Poisson-driven jump-duration mechanism. At each step, a jump trigger occurs with probability $\delta t \in [-0.1,0.1]$7; if triggered, a duration $\delta t \in [-0.1,0.1]$8 is sampled, and for the next $\delta t \in [-0.1,0.1]$9 steps the state is forced into either the negative-tail set $\mathrm{SNR} \in [3,35]$00 with probability $\mathrm{SNR} \in [3,35]$01 or the positive-tail set $\mathrm{SNR} \in [3,35]$02 otherwise. Outside jump episodes, the chain evolves according to $\mathrm{SNR} \in [3,35]$03. Conditional on state $\mathrm{SNR} \in [3,35]$04, emissions are drawn from a location-scale Student-$\mathrm{SNR} \in [3,35]$05 distribution with $\mathrm{SNR} \in [3,35]$06:
$\mathrm{SNR} \in [3,35]$07
where $\mathrm{SNR} \in [3,35]$08 and $\mathrm{SNR} \in [3,35]$09 are the sample mean and standard deviation of training observations assigned to state $\mathrm{SNR} \in [3,35]$10.
Synthetic-path generation proceeds by drawing an initial state from $\mathrm{SNR} \in [3,35]$11, advancing the hidden state either through the jump mechanism or the Markov transition matrix, and then sampling emissions from the state-conditional Student-$\mathrm{SNR} \in [3,35]$12 law. Hyperparameters are calibrated by grid search over $\mathrm{SNR} \in [3,35]$13 and $\mathrm{SNR} \in [3,35]$14 to minimize an objective
$\mathrm{SNR} \in [3,35]$15
with $\mathrm{SNR} \in [3,35]$16, $\mathrm{SNR} \in [3,35]$17, and averages over 200 paths per grid point. For SPY, the optimum is reported as $\mathrm{SNR} \in [3,35]$18 and $\mathrm{SNR} \in [3,35]$19.
Validation uses two-sample Kolmogorov–Smirnov and Anderson–Darling pass rates for distributional fidelity and an ACF mean absolute error on absolute returns for temporal fidelity. Pass rates are computed as the fraction of 1,000 simulated paths that do not reject the null at $\mathrm{SNR} \in [3,35]$20. On SPY, the benchmark compares the hybrid HMM with jumps (HMM-WJ), a standard HMM without jumps (HMM-NJ), and GARCH(1,1), using in-sample data from 2014–2024 with $\mathrm{SNR} \in [3,35]$21 and out-of-sample data from calendar year 2025 with $\mathrm{SNR} \in [3,35]$22.
The reported performance profile is explicitly non-dominating across all criteria. HMM-WJ achieves KS pass rates of 97.6% in-sample and 94.4% out-of-sample, AD pass rates of 91.3% and 95.1%, ACF-MAE values of 0.052 and 0.039, and simulated excess kurtosis of 7.6 and 6.1. HMM-NJ attains higher distributional pass rates—99.7% and 96.7% for KS, 99.1% and 96.7% for AD—but weaker temporal fidelity, with ACF-MAE 0.059 and 0.041, and the paper states that it cannot generate persistent high-volatility regimes. GARCH(1,1) attains the best temporal match, with ACF-MAE 0.031 in-sample and 0.026 out-of-sample, but performs poorly on marginal tests, with an in-sample KS pass rate of 5.5% and AD pass rate of 1.9%, and underestimates tails out-of-sample. The paper summarizes HMM-WJ as occupying the Pareto frontier by improving ACF-MAE over HMM-NJ at a modest cost to distributional fit.
The framework also includes a Single-Index Model extension to a 424-asset universe:
$\mathrm{SNR} \in [3,35]$23
Parameters are estimated by OLS on in-sample residuals, and synthetic asset returns are generated by propagating a synthetic SPY factor path through this equation. Across 424 assets, the in-sample mean $\mathrm{SNR} \in [3,35]$24 is 0.298 and the median KS pass rate is 66.7%; out-of-sample mean KS is 82.1% and median KS is 91.8%, with higher pass rates for high-$\mathrm{SNR} \in [3,35]$25 assets.
5. Benchmark design patterns across the three usages
Despite their disciplinary separation, the three SPY benchmark usages share a structural logic. Each defines a task-specific experimental environment, constrains the state or label space to the subset most relevant for the target phenomenon, and evaluates performance against explicit operational criteria.
In the Gravity Spy setting, this takes the form of a divide-and-conquer decomposition by CBC mass range, targeted inclusion of the glitch classes most often confused with signals, and evaluation through recall, glitch rejection, and real-alert outcomes (Jarov et al., 2023). In the execution setting, it appears as a reduced HJB state in the risk-neutral case, expansion to $\mathrm{SNR} \in [3,35]$26 in the risk-averse case, and evaluation through exposure, implementation shortfall, and terminal-inventory enforcement (Valin, 14 Dec 2025). In the synthetic-generation setting, it appears as a quantile-defined discrete regime representation, direct frequentist estimation of transitions, and validation through KS, AD, ACF-MAE, and excess kurtosis (Alswaidan et al., 10 Mar 2026).
A further commonality is the use of auxiliary mechanisms to address failure modes that a baseline model does not handle adequately. The gravitational-wave benchmark adds time-translation augmentation and a Mercator projection to stabilize hard classification regimes. The execution benchmark adds rollout loss and backpropagation-through-time to enforce the hard-zero terminal inventory constraint beyond what a vanilla PINN’s soft terminal penalty achieves. The synthetic-generation benchmark adds a Poisson jump-duration mechanism because a standard HMM without jumps fails to generate persistent tail-state dwell times. This suggests a recurring benchmark-design principle: the benchmark is not merely a dataset, but a protocol for exposing model deficiencies and testing targeted architectural or probabilistic remedies.
6. Ambiguities, limitations, and interpretive cautions
A central misconception is that “SPY Benchmark” designates a unique, field-wide object. In the cited literature, it does not. In one usage, it refers to a Gravity Spy image benchmark for gravitational-wave morphology discrimination, with no connection to the SPY ETF beyond the local shorthand assigned in the summary (Jarov et al., 2023). In two others, it refers to financial benchmarks built on SPY market data (Valin, 14 Dec 2025, Alswaidan et al., 10 Mar 2026). Disambiguation by domain is therefore necessary.
Each benchmark also has explicit scope conditions. The Gravity Spy-derived benchmark is single-detector and image-based, and its O3b evaluation compares classifier outputs to eventual catalog outcomes rather than to human relabeling. The MT-PINN execution benchmark excludes the first and last 15 minutes of the trading day, assumes no fees and no order-book microstructure, and uses continuous-time control within the Gatheral–Schied model. The HMM-WJ synthetic benchmark is calibrated to SPY excess growth rates and, by the paper’s own comparison, does not dominate GARCH(1,1) on temporal fidelity or HMM-NJ on distributional pass rates.
Another misconception is that high performance under one benchmark necessarily transfers to another evaluative axis. The cited results argue against that simplification. In the execution benchmark, risk aversion lowers exposure but raises cost. In the synthetic-generation benchmark, GARCH(1,1) attains the best ACF-MAE while failing marginal distribution tests, whereas HMM-NJ attains the highest KS and AD pass rates while missing persistent volatility clustering. In the Gravity Spy-derived benchmark, the hardest case requires a distinct projection and a dedicated binary classifier. A plausible implication is that, across these usages, “benchmark” should be understood as a controlled trade-off surface rather than a single scalar ranking device.