RegimeNAS: Regime-Aware NAS for Crypto Trading

• RegimeNAS is a regime-aware neural architecture search framework designed to adapt model architectures and loss functions based on market regime detection.
• It integrates Bayesian optimization with multi-head attention to dynamically activate specialized blocks for trending, ranging, and high-volatility periods.
• Empirical results demonstrate up to an 80% reduction in MAE over GRU benchmarks, with faster convergence and robust performance in non-stationary crypto markets.

Searching arXiv for the cited paper and closely related NAS work to ground the article. arxiv_search(query="(Devadiga et al., 15 Aug 2025)", max_results=5, sort_by="submittedDate") RegimeNAS is a regime-aware neural architecture search framework built for cryptocurrency trading and other highly non-stationary financial time series. Its central claim is that static deep models are fundamentally mismatched to markets that continuously switch between distinct behaviors—trending, ranging, and high-volatility phases—so the architecture itself should adapt to the detected market regime rather than remain fixed. In this formulation, RegimeNAS is not merely NAS applied to finance; it is NAS conditioned on market state, with the search loop, model blocks, and loss function all designed around regime changes (Devadiga et al., 15 Aug 2025).

1. Problem formulation and domain motivation

RegimeNAS targets the brittleness of conventional sequence models in crypto markets. The motivating observation is that a GRU or Transformer may do well in one period but fail badly when volatility spikes or the market becomes range-bound. The paper attributes this to the fact that crypto returns are heavy-tailed, highly volatile, and non-stationary, and that the predictive structure changes sharply over time. A model that is optimal in a trend regime may be wrong in a ranging regime, and one tuned on calm periods can become unstable during stress (Devadiga et al., 15 Aug 2025).

The framework therefore treats market regime awareness as an architectural requirement rather than a downstream diagnostic. Its overall workflow is: engineer multi-timeframe OHLCV and technical features; detect the current regime from attention over those features; use regime probabilities to gate specialized blocks; train candidate architectures under a market-aware loss; and use Bayesian optimization to search for the best architecture. The stated objective is to learn architectures that can switch their internal emphasis depending on the inferred market regime, while also being stable enough to deploy in real trading (Devadiga et al., 15 Aug 2025).

This design places RegimeNAS within a class of domain-conditioned NAS methods in which the search objective is tied to the operating environment rather than to a static validation metric alone. A plausible implication is that the framework treats non-stationarity as a first-class inductive bias instead of an ex post source of model drift.

2. Bayesian search space and the meaning of “regime-aware” NAS

The paper frames architecture search as Bayesian optimization over a search space $\mathcal{A}$ of candidate architectures. The search space includes recurrent cell types such as RNN/GRU/LSTM, hidden sizes, depth, dropout, activations, block-specific parameters, and connectivity patterns between blocks and the gating network. For a candidate architecture $\alpha \in \mathcal{A}$, validation performance is represented by a function $f(\alpha)$, such as negative loss or a composite metric. A Gaussian process surrogate is used:

$$f(\alpha) \sim \mathcal{GP}(\mu(\alpha), k(\alpha,\alpha'))$$

with a Matérn 5/2 kernel suggested in the text. Candidate selection proceeds by acquisition maximization, with UCB given explicitly as

$a(\alpha) = \mu(\alpha) + \beta_t \sigma(\alpha),$

where $\sigma(\alpha)$ is predictive uncertainty and $\beta_t$ controls exploration versus exploitation (Devadiga et al., 15 Aug 2025).

A distinctive mechanism is that $\beta_t$ is adjusted using regime-detection uncertainty:

$$\beta_t = \beta_{\text{base}} \cdot (1 + \gamma \cdot \text{uncertainty}_t).$$

If the regime detector is uncertain, the NAS process explores more aggressively. This is the paper’s concrete interpretation of regime-aware search at the optimizer level: uncertainty about market state directly modulates search behavior rather than affecting only the downstream predictor (Devadiga et al., 15 Aug 2025).

The paper also states an informal convergence theorem for the BO procedure. Under standard assumptions—well-behaved validation function, suitable GP kernel, compact search space—the search asymptotically finds the globally optimal architecture $\alpha^*$ given enough evaluations, with regret decreasing on the order of $\alpha \in \mathcal{A}$0 or faster. The paper characterizes this as a standard BO-style guarantee rather than a new proof. It also notes that, although the title uses the phrase “differentiable architecture search,” the concrete search engine described is Bayesian optimization rather than DARTS-style continuous relaxation; the differentiability primarily appears in the gating and end-to-end training once an architecture is chosen (Devadiga et al., 15 Aug 2025).

3. Regime identification through multi-head attention and uncertainty estimation

Regime detection is formulated as a learned classification problem over three named market states: Trending, Ranging, and High Volatility. The input is a sequence tensor

$\alpha \in \mathcal{A}$1

constructed from daily OHLCV data plus engineered indicators such as log returns, high-low range, volume changes, SMA/EMA, MACD, RSI, Bollinger Bands, ATR, realized volatility, and market-cap rank changes if available (Devadiga et al., 15 Aug 2025).

The detector uses multi-head self-attention. For time $\alpha \in \mathcal{A}$2,

$\alpha \in \mathcal{A}$3

For each head $\alpha \in \mathcal{A}$4, the paper gives

$\alpha \in \mathcal{A}$5

where $\alpha \in \mathcal{A}$6 may encode relative positional or market-specific masking. The outputs of all $\alpha \in \mathcal{A}$7 heads are concatenated and projected:

$\alpha \in \mathcal{A}$8

Pooling plus a linear layer and softmax then yields regime probabilities:

$\alpha \in \mathcal{A}$9

These probabilities are not only classifier outputs; they are the conditioning signal for the architecture adaptation process (Devadiga et al., 15 Aug 2025).

Uncertainty is estimated from disagreement among attention heads. The paper suggests computing variance or entropy across per-head regime predictions:

$f(\alpha)$0

This quantity is then used to adjust the BO exploration parameter $f(\alpha)$1. In operational terms, when the detector is unsure whether the market is trending or volatile, the search process explores more architecture options rather than over-committing to one (Devadiga et al., 15 Aug 2025).

A common misconception would be to read regime identification here as a hard market-state switch. The paper explicitly rejects that interpretation: regime probabilities are soft and remain coupled to downstream gating, allowing partial activation of multiple modules when the regime is ambiguous.

4. Specialized blocks and dynamic regime-conditioned gating

The search space includes three specialized modules, each tailored to a market regime and dynamically weighted by a gating network fed by regime probabilities. The gating network is

$f(\alpha)$2

and the adaptive output is

$f(\alpha)$3

The three blocks are Volatility, Trend, and Range blocks (Devadiga et al., 15 Aug 2025).

The Volatility blocks $f(\alpha)$4 are for turbulent, high-volatility periods. The paper describes volatility-gated recurrence, in which GRU/LSTM gates are modulated by a recent volatility estimate $f(\alpha)$5; adaptive activations, described as PLU/Swish-like nonlinearities whose parameters vary with $f(\alpha)$6; and volume-sensitive connections, in which skip or residual strengths can depend on trading volume. These blocks are intended to become more active when the detector identifies high volatility and are described as the main line of defense against unstable price jumps (Devadiga et al., 15 Aug 2025).

The Trend blocks $f(\alpha)$7 are intended for directional, persistent moves. They use multi-scale temporal convolutions with different kernel sizes and dilation rates to capture trend patterns at multiple horizons, together with adaptive pooling or momentum aggregation that emphasizes recent data when it supports the trend. The Range blocks $f(\alpha)$8 are for sideways, mean-reverting, range-bound markets. They use mean-reversion attention or oscillators, comparing current features to an estimated local mean or central tendency, and liquidity-aware processing, potentially weighting signals using support/resistance or depth information (Devadiga et al., 15 Aug 2025).

The regime detector therefore does not hard-switch the model. Instead, it produces probabilities, and the gating network converts those probabilities into soft weights. This allows simultaneous partial activation of multiple modules. A plausible implication is that the architecture is designed for mixtures of market behaviors rather than for a discrete taxonomy with perfectly separated states.

5. Multi-objective loss, stability constraints, and optimization procedure

Training uses a weighted sum of prediction accuracy and finance-specific regularizers:

$f(\alpha)$9

with reported weights

$$f(\alpha) \sim \mathcal{GP}(\mu(\alpha), k(\alpha,\alpha'))$$0

The prediction loss is standard MSE,

$$f(\alpha) \sim \mathcal{GP}(\mu(\alpha), k(\alpha,\alpha'))$$1

Volatility matching is

$$f(\alpha) \sim \mathcal{GP}(\mu(\alpha), k(\alpha,\alpha'))$$2

which penalizes mismatch between predicted and realized volatility over a window. Output smoothness is

$$f(\alpha) \sim \mathcal{GP}(\mu(\alpha), k(\alpha,\alpha'))$$3

Stability regularization is written as

$$f(\alpha) \sim \mathcal{GP}(\mu(\alpha), k(\alpha,\alpha'))$$4

with implementation described through adaptive spectral normalization, gradient clipping, Lipschitz regularization, careful activation choices, and stability-preserving residual connections (Devadiga et al., 15 Aug 2025).

A key theorem states that, by enforcing Lipschitz continuity on the specialized blocks and the gating network, the change in output caused by a change in regime probabilities is bounded:

$$f(\alpha) \sim \mathcal{GP}(\mu(\alpha), k(\alpha,\alpha'))$$5

where $$f(\alpha) \sim \mathcal{GP}(\mu(\alpha), k(\alpha,\alpha'))$$6. This is the paper’s formal statement that small regime shifts should not cause large output jumps. Two practical safeguards are given explicitly:

$$f(\alpha) \sim \mathcal{GP}(\mu(\alpha), k(\alpha,\alpha'))$$7

for spectral normalization, and

$$f(\alpha) \sim \mathcal{GP}(\mu(\alpha), k(\alpha,\alpha'))$$8

for gradient clipping (Devadiga et al., 15 Aug 2025).

The optimization loop is Bayesian. The algorithm initializes a GP surrogate, samples initial architectures, trains each candidate with $$f(\alpha) \sim \mathcal{GP}(\mu(\alpha), k(\alpha,\alpha'))$$9, evaluates on validation data, updates the GP, uses acquisition maximization to choose new candidates, and repeats for 10 generations. The implementation details reported are: Python 3.8+, PyTorch 1.10+, NVIDIA T4 GPUs, 10 generations, about 100 evaluations total, search time of approximately 3 GPU hours, final training of about 15 minutes, AdamW with learning rate $a(\alpha) = \mu(\alpha) + \beta_t \sigma(\alpha),$0 and cosine annealing, batch size 256, $a(\alpha) = \mu(\alpha) + \beta_t \sigma(\alpha),$1 regularization of $a(\alpha) = \mu(\alpha) + \beta_t \sigma(\alpha),$2, gradient clipping threshold 1.0, early stopping with max 20 epochs and patience 3, and a regime detector with 4 attention heads, $a(\alpha) = \mu(\alpha) + \beta_t \sigma(\alpha),$3, and 3 regimes (Devadiga et al., 15 Aug 2025).

6. Empirical results, ablations, limitations, and position within NAS research

The dataset is daily OHLCV data for over 20 major cryptocurrencies from CoinMarketCap, spanning Jan 1, 2013 to Dec 31, 2021, with a chronological split of 70% train, 15% validation, and 15% test. To combat non-stationarity, the data undergo adaptive normalization, such as rolling z-scores. Reported baselines are LSTM, GRU, RNN, Transformer, ConvLSTM, KAN, N-BEATS, D-PAD, and XGBoost on time features. Metrics are Loss, MAE, RMSE, $a(\alpha) = \mu(\alpha) + \beta_t \sigma(\alpha),$4, epochs to convergence, parameter count, and estimated final training time (Devadiga et al., 15 Aug 2025).

The best RegimeNAS architecture, discovered in generation 4 and then retrained fully, achieves Test MAE = 0.7570, Test RMSE = 2.2237, Test $a(\alpha) = \mu(\alpha) + \beta_t \sigma(\alpha),$5, Only 9 epochs for final training, and around 1.8M parameters. The paper emphasizes an 80.3% reduction in MAE compared with GRU, where GRU’s MAE is 3.8126. It also reports substantially faster convergence than the listed sequence-model baselines: GRU/LSTM at about 49–50 epochs, D-PAD at 100 epochs, N-BEATS at 38 epochs, and RegimeNAS at 9 epochs (Devadiga et al., 15 Aug 2025).

The ablation study assigns central importance to regime awareness. If regime detection is disabled and the blocks are statically averaged, MAE worsens by 63.4%. Removing specialized blocks yields +22.0% MAE for no Volatility blocks, +16.8% MAE for no Trend blocks, and +7.7% MAE for no Range blocks. Disabling stability regularization, spectral normalization, and adaptive gradient clipping causes a 12.4% MAE increase. In regime-specific evaluation under post-hoc labels, GRU reports MAEs of 4.10 / 8.55 / 5.20 for Trend / High Volatility / Range, whereas RegimeNAS reports 0.80 / 1.55 / 0.95. The largest reported advantage is in high volatility, consistent with the function assigned to the $a(\alpha) = \mu(\alpha) + \beta_t \sigma(\alpha),$6-blocks and the stability controls (Devadiga et al., 15 Aug 2025).

The paper also states several caveats. Real trading execution is not fully solved; the results are predictive rather than a full live-trading PnL study; regime definitions in the regime-specific analysis are simplified and assigned post-hoc using ADX and ATR thresholds; BO search still incurs cost, at about 3 GPU hours and approximately 100 evaluations; the theoretical guarantees are informal or standard; and metric comparability across baselines can be imperfect, especially for N-BEATS and XGBoost, due to scale or modeling differences. The regime detector additionally depends on engineered features and multi-timeframe inputs, which adds complexity and may make performance sensitive to feature quality (Devadiga et al., 15 Aug 2025).

Within the broader NAS literature, RegimeNAS belongs to a family of methods that alter the search process to reflect the target operating condition. "Robust Neural Architecture Search" searches for architectures that are accurate on clean data and stable under perturbations by adding a robustness regularizer, with RNAS-max using adversarial examples and RNAS-uniform using random noise examples (Zhu et al., 2023). "SAR-NAS: Lightweight SAR Object Detection with Neural Architecture Search" uses a one-shot supernet and hardware-aware evolutionary search over backbone channel widths to optimize the accuracy-efficiency trade-off for SAR object detection under on-board and edge constraints (Yu et al., 1 Sep 2025). This suggests that RegimeNAS is best understood as the financial, market-state-conditioned analogue of a more general pattern: embedding domain-specific structure directly into NAS rather than treating architecture search as regime-agnostic.