Chebyshev-DQN: A Polynomial Representation for Deep RL

• Chebyshev-DQN is a DQN variant that replaces the standard MLP feature extractor with a fixed Chebyshev polynomial basis to improve value function approximation.
• The method retains core DQN mechanisms such as experience replay and ε-greedy policies while altering only the representation layer via structured polynomial expansion.
• Empirical results on CartPole-v1 show that a moderate degree (N=4) can boost performance by ~39%, whereas a higher degree (N=8) may introduce instability due to overfitting.

Chebyshev-DQN (Ch-DQN) is a Deep Q-Network variant that replaces the usual front-end multi-layer perceptron feature formation with a fixed Chebyshev polynomial basis expansion while preserving the standard DQN reinforcement-learning algorithm. It is introduced in "Beyond ReLU: Chebyshev-DQN for Enhanced Deep Q-Networks" (Yazdannik et al., 20 Aug 2025) as a direct attempt to improve the function-approximation core of DQN by engineering the input representation of the action-value function $Q(s,a)$. The method keeps experience replay, a target network, $\epsilon$-greedy action selection, and Bellman-error minimization intact, but inserts a mathematically defined feature extractor based on Chebyshev polynomials before the downstream value-estimation network. On CartPole-v1, the reported outcome is that a moderate polynomial degree, $N=4$, achieves a final score of 347.9 versus 250.5 for a standard DQN baseline, whereas a higher degree, $N=8$, degrades performance to 144.0, highlighting a task-dependent trade-off between approximation power and instability (Yazdannik et al., 20 Aug 2025).

1. Conceptual basis and motivation

Ch-DQN is motivated by the claim that DQN performance is critically dependent on how accurately the neural network approximates the action-value function. In the formulation presented in the paper, standard DQN uses a deep neural network, typically an MLP, as the approximator, but this requires the agent to discover an effective representation from scratch. Ch-DQN instead introduces a structured basis expansion grounded in classical approximation theory, with the stated aim of creating a more effective feature representation for value estimation (Yazdannik et al., 20 Aug 2025).

The central mathematical rationale is the use of Chebyshev polynomials as an orthogonal basis with strong approximation properties. The paper emphasizes their minimax behavior, stating that truncated Chebyshev expansions are near-optimal polynomial approximants under the $L_\infty$ norm. On that basis, it hypothesizes that transforming the state into a Chebyshev feature space can permit more efficient representation of value functions, with lower approximation error and potentially better-conditioned learning dynamics (Yazdannik et al., 20 Aug 2025).

This framing is narrow and specific. The method is not presented as a modification of the reinforcement-learning objective or update rule; the novelty is in the representation layer. A plausible implication is that Ch-DQN belongs to a class of methods that seek gains from approximation structure rather than from changes to temporal-difference learning itself.

2. Architecture and feature construction

Architecturally, Ch-DQN preserves the DQN training loop and swaps in a Chebyshev-based feature extractor at the front of the Q-network. The input state $\mathbf{s}$ is first normalized into $[-1,1]$, because the Chebyshev polynomials are defined on that interval. For each state dimension $s_i$, the network computes the first $N$ Chebyshev polynomials of the first kind, and these values are concatenated across dimensions to form the feature vector (Yazdannik et al., 20 Aug 2025).

The polynomial family is defined recursively as

$$T_0(x)=1,\quad T_1(x)=x,\quad T_{n+1}(x)=2xT_n(x)-T_{n-1}(x)\quad \text{for } n\ge 1.$$

If the normalized state is $\epsilon$0, the resulting representation is

$\epsilon$1

The feature vector therefore has size $\epsilon$2. The Q-function is then approximated by a fully connected network applied to those features: $\epsilon$3 where $\epsilon$4 is the post-Chebyshev feed-forward network and $\epsilon$5 indexes the output corresponding to a specific action (Yazdannik et al., 20 Aug 2025).

The paper explicitly describes this as separating feature extraction from value estimation. The Chebyshev layer is fixed and mathematically defined, whereas the subsequent MLP learns the mapping from those features to Q-values. The degree $\epsilon$6 is identified as the critical hyperparameter: increasing $\epsilon$7 enriches the representation, but also increases expressivity and may introduce noise sensitivity. This suggests that Ch-DQN should be understood less as a universally stronger approximator than as a controlled basis-expansion variant whose efficacy depends on matching basis complexity to task complexity.

3. Learning objective and retained DQN mechanics

Training in Ch-DQN follows standard DQN. For a transition $\epsilon$8 sampled from replay memory, the target is

$\epsilon$9

with $N=4$0 denoting the periodically updated target network. The loss is the mean squared TD error: $N=4$1 Accordingly, the paper characterizes the state-to-feature transform as the only real novelty, while the reinforcement-learning objective remains standard DQN (Yazdannik et al., 20 Aug 2025).

This point matters for interpretation. Ch-DQN is not presented as altering bootstrapping, off-policy learning, target computation, or exploration policy. Experience replay, target-network usage, $N=4$2-greedy action selection, and Bellman-error minimization are all preserved. The empirical differences reported for Ch-DQN are therefore attributed to representation change rather than to modified optimization targets or altered credit assignment (Yazdannik et al., 20 Aug 2025).

A common misunderstanding would be to treat Ch-DQN as a new RL algorithm in the broader sense. The description in the paper does not support that reading. It is more precisely a DQN architecture variant in which the front end is deliberately engineered using an orthogonal polynomial basis.

4. Experimental configuration on CartPole-v1

The experiments were run on three Gymnasium benchmarks—CartPole-v1, MountainCar-v0, and Acrobot-v1—but the CartPole-v1 results are the primary focus for the main comparison between a standard DQN baseline and Ch-DQN variants with polynomial degrees $N=4$3 (Yazdannik et al., 20 Aug 2025). For CartPole, the baseline DQN used two hidden layers of 64 neurons with ReLU activations. The Ch-DQN models used the same 64-64 downstream MLP after the Chebyshev feature layer. The comparison was designed so that MLP sizes were matched as closely as possible (Yazdannik et al., 20 Aug 2025).

The reported CartPole-v1 parameter counts and main outcomes are as follows.

Model	Trainable parameters	CartPole-v1 final score
Standard DQN	4,610	250.5
Ch-DQN ($N=4$4)	5,634	347.9
Ch-DQN ($N=4$5)	6,146	not specified in the provided details
Ch-DQN ($N=4$6)	6,658	144.0

The core hyperparameters were shared across models: Adam optimizer, $N=4$7, replay buffer size 50,000, batch size 64, and target-network update every 500 steps. The paper reports three independent runs with different random seeds. For CartPole, it reports final performance after training and states that the learning curves show the effect of polynomial degree to be highly non-monotonic (Yazdannik et al., 20 Aug 2025).

This setup is important because the paper explicitly uses parameter counts to argue that the observed gain is not merely from a massive capacity increase. The absolute parameter difference between the baseline and $N=4$8 Ch-DQN is present, but the interpretation advanced in the paper is that the benefit arises from structured features rather than raw scale alone.

5. Reported performance and degree sensitivity

The headline quantitative result on CartPole-v1 is that the moderate-degree Chebyshev model, $N=4$9, achieved a final score of 347.9, compared with 250.5 for the baseline DQN, which the paper describes as an improvement of roughly 39% (Yazdannik et al., 20 Aug 2025). This is also the result emphasized in the abstract. The same study, however, stresses that higher degree is not automatically better: $N=8$0 achieved a final score of only 144.0, substantially below the baseline (Yazdannik et al., 20 Aug 2025).

The reported pattern is therefore non-monotonic. Moderate polynomial degree improved asymptotic control performance, while an overly rich polynomial expansion harmed learning on this task. The paper interprets this as evidence that the Chebyshev basis can be useful, but only when the degree is appropriate to the complexity of the task (Yazdannik et al., 20 Aug 2025).

This degree sensitivity is central to the identity of Ch-DQN. It is not presented as a plug-and-play replacement for ReLU-based DQN in which increasing basis richness predictably improves results. Instead, the findings support the existence of a task-dependent “sweet spot” in polynomial degree. A plausible implication is that model selection for Ch-DQN is inseparable from assumptions about the smoothness and complexity of the target value landscape.

6. Interpretation, limitations, and relation to orthogonal basis methods

The paper’s explanation for the improvement at moderate degree is rooted in approximation theory and spectral bias. It argues that Chebyshev polynomials are orthogonal and well-conditioned relative to monomials, so projecting the state into a Chebyshev basis yields a feature space that is easier to fit and less prone to destructive interference than a generic learned representation. It also invokes the minimax property of truncated Chebyshev expansions, suggesting reduced worst-case approximation error for the Q-function class. On simpler tasks such as CartPole, the paper states that a low-to-moderate degree basis is sufficient to capture the true value landscape (Yazdannik et al., 20 Aug 2025).

The explanation for deterioration at higher degree is different. Although larger $N=8$1 increases expressive power, the paper states that it also introduces high-frequency basis functions that can overfit noise in TD targets. Because DQN targets are bootstrapped estimates generated by the agent itself rather than stationary truth labels, a highly expressive basis can fit these noisy targets too closely and amplify instability. The argument is framed in terms of spectral bias through the identity $N=8$2, with higher degrees corresponding to higher-frequency components. In the paper’s interpretation, CartPole’s value function is relatively simple, so $N=8$3 is unnecessarily expressive and degrades learning (Yazdannik et al., 20 Aug 2025).

The broader discussion places Ch-DQN in the literature on orthogonal function approximation in reinforcement learning. The paper emphasizes that the advantage of Chebyshev features is not only expressive power but also orthogonality, which can improve conditioning and reduce interference between features. At the same time, it is explicit about limitations: polynomial degree is a sensitive hyperparameter, the model is not universally better than standard DQN, and high-degree bases can be detrimental, especially when the task is simple or TD targets are noisy. The authors therefore suggest future work on adaptive degree selection and on combining Ch-DQN with stronger exploration methods, particularly for sparse-reward problems where approximation improvements alone may not be enough (Yazdannik et al., 20 Aug 2025).

Taken together, these claims define Ch-DQN as a representation-engineering approach within deep RL. Its core message is not that ReLU-based DQN is displaced in general, but that a fixed orthogonal polynomial basis can, under suitable degree choice, improve the input representation on which DQN depends. On CartPole-v1, the reported evidence supports that claim for $N=8$4 and simultaneously delineates its boundary conditions through the failure mode at $N=8$5 (Yazdannik et al., 20 Aug 2025).