Frictional Q-learning: Concepts & Applications

Updated 12 July 2026

Frictional Q-learning is a family of value-based RL strategies that introduce resistance mechanisms—such as policy bias, entropy brakes, or state augmentation—to mitigate common estimation and overtraining failures.
These methods address issues like mismeasured Q-values, unsupported action extrapolation, and inadequate physical modeling, enhancing performance in repeated games, robotic control, and simulation tasks.
Empirical results demonstrate improved cooperation, efficiency gains, and robust adaptation across diverse environments, underscoring the practical impact of incorporating friction into Q-learning.

Frictional Q-learning denotes a family of value-based reinforcement-learning constructions in which some form of “friction” is introduced between estimated action values and realized policy updates. In the current literature, the term is not used in a single canonical sense. It appears as a policy bias added to Q-based action selection in repeated games, as an external entropy-based brake on overtraining in tabular Q-learning, as explicit conditioning of a critic on friction estimates in physical control, and as the proper name of a continuous-control off-policy algorithm that interprets extrapolation error through an analogy to static friction (Compte, 2023). A broader, but still value-based, usage also includes Q-critic methods for nonsmooth Coulomb-friction dynamics and undiscounted stochastic shortest path games, where convergence depends on structural conditions rather than discounting (Riley et al., 15 Jan 2025).

1. Terminological scope and core idea

The common element across these usages is that standard Q-learning is taken to be insufficiently robust when the learned value function is either structurally mismeasured, progressively distorted, physically incomplete, or extrapolated outside the support of available data. “Friction” then denotes a mechanism that resists that failure mode. Depending on the setting, the resisting mechanism is an additive policy distortion, an entropy-based stopping rule, an explicit friction variable in the state, or a constraint that prevents unsupported action drift (Kim et al., 24 Sep 2025).

Usage	Friction mechanism	Representative source
Learned collusion	Additive policy bias $Q_i^t(a_i)+b_iG_i(a_i)$	(Compte, 2023)
Information-loss control	DE-QT-based early stopping	(Nguyen et al., 2020)
Physical friction awareness	Critic conditioned on estimated $\mu_e$	(Raei et al., 24 Feb 2025)
Named algorithm “Frictional Q-Learning”	Batch-constrained, orthogonal-action constraint	(Kim et al., 24 Sep 2025)

A recurrent misconception is that frictional Q-learning names a single established algorithmic family analogous to DQN or SAC. The literature summarized here does not support that interpretation. Only "Frictional Q-Learning" (Kim et al., 24 Sep 2025) uses the phrase as the title of a specific algorithm. The other works use “friction” more as an interpretive device or application-specific organizing principle. This suggests that the term is presently best understood as a cross-cutting conceptual label rather than a universally standardized method.

2. Policy friction as biased Q-based action selection

In "Learned Collusion" (Compte, 2023), Q-learning is treated as a memoryless reinforcement-learning automaton whose benchmark policy is $\varepsilon$ -greedy over action values $Q_i^t(a_i)$ . The paper then introduces a family of Q-based automata with standard Q-updates but biased action selection: $a_i^t \in \arg\max_{a_i} \Big[ Q_i^t(a_i) + b_i\, G_i(a_i) \Big].$ Here $b_i \in \mathbb{R}$ is a one-dimensional bias parameter and $G_i(a_i)$ is an exogenous distortion function over actions. The standard naïve Q-learner is the special case $b_i=0$ (Compte, 2023).

The Q-process itself is unchanged. When action $a_i$ is played, the update is

$Q_{i}^{t+1}(a_{i}) = (1-\alpha)\,Q_{i}^{t}(a_{i}) + \alpha\Big((1-\delta)\,r_{i}^{t} + \delta\,\max_{x_{i}} Q_{i}^{t}(x_{i})\Big),$

and if $\mu_e$ 0 is not played, $\mu_e$ 1. The friction is therefore not a change in value backup but a distortion of the best-response map. The paper explicitly interprets this as a frictional departure from pure “best-response to Q”: the agent becomes systematically more willing to choose certain actions even when Q-values alone would favor alternatives (Compte, 2023).

This construction is developed in repeated environments with coarse state representation, including a single-agent hidden-state problem, repeated Prisoner’s Dilemma, stochastic payoff variants, and an oligopoly pricing game. Because the Q-function is action-based rather than state-conditioned, the paper emphasizes the emergence of “Q-traps,” in which continuation values of cooperative actions are underestimated. The bias $\mu_e$ 2 then acts as a corrective friction for that structural mismeasurement. In the Prisoner’s Dilemma, the operative rule is often written as

$\mu_e$ 3

while in the oligopoly game the distortion is

$\mu_e$ 4

so the policy favors prices that are profitable under symmetric behavior (Compte, 2023).

The same paper defines a meta-game over biases. Each player chooses $\mu_e$ 5 from a finite grid $\mu_e$ 6, long-run payoffs $\mu_e$ 7 are estimated by simulation of the underlying Q-learning dynamics, and a Qb-equilibrium is a Nash equilibrium in that bias game: $\mu_e$ 8 In the reported Prisoner’s Dilemma example with $\mu_e$ 9, $\varepsilon$ 0, $\varepsilon$ 1, $\varepsilon$ 2, $\varepsilon$ 3, and $\varepsilon$ 4, the anonymized payoff matrix shows two pure equilibria, $\varepsilon$ 5 and $\varepsilon$ 6, each with substantially higher average payoffs and much higher frequencies of joint cooperation than the naïve case (Compte, 2023).

The central mechanism is the shortening of low-Q defection phases and the preservation of high-Q cooperative phases. In stochastic-payoff variants, biased Q-learning with equilibrium biases $\varepsilon$ 7 yields high cooperation under both correlated and independent shocks, with payoffs around $\varepsilon$ 8– $\varepsilon$ 9, whereas naïve Q-learning under private shocks maintains very low cooperation rates. In the oligopoly pricing game, naïve Q-learning places about $Q_i^t(a_i)$ 0 of time at Nash prices and average profits around $Q_i^t(a_i)$ 1, while biased policies with equilibrium $Q_i^t(a_i)$ 2 shift the long-run distribution toward $Q_i^t(a_i)$ 3 and mean profits around $Q_i^t(a_i)$ 4 or higher (Compte, 2023).

The paper’s abstract summarizes the broader claim: stable equilibrium biases, easily learned under converging logit/best-response dynamics over biases and not requiring tacit agreement, strongly foster collusion or cooperation across a rich array of payoff and monitoring structures, independently of initial Q-values (Compte, 2023). In this usage, frictional Q-learning is most precisely “learned biased Q-learning”: a best-response map regularized by a stable bias parameter.

3. Friction as an external brake on overtraining

"Q-Learning with Differential Entropy of Q-Tables" (Nguyen et al., 2020) addresses a different pathology. The paper starts from the empirical observation that prolonged training in classic tabular Q-learning can degrade performance because of information loss in the Q-table. The core update remains standard,

$Q_i^t(a_i)$ 5

with Boltzmann action selection. The proposed intervention does not alter this update. Instead, it introduces Differential Entropy of Q-Tables (DE-QT) as an external diagnostic of the information content of the Q-table over training episodes (Nguyen et al., 2020).

The entropy proxy is constructed by interpreting the collection of Q-values as samples from a continuous random variable and estimating a histogram density $Q_i^t(a_i)$ 6. Differential entropy is then approximated as

$Q_i^t(a_i)$ 7

The paper introduces a conceptual mapping $Q_i^t(a_i)$ 8, where $Q_i^t(a_i)$ 9 is an information value of the Q-table, and defines information loss by $a_i^t \in \arg\max_{a_i} \Big[ Q_i^t(a_i) + b_i\, G_i(a_i) \Big].$ 0 (Nguyen et al., 2020).

Operationally, DE-QT is computed per episode. For each flag state $a_i^t \in \arg\max_{a_i} \Big[ Q_i^t(a_i) + b_i\, G_i(a_i) \Big].$ 1, the method tracks an entropy time series $a_i^t \in \arg\max_{a_i} \Big[ Q_i^t(a_i) + b_i\, G_i(a_i) \Big].$ 2, optionally aggregates them as $a_i^t \in \arg\max_{a_i} \Big[ Q_i^t(a_i) + b_i\, G_i(a_i) \Big].$ 3, and defines a stopping point

$a_i^t \in \arg\max_{a_i} \Big[ Q_i^t(a_i) + b_i\, G_i(a_i) \Big].$ 4

The key claim is that stopping at the entropy peak prevents the agent from entering an overtrained regime where the Q-table becomes less diverse and more overconfident (Nguyen et al., 2020).

This use of friction is explicitly external to the Q-update. The paper contrasts DE-QT with entropy-regularized RL and soft Q-learning, noting that DE-QT does not regularize the objective or alter the Bellman target. Instead, it regulates when to stop training. In the paper’s own phrasing, it is a friction-like control on training duration rather than a direct friction term in the update equation (Nguyen et al., 2020).

The empirical study uses a dynamic flag-collection gridworld with several state representations. DE-QT trajectories show early increase or stabilization followed by decay; higher-entropy regimes correlate with better generalization and efficient policies. At $a_i^t \in \arg\max_{a_i} \Big[ Q_i^t(a_i) + b_i\, G_i(a_i) \Big].$ 5, discounted reward is reported as $a_i^t \in \arg\max_{a_i} \Big[ Q_i^t(a_i) + b_i\, G_i(a_i) \Big].$ 6– $a_i^t \in \arg\max_{a_i} \Big[ Q_i^t(a_i) + b_i\, G_i(a_i) \Big].$ 7 higher than at episode $a_i^t \in \arg\max_{a_i} \Big[ Q_i^t(a_i) + b_i\, G_i(a_i) \Big].$ 8, success rates are often equal or higher, and efficiency improves substantially. For example, in the Global-8-8 setup, average steps drop from about $a_i^t \in \arg\max_{a_i} \Big[ Q_i^t(a_i) + b_i\, G_i(a_i) \Big].$ 9 at $b_i \in \mathbb{R}$ 0 to about $b_i \in \mathbb{R}$ 1 at $b_i \in \mathbb{R}$ 2 (Nguyen et al., 2020).

The paper also sketches possible direct extensions, such as entropy-dependent learning rates, exploration schedules, damping, or Q-shrinking when entropy falls. Those constructions are explicitly presented as proposed extensions rather than the algorithm itself. A careful reading therefore distinguishes the paper’s actual contribution—external information-loss detection and stopping—from more speculative forms of embedded frictional Q-learning (Nguyen et al., 2020).

4. Friction as a physical state variable in value-based control

A third usage arises in physical control problems where friction is part of the environment and the critic must be friction-aware. "A Reinforcement Learning Approach to Non-prehensile Manipulation through Sliding" (Raei et al., 24 Feb 2025) uses DDPG rather than classical tabular Q-learning, but its critic is explicitly a Q-function $b_i \in \mathbb{R}$ 3, making it a value-based example of friction-aware learning. The task is to slide an object on a surface by controlling the acceleration of a robotic arm rigidly coupled to the surface. Sliding begins only when platform acceleration exceeds maximum static friction,

$b_i \in \mathbb{R}$ 4

and relative motion during sliding satisfies

$b_i \in \mathbb{R}$ 5

The environment is therefore explicitly frictional (Raei et al., 24 Feb 2025).

In this setting, friction enters the RL loop through the state. The actor and critic receive an estimated kinetic friction coefficient $b_i \in \mathbb{R}$ 6 together with desired remaining displacement, previous actions, and previous realized displacements. A typical augmented state is described as

$b_i \in \mathbb{R}$ 7

The DDPG critic satisfies the Bellman equation

$b_i \in \mathbb{R}$ 8

so the value function is directly conditioned on estimated friction (Raei et al., 24 Feb 2025).

Two online friction estimation algorithms are developed. The first is an analytical estimator based on the kinematics of relative displacement across phases of a four-phase acceleration profile. The second is an LSTM regressor trained in MuJoCo on time series of platform acceleration and relative object velocity. The paper reports that the LSTM generally outperforms the analytical estimator in terms of percentage reduction of friction estimation error, measured by

$b_i \in \mathbb{R}$ 9

These updated estimates are fed back into the actor after each action, making the policy adaptive to surface variation (Raei et al., 24 Feb 2025).

This usage of frictional Q-learning is distinct from the bias-based and DE-QT interpretations. Friction is neither a regularizer nor a brake on updates; it is a latent physical parameter that the value function must represent. Domain randomization further reinforces that interpretation. During training, the state friction input is randomized by

$G_i(a_i)$ 0

with specified ranges for $G_i(a_i)$ 1, so that actor and critic learn to cope with uncertain friction estimates (Raei et al., 24 Feb 2025).

Empirically, the framework is validated in MuJoCo and on a Franka Emika Panda robot with a tray mounted on the end-effector. For desired displacement $G_i(a_i)$ 2 cm, displacement error remains within $G_i(a_i)$ 3 cm with accurate friction estimate and below $G_i(a_i)$ 4 cm under small friction mismatch in real experiments. The trained model is reported to exhibit zero-shot sim-to-real transfer, and the paper emphasizes that explicit friction modeling and online friction estimation substantially improve adaptability (Raei et al., 24 Feb 2025). A plausible implication is that, in robotics, “frictional Q-learning” is best read as Q-learning in an augmented state space whose critic explicitly conditions on friction.

5. The algorithm named “Frictional Q-Learning”

The most literal use of the term appears in "Frictional Q-Learning" (Kim et al., 24 Sep 2025), which presents a deep reinforcement-learning algorithm for continuous control. The paper draws an analogy between static friction in classical mechanics and extrapolation error in off-policy RL. Extrapolation error arises when TD targets depend on state-action pairs not well supported by the replay buffer $G_i(a_i)$ 5. The paper writes this error, in the BCQ formulation, as

$G_i(a_i)$ 6

Unsupported actions are then treated as analogous to moving into regions with larger frictional resistance (Kim et al., 24 Sep 2025).

The algorithm extends batch-constrained reinforcement learning. Its objective is to constrain the agent’s action space so that the policy remains close to behavior seen in the replay buffer while maintaining distance from an orthonormal action manifold represented by a heterogeneous buffer $G_i(a_i)$ 7. This is implemented by three components: a deterministic actor-critic, a state-conditioned contrastive VAE (cVAE), and explicitly constructed orthonormal actions used as background samples (Kim et al., 24 Sep 2025).

The critic networks $G_i(a_i)$ 8 and $G_i(a_i)$ 9 follow a TD3-style double-Q structure. Candidate actions for next states are generated by the cVAE, and the target uses max over candidates and min over critics: $b_i=0$ 0 The policy at test time likewise chooses the highest-Q candidate among cVAE-generated samples: $b_i=0$ 1 The “frictional” constraint therefore acts through the generative action space rather than through an explicit penalty in the critic loss (Kim et al., 24 Sep 2025).

A defining element is the heterogeneous buffer of orthonormal actions. For a buffer action $b_i=0$ 2, the paper states that orthonormal vectors $b_i=0$ 3 exist under $b_i=0$ 4 and $b_i=0$ 5. These orthogonal actions are treated as background data in the cVAE objective, while buffer actions are target data. The cVAE separates salient latent structure associated with buffer-supported actions from nuisance structure associated with orthogonal actions, with a total-correlation term used to enforce independence (Kim et al., 24 Sep 2025).

The paper’s main theoretical statement, Theorem 5, gives an extrapolation-error expression comparing true dynamics $b_i=0$ 6 with orthogonal dynamics $b_i=0$ 7, and Remark 1 provides a bound

$b_i=0$ 8

together with an angular constraint

$b_i=0$ 9

In the paper’s interpretation, this angle quantifies the mismatch between replay-buffer dynamics and orthogonal dynamics, and the frictional constraint controls extrapolation through that geometry (Kim et al., 24 Sep 2025).

Empirical evaluation is conducted on MuJoCo tasks via Gymnasium. The reported baselines are SAC, MEOW, TD3, and DDPG. FQL achieves state-of-the-art performance on Walker2D-v4 and Humanoid-v4 across the paper’s Step, Seed, and Final metrics, while underperforming some baselines on lower-dimensional Hopper-v4. Selected reported results include Humanoid-v4 Step $a_i$ 0 and Walker2D-v4 Seed $a_i$ 1, both marked as best among listed methods (Kim et al., 24 Sep 2025). The paper also notes hyperparameter sensitivity, especially to the cVAE KL weight $a_i$ 2, and additional computational cost from cVAE training and orthonormal augmentation.

A broader research context shows that frictional Q-learning also arises when Q-critics are deployed in friction-dominated dynamics, even when the method is not called by that name. "Reinforcement learning-based adaptive time-integration for nonsmooth dynamics" (Riley et al., 15 Jan 2025) applies Truncated Quantile Critics (TQC), an off-policy actor-critic algorithm with distributional Q-critics, to choose time steps for a seismic fault with Coulomb friction. The underlying dynamics include the linear momentum equation

$a_i$ 3

a slip-weakening friction law

$a_i$ 4

and a reward that balances large time steps, low runtime, and low local integration error. The paper reports up to a fourfold speed-up and describes the method as a concrete example of what one might call “frictional Q-learning,” even though the algorithm is TQC rather than tabular Q-learning (Riley et al., 15 Jan 2025).

In that work, the critic learns which time-step actions are valuable under different frictional regimes summarized by normalized internal energy, normalized slip velocity, solver iteration error, relative local error, normalized current time step, and a convergence indicator. The learned policy differs from a heuristic adaptive integrator by maintaining smaller, steadier time steps around slip events and increasing them during sticking phases. This suggests a distinct interpretation of frictional Q-learning: Q-value learning over actions that control the numerical treatment of a frictional physical system rather than the physical friction law itself (Riley et al., 15 Jan 2025).

At a more foundational level, "Stochastic Shortest Path Games and Q-Learning" (Yu, 2014) establishes convergence of classical asynchronous Q-learning for a broad class of undiscounted total-cost two-player zero-sum stochastic games with a cost-free termination state. The paper does not use the word “frictional” as an algorithmic label, but its structure is directly relevant to termination-based environments where costs accumulate until some resolution. It defines a Q-DP operator

$a_i$ 5

and proves boundedness of Q-learning iterates under proposed SSP model conditions, thereby establishing convergence for the finite-space case (Yu, 2014).

The significance of that result is methodological. In discounted problems, convergence typically relies on contraction. In SSP games, convergence is obtained instead through termination structure, proper policies, uniqueness of the dynamic-programming solution, and boundedness of iterates. A plausible implication is that some environments described informally as “frictional”—because costs persist until an absorbing resolution—are better analyzed through SSP theory than through any explicit regularizer or bias term (Yu, 2014).

7. Limitations, distinctions, and open directions

Across these works, the principal limitation is conceptual heterogeneity. In one line of work, friction is an additive policy bias chosen in a meta-game over automata (Compte, 2023). In another, it is an external training brake based on entropy of the Q-table (Nguyen et al., 2020). In robotics, it is a physical coefficient that enters the state and is estimated online (Raei et al., 24 Feb 2025). In FQL proper, it is a geometric analogy for extrapolation error that yields a constraint against unsupported actions (Kim et al., 24 Sep 2025). These are related by function rather than by formal identity.

Each formulation also has problem-specific restrictions. The collusion framework optimizes only the bias $a_i$ 6 while keeping $a_i$ 7 and $a_i$ 8 fixed, uses memoryless action-based Q-values, and identifies equilibria empirically via simulation rather than by general theorems (Compte, 2023). DE-QT is demonstrated in a single flag-collection gridworld, adds histogram-based computational overhead, and does not provide formal optimal-stopping guarantees (Nguyen et al., 2020). The sliding-manipulation framework relies on Coulomb friction as an approximation, is sensitive to noisy friction estimates, and uses DDPG rather than classical tabular Q-learning (Raei et al., 24 Feb 2025). FQL depends on continuous action spaces of dimension at least two, requires cVAE tuning, and can underperform in low-dimensional tasks such as Hopper-v4 (Kim et al., 24 Sep 2025). The seismic time-integration application required transfer learning for robust convergence on finer meshes (Riley et al., 15 Jan 2025).

A second important distinction concerns where friction enters the learning system. It may modify policy selection while leaving TD backup untouched, as in biased Q-based automata; act only on training duration, as in DE-QT; parameterize the critic’s state, as in friction-aware manipulation; or reshape the admissible action manifold, as in FQL. Conflating these mechanisms obscures the substantive differences between regularization, state augmentation, and support constraint.

Open directions are stated or implied in several papers. "Learned Collusion" notes that agents might also adjust $a_i$ 9 or $Q_{i}^{t+1}(a_{i}) = (1-\alpha)\,Q_{i}^{t}(a_{i}) + \alpha\Big((1-\delta)\,r_{i}^{t} + \delta\,\max_{x_{i}} Q_{i}^{t}(x_{i})\Big),$ 0 strategically, making learning rate or exploration another friction dimension (Compte, 2023). The DE-QT paper suggests future work on entropy-driven modulation of learning rate, exploration, or direct regularization of Q-values (Nguyen et al., 2020). The sliding-manipulation paper points to richer friction models beyond ideal Coulomb behavior and to more complex non-prehensile tasks (Raei et al., 24 Feb 2025). FQL identifies stabilization of the state-conditioned generative model and extensions to purely offline datasets or alternative heterogeneous-buffer constructions as future directions (Kim et al., 24 Sep 2025).

Taken together, the literature supports a precise but plural definition: frictional Q-learning is not one algorithm but a class of value-based strategies that introduce resistance against failure modes of naïve Q-learning—myopic best-response to mismeasured continuation values, information-loss through prolonged training, friction-unaware action valuation in physical control, or extrapolation outside replay support. The named algorithm "Frictional Q-Learning" (Kim et al., 24 Sep 2025) is the most specific instantiation, but the surrounding literature shows that the broader idea spans economic games, tabular diagnostics, robotic manipulation, nonsmooth numerical mechanics, and undiscounted dynamic games.