---
title: 'Frictional Q-learning: Concepts & Applications'
url: https://www.emergentmind.com/topics/frictional-q-learning
type: topic
---

# Frictional Q-learning: Concepts & Applications

Frictional Q-learning denotes a family of value-based reinforcement-learning constructions in which some form of “friction” is introduced between estimated action values and realized policy updates. In the current literature, the term is not used in a single canonical sense. It appears as a policy bias added to Q-based action selection in repeated games, as an external entropy-based brake on overtraining in tabular Q-learning, as explicit conditioning of a critic on friction estimates in physical control, and as the proper name of a continuous-control off-policy algorithm that interprets extrapolation error through an analogy to static friction [2304.12647]. A broader, but still value-based, usage also includes Q-critic methods for nonsmooth Coulomb-friction dynamics and undiscounted stochastic shortest path games, where convergence depends on structural conditions rather than discounting [2501.08934].

## 1. Terminological scope and core idea

The common element across these usages is that standard Q-learning is taken to be insufficiently robust when the learned value function is either structurally mismeasured, progressively distorted, physically incomplete, or extrapolated outside the support of available data. “Friction” then denotes a mechanism that resists that failure mode. Depending on the setting, the resisting mechanism is an additive policy distortion, an entropy-based stopping rule, an explicit friction variable in the state, or a constraint that prevents unsupported action drift [2509.19771].

| Usage | Friction mechanism | Representative source |
|---|---|---|
| Learned collusion | Additive policy bias \(Q_i^t(a_i)+b_iG_i(a_i)\) | [2304.12647] |
| Information-loss control | DE-QT-based early stopping | [2006.14795] |
| Physical friction awareness | Critic conditioned on estimated \(\mu_e\) | [2502.17221] |
| Named algorithm “Frictional Q-Learning” | Batch-constrained, orthogonal-action constraint | [2509.19771] |

A recurrent misconception is that frictional Q-learning names a single established algorithmic family analogous to DQN or SAC. The literature summarized here does not support that interpretation. Only "Frictional Q-Learning" [2509.19771] uses the phrase as the title of a specific algorithm. The other works use “friction” more as an interpretive device or application-specific organizing principle. This suggests that the term is presently best understood as a cross-cutting conceptual label rather than a universally standardized method.

## 2. Policy friction as biased Q-based action selection

In "Learned Collusion" [2304.12647], Q-learning is treated as a memoryless reinforcement-learning automaton whose benchmark policy is \(\varepsilon\)-greedy over action values \(Q_i^t(a_i)\). The paper then introduces a family of Q-based automata with standard Q-updates but biased action selection:
\[
a_i^t \in \arg\max_{a_i} \Big[ Q_i^t(a_i) + b_i\, G_i(a_i) \Big].
\]
Here \(b_i \in \mathbb{R}\) is a one-dimensional bias parameter and \(G_i(a_i)\) is an exogenous distortion function over actions. The standard naïve Q-learner is the special case \(b_i=0\) [2304.12647].

The Q-process itself is unchanged. When action \(a_i\) is played, the update is
\[
Q_{i}^{t+1}(a_{i}) = (1-\alpha)\,Q_{i}^{t}(a_{i}) + \alpha\Big((1-\delta)\,r_{i}^{t} + \delta\,\max_{x_{i}} Q_{i}^{t}(x_{i})\Big),
\]
and if \(a_i\) is not played, \(Q_i^{t+1}(a_i)=Q_i^t(a_i)\). The friction is therefore not a change in value backup but a distortion of the best-response map. The paper explicitly interprets this as a frictional departure from pure “best-response to Q”: the agent becomes systematically more willing to choose certain actions even when Q-values alone would favor alternatives [2304.12647].

This construction is developed in repeated environments with coarse state representation, including a single-agent hidden-state problem, repeated Prisoner’s Dilemma, stochastic payoff variants, and an oligopoly pricing game. Because the Q-function is action-based rather than state-conditioned, the paper emphasizes the emergence of “Q-traps,” in which continuation values of cooperative actions are underestimated. The bias \(b_i\) then acts as a corrective friction for that structural mismeasurement. In the Prisoner’s Dilemma, the operative rule is often written as
\[
Q_i(C)+b_i > Q_i(D),
\]
while in the oligopoly game the distortion is
\[
Q_{i,b_i}(p_i)=Q_i(p_i)+b_i\,\pi_i(p_i,p_i),
\]
so the policy favors prices that are profitable under symmetric behavior [2304.12647].

The same paper defines a meta-game over biases. Each player chooses \(b_i\) from a finite grid \(B_i\), long-run payoffs \(v_i(b_i,b_{-i})\) are estimated by simulation of the underlying Q-learning dynamics, and a **Qb-equilibrium** is a Nash equilibrium in that bias game:
\[
b_i^* \in \arg\max_{b_i\in B_i} v_i(b_i,b_{-i}^*).
\]
In the reported Prisoner’s Dilemma example with \(\alpha=0.5\), \(\varepsilon=0.1\), \(x=2.5\), \(y=-0.5\), \(\varpi=0.02\), and \(\kappa_i\in\{0,\dots,4\}\), the anonymized payoff matrix shows two pure equilibria, \(b^*=2\varpi\) and \(b^*=3\varpi\), each with substantially higher average payoffs and much higher frequencies of joint cooperation than the naïve case [2304.12647].

The central mechanism is the shortening of low-Q defection phases and the preservation of high-Q cooperative phases. In stochastic-payoff variants, biased Q-learning with equilibrium biases \(\kappa^*\in\{2,3,4\}\) yields high cooperation under both correlated and independent shocks, with payoffs around \(1.8\)–\(1.9\), whereas naïve Q-learning under private shocks maintains very low cooperation rates. In the oligopoly pricing game, naïve Q-learning places about \(60\%\) of time at Nash prices and average profits around \(2.25\), while biased policies with equilibrium \(b^*>0\) shift the long-run distribution toward \(p^3,p^4,p^5\) and mean profits around \(3.0\) or higher [2304.12647].

The paper’s abstract summarizes the broader claim: stable equilibrium biases, easily learned under converging logit/best-response dynamics over biases and not requiring tacit agreement, strongly foster collusion or cooperation across a rich array of payoff and monitoring structures, independently of initial Q-values [2304.12647]. In this usage, frictional Q-learning is most precisely “learned biased Q-learning”: a best-response map regularized by a stable bias parameter.

## 3. Friction as an external brake on overtraining

"Q-Learning with Differential Entropy of Q-Tables" [2006.14795] addresses a different pathology. The paper starts from the empirical observation that prolonged training in classic tabular Q-learning can degrade performance because of information loss in the Q-table. The core update remains standard,
\[
Q_{t+1}(s_t,a_t) = Q_t(s_t,a_t) + \alpha \Bigl(r_t + \gamma \max_{a'} Q_t(s_{t+1},a') - Q_t(s_t,a_t)\Bigr),
\]
with Boltzmann action selection. The proposed intervention does not alter this update. Instead, it introduces **Differential Entropy of Q-Tables (DE-QT)** as an external diagnostic of the information content of the Q-table over training episodes [2006.14795].

The entropy proxy is constructed by interpreting the collection of Q-values as samples from a continuous random variable and estimating a histogram density \(\hat f(x)\). Differential entropy is then approximated as
\[
\varepsilon(Q(X_t)) = -\sum_{i=1}^{n} \hat{f}(x^{(i)}) \log \frac{\hat{f}(x^{(i)})}{w(x^{(i)})}.
\]
The paper introduces a conceptual mapping \(f:\varepsilon(Q(X_t))\mapsto J(X_t)\), where \(J(X_t)\) is an information value of the Q-table, and defines information loss by \(\delta J_t = J^*-J(\varepsilon_t)\) [2006.14795].

Operationally, DE-QT is computed per episode. For each flag state \(j\), the method tracks an entropy time series \(\varepsilon_j(t)\), optionally aggregates them as \(\varepsilon_{\text{sum}}(t)\), and defines a stopping point
\[
t_{\max} = \arg\max_t \varepsilon_{\text{sum}}(t).
\]
The key claim is that stopping at the entropy peak prevents the agent from entering an overtrained regime where the Q-table becomes less diverse and more overconfident [2006.14795].

This use of friction is explicitly external to the Q-update. The paper contrasts DE-QT with entropy-regularized RL and soft Q-learning, noting that DE-QT does not regularize the objective or alter the Bellman target. Instead, it regulates **when to stop training**. In the paper’s own phrasing, it is a friction-like control on training duration rather than a direct friction term in the update equation [2006.14795].

The empirical study uses a dynamic flag-collection gridworld with several state representations. DE-QT trajectories show early increase or stabilization followed by decay; higher-entropy regimes correlate with better generalization and efficient policies. At \(t_{\max}\), discounted reward is reported as \(17\)–\(200\%\) higher than at episode \(10{,}000\), success rates are often equal or higher, and efficiency improves substantially. For example, in the Global-8-8 setup, average steps drop from about \(298\) at \(t_{10000}\) to about \(111\) at \(t_{\max}\) [2006.14795].

The paper also sketches possible direct extensions, such as entropy-dependent learning rates, exploration schedules, damping, or Q-shrinking when entropy falls. Those constructions are explicitly presented as proposed extensions rather than the algorithm itself. A careful reading therefore distinguishes the paper’s actual contribution—external information-loss detection and stopping—from more speculative forms of embedded frictional Q-learning [2006.14795].

## 4. Friction as a physical state variable in value-based control

A third usage arises in physical control problems where friction is part of the environment and the critic must be friction-aware. "A Reinforcement Learning Approach to Non-prehensile Manipulation through Sliding" [2502.17221] uses DDPG rather than classical tabular Q-learning, but its critic is explicitly a Q-function \(Q(s,a)\), making it a value-based example of friction-aware learning. The task is to slide an object on a surface by controlling the acceleration of a robotic arm rigidly coupled to the surface. Sliding begins only when platform acceleration exceeds maximum static friction,
\[
|\ddot{X}| > \mu_s g,
\]
and relative motion during sliding satisfies
\[
\ddot{x}_A^B = -\ddot{X} - \frac{\dot{x}_A^B}{|\dot{x}_A^B|}\,\mu_k g.
\]
The environment is therefore explicitly frictional [2502.17221].

In this setting, friction enters the RL loop through the state. The actor and critic receive an estimated kinetic friction coefficient \(\mu_e\) together with desired remaining displacement, previous actions, and previous realized displacements. A typical augmented state is described as
\[
s_t = \bigl[ D_{\mathrm{des},t}, A_{t-1}, A_{t-2}, A_{t-3}, D_{t-1}, D_{t-2}, D_{t-3}, \mu_{e,t} \bigr].
\]
The DDPG critic satisfies the Bellman equation
\[
Q(s,a)=r(s,a)+\gamma\,\mathbb{E}_{s'}[Q(s',\pi(s'))],
\]
so the value function is directly conditioned on estimated friction [2502.17221].

Two online friction estimation algorithms are developed. The first is an analytical estimator based on the kinematics of relative displacement across phases of a four-phase acceleration profile. The second is an LSTM regressor trained in MuJoCo on time series of platform acceleration and relative object velocity. The paper reports that the LSTM generally outperforms the analytical estimator in terms of percentage reduction of friction estimation error, measured by
\[
\text{Correction of } \mu_e
= 1 - \frac{|\mu_k - \mu_e'|}{|\mu_k - \mu_e|}.
\]
These updated estimates are fed back into the actor after each action, making the policy adaptive to surface variation [2502.17221].

This usage of frictional Q-learning is distinct from the bias-based and DE-QT interpretations. Friction is neither a regularizer nor a brake on updates; it is a latent physical parameter that the value function must represent. Domain randomization further reinforces that interpretation. During training, the state friction input is randomized by
\[
\bar{\mu} = \mu \pm \eta,
\]
with specified ranges for \(\eta\), so that actor and critic learn to cope with uncertain friction estimates [2502.17221].

Empirically, the framework is validated in MuJoCo and on a Franka Emika Panda robot with a tray mounted on the end-effector. For desired displacement \(D_{\mathrm{des}}=8\) cm, displacement error remains within \(1\) cm with accurate friction estimate and below \(2\) cm under small friction mismatch in real experiments. The trained model is reported to exhibit zero-shot sim-to-real transfer, and the paper emphasizes that explicit friction modeling and online friction estimation substantially improve adaptability [2502.17221]. A plausible implication is that, in robotics, “frictional Q-learning” is best read as Q-learning in an augmented state space whose critic explicitly conditions on friction.

## 5. The algorithm named “Frictional Q-Learning”

The most literal use of the term appears in "Frictional Q-Learning" [2509.19771], which presents a deep reinforcement-learning algorithm for continuous control. The paper draws an analogy between static friction in classical mechanics and extrapolation error in off-policy RL. Extrapolation error arises when TD targets depend on state-action pairs not well supported by the replay buffer \(\mathcal{B}\). The paper writes this error, in the BCQ formulation, as
\[
\mathcal{E}_{\theta}(s,a) = \sum_{s'} \big( p_0(s'|s,a) - p_\theta(s'|s,a) \big) \Big( r(s,a,s') + \gamma \sum_{a'} \pi(a'|s') Q^{\pi}_\theta(s', a') \Big) + \sum_{s'} p_0(s'|s,a) \gamma \sum_{a'} \pi(a'|s') \mathcal{E}_{\theta}(s', a') .
\]
Unsupported actions are then treated as analogous to moving into regions with larger frictional resistance [2509.19771].

The algorithm extends batch-constrained reinforcement learning. Its objective is to constrain the agent’s action space so that the policy remains close to behavior seen in the replay buffer while maintaining distance from an orthonormal action manifold represented by a heterogeneous buffer \(\mathcal{H}\). This is implemented by three components: a deterministic actor-critic, a state-conditioned contrastive VAE (cVAE), and explicitly constructed orthonormal actions used as background samples [2509.19771].

The critic networks \(Q_{\varphi_1}\) and \(Q_{\varphi_2}\) follow a TD3-style double-Q structure. Candidate actions for next states are generated by the cVAE, and the target uses max over candidates and min over critics:
\[
y_{i+1} =
\max_{d \in \{1,\dots,c\}}
\min_{\ell \in \{1,2\}}
Q_{\varphi_\ell^-}\!\bigl(s_{i+1}, \tilde a_{i+1}^{(d)}\bigr).
\]
The policy at test time likewise chooses the highest-Q candidate among cVAE-generated samples:
\[
\pi(s) = \argmax_{i\in\{1,\dots,m\}} Q_\varphi\!\bigl(s,\tilde a_i\bigr), \quad \{\tilde a_i\}_{i=1}^{m} \sim f_{\theta}(s,\bar{s},z).
\]
The “frictional” constraint therefore acts through the generative action space rather than through an explicit penalty in the critic loss [2509.19771].

A defining element is the heterogeneous buffer of orthonormal actions. For a buffer action \(a\in\mathcal{A}\subset\mathbb{R}^n\), the paper states that orthonormal vectors \(v_1,\dots,v_{k-1}\) exist under \(n\ge 2\) and \(\mathrm{rank}(\mathcal{A})=k\). These orthogonal actions are treated as background data in the cVAE objective, while buffer actions are target data. The cVAE separates salient latent structure associated with buffer-supported actions from nuisance structure associated with orthogonal actions, with a total-correlation term used to enforce independence [2509.19771].

The paper’s main theoretical statement, Theorem 5, gives an extrapolation-error expression comparing true dynamics \(p_0\) with orthogonal dynamics \(p_\theta(\cdot|s,v)\), and Remark 1 provides a bound
\[
\|\mathcal{E}_\rho\|_\infty \le \frac{2\,\sup_{s,a}\mathrm{TV}\!\big(p_0(s'|s,a),\,p_\rho(s'|s,a)\big)}{(1-\gamma)^2}\,R_{\max},
\]
together with an angular constraint
\[
\theta \le \arctan\frac{\sup_{s,a}\mathrm{TV}\!\big(p_0(s'|s,a),\,p_\theta(s'|s,a)\big)}{\sup_{s,a}\mathrm{TV}\!\big(p_0(s'|s,a),\,p_\rho(s'|s,a)\big)} \le \frac{\pi}{4}.
\]
In the paper’s interpretation, this angle quantifies the mismatch between replay-buffer dynamics and orthogonal dynamics, and the frictional constraint controls extrapolation through that geometry [2509.19771].

Empirical evaluation is conducted on MuJoCo tasks via Gymnasium. The reported baselines are SAC, MEOW, TD3, and DDPG. FQL achieves state-of-the-art performance on Walker2D-v4 and Humanoid-v4 across the paper’s Step, Seed, and Final metrics, while underperforming some baselines on lower-dimensional Hopper-v4. Selected reported results include Humanoid-v4 Step \(6486.43 \pm 144.90\) and Walker2D-v4 Seed \(5810.95 \pm 165.41\), both marked as best among listed methods [2509.19771]. The paper also notes hyperparameter sensitivity, especially to the cVAE KL weight \(\beta\), and additional computational cost from cVAE training and orthonormal augmentation.

## 6. Related value-based settings: nonsmooth frictional mechanics and total-cost games

A broader research context shows that frictional Q-learning also arises when Q-critics are deployed in friction-dominated dynamics, even when the method is not called by that name. "Reinforcement learning-based adaptive time-integration for nonsmooth dynamics" [2501.08934] applies Truncated Quantile Critics (TQC), an off-policy actor-critic algorithm with distributional Q-critics, to choose time steps for a seismic fault with Coulomb friction. The underlying dynamics include the linear momentum equation
\[
M \dot{v} = -F_r + F_{int},
\]
a slip-weakening friction law
\[
\mu(\delta) = \mu_d + \Delta \mu \, e^{-\delta / d_c},
\]
and a reward that balances large time steps, low runtime, and low local integration error. The paper reports up to a fourfold speed-up and describes the method as a concrete example of what one might call “frictional Q-learning,” even though the algorithm is TQC rather than tabular Q-learning [2501.08934].

In that work, the critic learns which time-step actions are valuable under different frictional regimes summarized by normalized internal energy, normalized slip velocity, solver iteration error, relative local error, normalized current time step, and a convergence indicator. The learned policy differs from a heuristic adaptive integrator by maintaining smaller, steadier time steps around slip events and increasing them during sticking phases. This suggests a distinct interpretation of frictional Q-learning: Q-value learning over actions that control the numerical treatment of a frictional physical system rather than the physical friction law itself [2501.08934].

At a more foundational level, "Stochastic Shortest Path Games and Q-Learning" [1412.8570] establishes convergence of classical asynchronous Q-learning for a broad class of undiscounted total-cost two-player zero-sum stochastic games with a cost-free termination state. The paper does not use the word “frictional” as an algorithmic label, but its structure is directly relevant to termination-based environments where costs accumulate until some resolution. It defines a Q-DP operator
\[
(FQ)(i,u,v) = g(i,u,v) + \sum_{j\in S} p_{ij}(u,v)\,\inf_{\sigma\in\bar U(j)} \sup_{\tau\in\bar V(j)} \sum_{u'\in U(j)}\sum_{v'\in V(j)}\sigma(u')\tau(v')Q(j,u',v'),
\]
and proves boundedness of Q-learning iterates under proposed SSP model conditions, thereby establishing convergence for the finite-space case [1412.8570].

The significance of that result is methodological. In discounted problems, convergence typically relies on contraction. In SSP games, convergence is obtained instead through termination structure, proper policies, uniqueness of the dynamic-programming solution, and boundedness of iterates. A plausible implication is that some environments described informally as “frictional”—because costs persist until an absorbing resolution—are better analyzed through SSP theory than through any explicit regularizer or bias term [1412.8570].

## 7. Limitations, distinctions, and open directions

Across these works, the principal limitation is conceptual heterogeneity. In one line of work, friction is an additive policy bias chosen in a meta-game over automata [2304.12647]. In another, it is an external training brake based on entropy of the Q-table [2006.14795]. In robotics, it is a physical coefficient that enters the state and is estimated online [2502.17221]. In FQL proper, it is a geometric analogy for extrapolation error that yields a constraint against unsupported actions [2509.19771]. These are related by function rather than by formal identity.

Each formulation also has problem-specific restrictions. The collusion framework optimizes only the bias \(b_i\) while keeping \(\alpha\) and \(\varepsilon\) fixed, uses memoryless action-based Q-values, and identifies equilibria empirically via simulation rather than by general theorems [2304.12647]. DE-QT is demonstrated in a single flag-collection gridworld, adds histogram-based computational overhead, and does not provide formal optimal-stopping guarantees [2006.14795]. The sliding-manipulation framework relies on Coulomb friction as an approximation, is sensitive to noisy friction estimates, and uses DDPG rather than classical tabular Q-learning [2502.17221]. FQL depends on continuous action spaces of dimension at least two, requires cVAE tuning, and can underperform in low-dimensional tasks such as Hopper-v4 [2509.19771]. The seismic time-integration application required transfer learning for robust convergence on finer meshes [2501.08934].

A second important distinction concerns where friction enters the learning system. It may modify policy selection while leaving TD backup untouched, as in biased Q-based automata; act only on training duration, as in DE-QT; parameterize the critic’s state, as in friction-aware manipulation; or reshape the admissible action manifold, as in FQL. Conflating these mechanisms obscures the substantive differences between regularization, state augmentation, and support constraint.

Open directions are stated or implied in several papers. "Learned Collusion" notes that agents might also adjust \(\alpha\) or \(\varepsilon\) strategically, making learning rate or exploration another friction dimension [2304.12647]. The DE-QT paper suggests future work on entropy-driven modulation of learning rate, exploration, or direct regularization of Q-values [2006.14795]. The sliding-manipulation paper points to richer friction models beyond ideal Coulomb behavior and to more complex non-prehensile tasks [2502.17221]. FQL identifies stabilization of the state-conditioned generative model and extensions to purely offline datasets or alternative heterogeneous-buffer constructions as future directions [2509.19771].

Taken together, the literature supports a precise but plural definition: frictional Q-learning is not one algorithm but a class of value-based strategies that introduce resistance against failure modes of naïve Q-learning—myopic best-response to mismeasured continuation values, information-loss through prolonged training, friction-unaware action valuation in physical control, or extrapolation outside replay support. The named algorithm "Frictional Q-Learning" [2509.19771] is the most specific instantiation, but the surrounding literature shows that the broader idea spans economic games, tabular diagnostics, robotic manipulation, nonsmooth numerical mechanics, and undiscounted dynamic games.

Source: https://www.emergentmind.com/topics/frictional-q-learning