---
title: Fuzzy Q Reinforcement Learning
url: https://www.emergentmind.com/topics/fuzzy-q-reinforcement-learning-fql
type: topic
---

# Fuzzy Q Reinforcement Learning

Fuzzy Q Reinforcement Learning, usually called Fuzzy Q-learning (FQL), denotes a family of model-free reinforcement-learning methods that combine Q-learning with fuzzy inference systems so that continuous, uncertain, or linguistically described operating conditions can be controlled without relying on a purely crisp state-action table. In the canonical formulation represented across the supplied literature, fuzzy sets provide state abstraction, rule firing strengths distribute credit and blame, rule-action pairs carry learned values, and the executed control is obtained by weighted aggregation of local actions rather than by a single discrete lookup [1507.00567] [1705.07114] [2302.06236]. The label is also used more broadly for hybrid methods in which fuzzy logic only shapes rewards or neural representations around Q-learning, so the term spans both strict and looser usages in recent work [2405.01057] [2604.16436].

## 1. Canonical representation and fuzzy inference structure

A recurring structural pattern in classical FQL is the replacement of a crisp tabular state by a fuzzy rule base. In the traffic-control formulation, the generic rule template is written as
\[
\begin{eqnarray}
If\  x\  is\  s_i\ &then&\  a[i,1]\  with\  q[i,1] \nonumber\\
&or&\    a[i,2]\  with\  q[i,2] \nonumber\\
&\vdots&\\
&or&\    a[i,j]\   with\  q[i,j]\nonumber
\end{eqnarray}
\]
so each fuzzy antecedent \(s_i\) acts as a local state region, each rule contains multiple candidate actions \(a[i,j]\), and each rule-action pair has an associated learned value \(q[i,j]\) [1905.01361]. This rule-local parameterization is the defining departure from ordinary crisp \(Q(s,a)\) tables.

The inference layer converts a continuous observation into degrees of rule activation and then blends rule-local decisions into a global action. In cloud auto-scaling, the controller output is expressed as a weighted combination of rule consequents,
\[
a=\sum_{i=1}^{N}\mu_i(x)\,a_i,
\]
while the FCHEV energy-management formulation uses normalized weighted averaging,
\[
a(s)=\frac{\sum_i a_i \phi_i(s)}{\sum_i \phi_i(s)}.
\]
Both equations implement the same idea: local rule actions are discrete or low-cardinality, but the executed control can be continuous because multiple rules fire simultaneously [1705.07114] [2302.06236].

The fuzzy layer itself is not uniform across papers. The traffic controller explicitly uses a Mamdani FIS, whereas the adaptive lighting controller uses a zero-order Takagi–Sugeno system with Gaussian membership functions and singleton consequents \(k_j\) [1905.01361] [2310.00525]. This variation does not alter the core FQL idea that fuzzy memberships provide structured interpolation over continuous operating conditions.

## 2. Learning equations and temporal-difference credit assignment

The canonical FQL update in the cloud-controller literature expresses the global fuzzy action-value as a weighted sum of rule-action entries,
\[
Q(s(t),a)=\sum_{i=1}^N \alpha_i(s)\, q[i,a_i],
\]
estimates the next-state value by rule-wise maximization,
\[
V(s(t+1))=\sum_{i=1}^N \alpha_i(s(t+1))\max_k q[i,k],
\]
forms a temporal-difference error,
\[
\Delta Q=r(t+1)+\gamma V(s(t+1))-Q(s(t),a),
\]
and updates each fired rule-action entry proportionally to its firing level,
\[
q[i,a_i]=q[i,a_i]+\eta \cdot \Delta Q \cdot \alpha_i(s(t)).
\]
This is the basic mechanism by which continuous states are handled without abandoning value-based RL [1507.00567].

The OpenStack auto-scaling comparison makes the off-policy character of FQL explicit. There,
\[
\Delta Q_{FQL}(s,a) = r + \gamma \times V(s') - Q(s,a),
\]
with
\[
V(s')= \sum_{i=1}^{N} \mu_i(s') \max_k(q[i,a_k]).
\]
The contrasting Fuzzy SARSA Learning (FSL) update uses the actually selected next action instead. The paper therefore treats FQL as the off-policy member of a pair of fuzzy RL methods sharing the same fuzzy rule base, \(\epsilon\)-greedy exploration, and firing-degree-weighted updates [1705.07114].

A recent extension, Enhanced-FQL(\(\lambda\)), generalizes one-step FQL by defining a fuzzified Bellman value
\[
\Upsilon(s')=\sum_{i=1}^{N_s} w_i(s') \max_j \widehat Q_{i,j},
\]
a fuzzified activation matrix
\[
\zeta(s,a)=\mu_S(s)\mu_A(a)^\top,
\]
eligibility traces
\[
E_{i,j}(t)=\min\left\{\gamma\lambda E_{i,j}(t-1)+\zeta_{i,j}(s_t,a_t),\,1\right\},
\]
and the trace-weighted update
\[
\widehat Q_{i,j}(t+1)=\widehat Q_{i,j}(t)+\alpha E_{i,j}(t)\delta_{i,j}(t).
\]
That work also introduces Segmented Experience Replay and states a convergence result under bounded rewards, Robbins–Monro step sizes, persistent excitation, and related standard assumptions [2601.04392].

## 3. Architectural extensions and problem-specific reformulations

Classical FQL has been extended in several orthogonal directions. In multi-intersection traffic control, each intersection is an autonomous agent in a “classical non-stationary environment,” and the paper modifies the Q-update to include neighboring rewards weighted by a fuzzy function \(f(i,j)\):
\[
Q_i(s_i,a_i )=(1-\alpha_i)Q_i(s_i,a_i )+\alpha_i [r_i(s_i,a_i)+ \sum_{j=1,j\neq i}^n f(i,j)r_j(s_i,a_i )+ \gamma \max\left(Q_i(s_i',a_i')-Q_i(s_i,a_i)\right)].
\]
The resulting method is best characterized as a multi-agent, game-theoretically inspired FQL update rather than purely local FQL [1905.01361].

In quadrotor control, FQL is used as a supervisory tuner rather than as a direct control law. The plant is first feedback-linearized into an NI model, then FQL adapts the SNI controller parameters \(\gamma\) and \(\tau\) in
\[
N(s)=\frac{\gamma}{\tau s+1}-\beta,
\]
with reward defined from step-to-step improvement in tracking error. The RL component therefore tunes a structured controller online instead of producing thrust and moments directly [2203.13959].

Other papers reformulate FQL still more aggressively. The pursuit-evasion work replaces scalar rule-action values with sets of non-dominated three-objective vectors, defines a generalized Bellman relation
\[
\hat Q(s,a) = \vec r(s,a) \oplus \gamma ND_t(s,a),
\]
and uses a simplified three-dimensional hypervolume
\[
{\cal H}_a^l = q_1^l q_2^l q_3^l
\]
to guide action selection and Pareto-front sampling [2503.06741]. The adaptive lighting study, by contrast, is closer to a Q-learning Fuzzy Inference System: user corrections are converted into rewards, and Q-learning updates the Takagi–Sugeno consequent constants \(k_j\) and Gaussian means \(m_i^j\) rather than learning a purely canonical rule-action value table [2310.00525].

## 4. Application domains and reported empirical behavior

The supplied literature applies FQL to resource provisioning, traffic control, flight control, vehicle energy management, and robust gain tuning. Reported outcomes are strongly task-specific.

| Domain | FQL instantiation | Reported result |
|---|---|---|
| Cloud auto-scaling | FQL4KE learns rule consequents online for 9 fuzzy states and actions \(\{-2,-1,0,+1,+2\}\) | In “Big spike,” strategy \(S1\) reported \(rt_{95\%}=1212\) ms and \(\overline{vm}=2.2\), versus static fuzzy \(1339\) ms and \(3.2\) VMs [1507.00567] |
| OpenStack auto-scaling | Off-policy FQL compared with FSL | Average maximum VM usage was reported as \(18.3\%\) for FQL and \(22.6\%\) for FSL; FQL was superior for the Variations workload pattern but significantly worse than FSL for Predictable Burst [1705.07114] |
| Traffic signals | Multi-agent FQL with neighbor-aware reward coupling | Total average delay decreased from more than 50 sec for fixed-time scheduling to approximately 15 sec for the proposed method [1905.01361] |
| FCHEV energy management | 35-rule, 8-action FQL with startup penalty in the episode objective | Training time was reported as 15 min for the FQL-based EMS and 17 min for the degradation-aware variant, with fuel-cell starts significantly reduced [2302.06236] |
| Aircraft auto-landing | Grid-based FQL with Gaussian fuzzy interpolation and continuous elevator blending | Under noise + disturbance, FQL reported \(TE_\theta=2.629\) deg and \(TE_h=2.17\) m, versus Q-learning \(11.178\) deg and \(21.655\) m [2302.10997] |
| Quadrotor tracking | FQL tunes \(\gamma,\tau\) of an SNI controller | Under disturbances and uncertainties, the Fuzzy-QL-SNI controller reported lower RMSE than SNI and PID in several channels [2203.13959] |

These results are not directly commensurate because the reward definitions, controller roles, and success criteria differ markedly across domains. This suggests caution in treating any single positive empirical result as a domain-independent statement about FQL itself.

## 5. Reproducibility, scaling, and recurrent limitations

Several implementation limitations recur across the literature. The OpenStack comparison states that the reward is tied to SLO violations, response time, and VM usage but does not provide an explicit closed-form reward equation; it also notes that the exact action-set notation is partially corrupted and that membership breakpoints are shown mainly in figures [1705.07114]. The traffic-control paper likewise leaves exact numerical membership-function coordinates, fuzzy rule tables, and the exploration policy unspecified, even though it reports learning parameters \(\alpha=0.5\) and \(\gamma=0.7\) [1905.01361]. In the FCHEV work, the reward sign for the startup penalty is ambiguous in print, antecedent membership functions are fixed rather than learned, and the method is compared mainly against a basic FQL variant rather than a wide range of modern RL baselines [2302.06236]. The aircraft auto-landing study reports the state grid, action discretization, and reward shaping, but does not give the Gaussian widths \(\sigma_\theta\) and \(\sigma_{\dot\theta}\) that determine the fuzzy interpolation layer [2302.10997].

A second limitation is scaling. In FQL4KE, the learned object is still a table \(q[i

Source: https://www.emergentmind.com/topics/fuzzy-q-reinforcement-learning-fql