Fuzzy Q Reinforcement Learning

Updated 8 July 2026

Fuzzy Q Reinforcement Learning is a method combining Q-learning with fuzzy inference to manage continuous and uncertain state conditions.
It leverages fuzzy rule bases and weighted aggregation to interpolate local actions and distribute credit in temporal-difference updates.
Applications in traffic control, cloud auto-scaling, and energy management show its potential to optimize performance and reduce operational delays.

Fuzzy Q Reinforcement Learning, usually called Fuzzy Q-learning (FQL), denotes a family of model-free reinforcement-learning methods that combine Q-learning with fuzzy inference systems so that continuous, uncertain, or linguistically described operating conditions can be controlled without relying on a purely crisp state-action table. In the canonical formulation represented across the supplied literature, fuzzy sets provide state abstraction, rule firing strengths distribute credit and blame, rule-action pairs carry learned values, and the executed control is obtained by weighted aggregation of local actions rather than by a single discrete lookup (Jamshidi et al., 2015, Arabnejad et al., 2017, Guo et al., 2023). The label is also used more broadly for hybrid methods in which fuzzy logic only shapes rewards or neural representations around Q-learning, so the term spans both strict and looser usages in recent work (Nguyen et al., 2024, Ghoreishee et al., 6 Apr 2026).

1. Canonical representation and fuzzy inference structure

A recurring structural pattern in classical FQL is the replacement of a crisp tabular state by a fuzzy rule base. In the traffic-control formulation, the generic rule template is written as

$\begin{eqnarray} If\ x\ is\ s_i\ &then&\ a[i,1]\ with\ q[i,1] \nonumber\ &or&\ a[i,2]\ with\ q[i,2] \nonumber\ &\vdots&\ &or&\ a[i,j]\ with\ q[i,j]\nonumber \end{eqnarray}$

so each fuzzy antecedent $s_i$ acts as a local state region, each rule contains multiple candidate actions $a[i,j]$ , and each rule-action pair has an associated learned value $q[i,j]$ (Daeichian et al., 2019). This rule-local parameterization is the defining departure from ordinary crisp $Q(s,a)$ tables.

The inference layer converts a continuous observation into degrees of rule activation and then blends rule-local decisions into a global action. In cloud auto-scaling, the controller output is expressed as a weighted combination of rule consequents,

$a=\sum_{i=1}^{N}\mu_i(x)\,a_i,$

while the FCHEV energy-management formulation uses normalized weighted averaging,

$a(s)=\frac{\sum_i a_i \phi_i(s)}{\sum_i \phi_i(s)}.$

Both equations implement the same idea: local rule actions are discrete or low-cardinality, but the executed control can be continuous because multiple rules fire simultaneously (Arabnejad et al., 2017, Guo et al., 2023).

The fuzzy layer itself is not uniform across papers. The traffic controller explicitly uses a Mamdani FIS, whereas the adaptive lighting controller uses a zero-order Takagi–Sugeno system with Gaussian membership functions and singleton consequents $k_j$ (Daeichian et al., 2019, Vashishtha et al., 2023). This variation does not alter the core FQL idea that fuzzy memberships provide structured interpolation over continuous operating conditions.

2. Learning equations and temporal-difference credit assignment

The canonical FQL update in the cloud-controller literature expresses the global fuzzy action-value as a weighted sum of rule-action entries,

$Q(s(t),a)=\sum_{i=1}^N \alpha_i(s)\, q[i,a_i],$

estimates the next-state value by rule-wise maximization,

$V(s(t+1))=\sum_{i=1}^N \alpha_i(s(t+1))\max_k q[i,k],$

forms a temporal-difference error,

$s_i$ 0

and updates each fired rule-action entry proportionally to its firing level,

$s_i$ 1

This is the basic mechanism by which continuous states are handled without abandoning value-based RL (Jamshidi et al., 2015).

The OpenStack auto-scaling comparison makes the off-policy character of FQL explicit. There,

$s_i$ 2

with

$s_i$ 3

The contrasting Fuzzy SARSA Learning (FSL) update uses the actually selected next action instead. The paper therefore treats FQL as the off-policy member of a pair of fuzzy RL methods sharing the same fuzzy rule base, $s_i$ 4-greedy exploration, and firing-degree-weighted updates (Arabnejad et al., 2017).

A recent extension, Enhanced-FQL( $s_i$ 5), generalizes one-step FQL by defining a fuzzified Bellman value

$s_i$ 6

a fuzzified activation matrix

$s_i$ 7

eligibility traces

$s_i$ 8

and the trace-weighted update

$s_i$ 9

That work also introduces Segmented Experience Replay and states a convergence result under bounded rewards, Robbins–Monro step sizes, persistent excitation, and related standard assumptions (Jalaeian-Farimani, 7 Jan 2026).

3. Architectural extensions and problem-specific reformulations

Classical FQL has been extended in several orthogonal directions. In multi-intersection traffic control, each intersection is an autonomous agent in a “classical non-stationary environment,” and the paper modifies the Q-update to include neighboring rewards weighted by a fuzzy function $a[i,j]$ 0: $a[i,j]$ 1 The resulting method is best characterized as a multi-agent, game-theoretically inspired FQL update rather than purely local FQL (Daeichian et al., 2019).

In quadrotor control, FQL is used as a supervisory tuner rather than as a direct control law. The plant is first feedback-linearized into an NI model, then FQL adapts the SNI controller parameters $a[i,j]$ 2 and $a[i,j]$ 3 in

$a[i,j]$ 4

with reward defined from step-to-step improvement in tracking error. The RL component therefore tunes a structured controller online instead of producing thrust and moments directly (Tran et al., 2022).

Other papers reformulate FQL still more aggressively. The pursuit-evasion work replaces scalar rule-action values with sets of non-dominated three-objective vectors, defines a generalized Bellman relation

$a[i,j]$ 5

and uses a simplified three-dimensional hypervolume

$a[i,j]$ 6

to guide action selection and Pareto-front sampling (Hu et al., 9 Mar 2025). The adaptive lighting study, by contrast, is closer to a Q-learning Fuzzy Inference System: user corrections are converted into rewards, and Q-learning updates the Takagi–Sugeno consequent constants $a[i,j]$ 7 and Gaussian means $a[i,j]$ 8 rather than learning a purely canonical rule-action value table (Vashishtha et al., 2023).

4. Application domains and reported empirical behavior

The supplied literature applies FQL to resource provisioning, traffic control, flight control, vehicle energy management, and robust gain tuning. Reported outcomes are strongly task-specific.

Domain	FQL instantiation	Reported result
Cloud auto-scaling	FQL4KE learns rule consequents online for 9 fuzzy states and actions $a[i,j]$ 9	In “Big spike,” strategy $q[i,j]$ 0 reported $q[i,j]$ 1 ms and $q[i,j]$ 2, versus static fuzzy $q[i,j]$ 3 ms and $q[i,j]$ 4 VMs (Jamshidi et al., 2015)
OpenStack auto-scaling	Off-policy FQL compared with FSL	Average maximum VM usage was reported as $q[i,j]$ 5 for FQL and $q[i,j]$ 6 for FSL; FQL was superior for the Variations workload pattern but significantly worse than FSL for Predictable Burst (Arabnejad et al., 2017)
Traffic signals	Multi-agent FQL with neighbor-aware reward coupling	Total average delay decreased from more than 50 sec for fixed-time scheduling to approximately 15 sec for the proposed method (Daeichian et al., 2019)
FCHEV energy management	35-rule, 8-action FQL with startup penalty in the episode objective	Training time was reported as 15 min for the FQL-based EMS and 17 min for the degradation-aware variant, with fuel-cell starts significantly reduced (Guo et al., 2023)
Aircraft auto-landing	Grid-based FQL with Gaussian fuzzy interpolation and continuous elevator blending	Under noise + disturbance, FQL reported $q[i,j]$ 7 deg and $q[i,j]$ 8 m, versus Q-learning $q[i,j]$ 9 deg and $Q(s,a)$ 0 m (Zahmatkesh et al., 2023)
Quadrotor tracking	FQL tunes $Q(s,a)$ 1 of an SNI controller	Under disturbances and uncertainties, the Fuzzy-QL-SNI controller reported lower RMSE than SNI and PID in several channels (Tran et al., 2022)

These results are not directly commensurate because the reward definitions, controller roles, and success criteria differ markedly across domains. This suggests caution in treating any single positive empirical result as a domain-independent statement about FQL itself.

5. Reproducibility, scaling, and recurrent limitations

Several implementation limitations recur across the literature. The OpenStack comparison states that the reward is tied to SLO violations, response time, and VM usage but does not provide an explicit closed-form reward equation; it also notes that the exact action-set notation is partially corrupted and that membership breakpoints are shown mainly in figures (Arabnejad et al., 2017). The traffic-control paper likewise leaves exact numerical membership-function coordinates, fuzzy rule tables, and the exploration policy unspecified, even though it reports learning parameters $Q(s,a)$ 2 and $Q(s,a)$ 3 (Daeichian et al., 2019). In the FCHEV work, the reward sign for the startup penalty is ambiguous in print, antecedent membership functions are fixed rather than learned, and the method is compared mainly against a basic FQL variant rather than a wide range of modern RL baselines (Guo et al., 2023). The aircraft auto-landing study reports the state grid, action discretization, and reward shaping, but does not give the Gaussian widths $Q(s,a)$ 4 and $Q(s,a)$ 5 that determine the fuzzy interpolation layer (Zahmatkesh et al., 2023).