TQL: RL, Query & Quantum Constructs

• TQL is an overloaded acronym that defines distinct constructs in reinforcement learning, data query languages, temporal logics, and topological quantum phases.
• Studies show that TQL variants drive advances through specialized methods like transfer mechanisms, type-driven querying, and entropy control in transformer-based models.
• Empirical and theoretical analyses demonstrate TQL’s impact with faster convergence in RL, enhanced query expressiveness, and robust topological order in condensed-matter physics.

TQL is an overloaded acronym that denotes several unrelated technical constructs across contemporary research. In reinforcement learning, it names a family of Q-learning methods that includes Tabular Q-learning, Transfer Q-learning, Trajectory Q-Learning, Transformer Q-Learning, and Transition Transfer Q-Learning; in data systems and knowledge representation, it denotes both a Type-driven Query Language for data discovery and a Temporal Description Logic extending OWL 2 QL; and in condensed-matter physics it abbreviates topological quantum liquid, including a toric-code phase proposed for Rydberg-atom arrays (Elsayed et al., 2020, Zhou et al., 2023, Dong et al., 1 Feb 2026, Kang et al., 11 Aug 2025, Artale et al., 2013, Verresen et al., 2020).

1. Acronymic scope and principal senses

The main technical uses of TQL in the cited literature are structurally distinct rather than variations of a single framework. In some papers TQL is an RL algorithm or algorithm family; in others it is a formal query language; in another it is a phase of matter. This terminological overloading is a recurrent source of ambiguity in citation, indexing, and cross-disciplinary reading.

Sense of TQL	Domain	Representative use
Transfer/Tabular/Trajectory/Transformer/Transition Transfer Q-Learning	Reinforcement learning	Control, transfer, risk-sensitive RL, pricing, offline RL
Type-driven Query Language	Data discovery	Search and transformation over unknown datasets
Temporal Description Logic	OBDA and knowledge representation	Temporalized OWL 2 QL with FO-rewritability
Topological quantum liquid	Condensed-matter physics	$\mathbb{Z}_2$ topological order in Rydberg systems

A common misconception is that “TQL” in machine learning papers always refers to one specific algorithm. The record is more heterogeneous: “TQL” can mean a baseline tabular method in algorithmic pricing, a transfer mechanism in 5G radio resource management, a history-dependent risk-sensitive RL algorithm, or a transformer-based value-learning method (Deng et al., 14 Mar 2025, Elsayed et al., 2020, Zhou et al., 2023, Dong et al., 1 Feb 2026).

2. TQL in classical reinforcement learning: tabular and transfer forms

In wireless communications, “Transfer Q-learning (TQL)” was proposed for interference mitigation in 5G-NR mm-Wave networks using beamforming and NOMA, with the objective of maximizing aggregate network capacity through joint user-cell association and selection of the number of beams (Elsayed et al., 2020). The source task is a simpler user-cell association problem solved offline by an expert agent, while the target task augments this with beam-number selection for a learner agent. The state is thresholded by the average SINR of associated users, the learner action jointly specifies user association and the number of beams, and the reward is a sigmoid function of average SINR. The transfer mechanism is formulated through inter-task mappings of states, actions, and Q-values, with action mapping relating intra-beam interference in the learner to intra-cell interference in the expert and inter-beam interference in the learner to inter-cell interference in the expert (Elsayed et al., 2020).

The reported performance emphasizes faster adaptation under mobility. TQL reaches convergence in 4245 TTIs, corresponding to a 29% reduction versus standard Q-learning, and under mobility TQL and Q-learning outperform BSDC by about 12% in aggregate sum rate at the highest traffic loads. The same summary also reports lower packet loss rates across all loads and runtime complexity $O(N)$ with space complexity $O(K2^N)$ for Q-learning and TQL. For stationary deployments, the reporting is less uniform: the abstract states that Q-learning and BSDC outperform TQL, whereas the detailed summary also reports a 23% improvement of TQL over Q-learning at the highest load and BSDC performance close to TQL. This suggests that stationary-scenario ranking depends on the specific operating point and comparison emphasized (Elsayed et al., 2020).

In algorithmic pricing, “TQL” typically means Tabular Q-learning. In repeated Bertrand oligopoly simulations, TQL was found to be strongly associated with supra-competitive pricing, elevated price dispersion, instability, and sensitivity to state representation (Deng et al., 14 Mar 2025). In the Logit Bertrand model, normalized profit $\Delta$ for TQL in homogeneous settings clustered around 0.6, while in Standard and Edgeworth Bertrand models it hovered around 0.5. With full information and memory length $l=1$, normalized price RPDI for TQL was around 0.48 in Logit Bertrand; when agents observed only their own history, RPDI fell below 0.05, approaching competitive benchmarks (Deng et al., 14 Mar 2025). The same study reports that agents with higher learning rates consistently achieve higher profits; for $\alpha_0=0.5$ versus $\alpha_1=0.01$, the fast agent’s normalized profit could be nearly twice that of the slow learner (Deng et al., 14 Mar 2025).

A related 2024 study on dynamic pricing likewise reported that TQL exhibits higher collusion and price dispersion than DRL algorithms, and in its duopoly settings TQL never yielded competitive pricing in Standard Bertrand, Edgeworth, or Logit cases (Deng et al., 2024). Taken together, these results position tabular TQL as a particularly strong collusion-inducing baseline in pricing experiments, while also showing that its outcomes are highly contingent on information structure, exploration dynamics, and market model (Deng et al., 14 Mar 2025, Deng et al., 2024).

3. Risk-sensitive and structural transfer variants of Q-learning

“Trajectory Q-Learning (TQL)” was introduced to address a foundational problem in risk-sensitive reinforcement learning. The paper proves that existing RSRL methods based on distributional Bellman operators do not achieve unbiased optimization and cannot guarantee optimality or even improvements with respect to risk measures over accumulated return distributions (Zhou et al., 2023). The proposed remedy is to optimize over history-dependent value distributions rather than purely Markovian state-action value distributions. TQL therefore introduces a history-action value distribution and a history-relied operator, making the objective explicitly trajectory-based. Within this framework, the algorithm supports general distortion risk measures, including mean, CVaR, Wang, CPW, POW, and expectile, and the authors state provable convergence to the optimal policy (Zhou et al., 2023).

Theoretical guarantees are central to this version of TQL. Policy evaluation under the history-relied operator is stated to be a $\gamma$-contraction in the maximum $p$-Wasserstein metric, and policy improvement is monotonic in the risk measure. Empirically, the method is evaluated in tabular, MiniGrid, and MountainCar settings with risky penalties, where it is reported to achieve better performance toward risk-sensitive objectives than distributional RL baselines (Zhou et al., 2023).

A different transfer-oriented formalization appears in “Transition Transfer $Q$-Learning for Composite Markov Decision Processes,” where UCB-TQL is designed for multi-task transfer RL with provable guarantees (Chai et al., 1 Feb 2025). The composite MDP framework models transitions as

$O(N)$0

with a low-rank shared component and a sparse task-specific component. For a source task and a target task,

$O(N)$1

and the task difference $O(N)$2 is assumed sparse with $O(N)$3 (Chai et al., 1 Feb 2025).

Under this structure, UCB-TQL achieves a regret bound

$O(N)$4

where $O(N)$5 is the number of target-task trajectories and $O(N)$6 quantifies the sparse differences between tasks. The paper emphasizes that this scaling is independent of the ambient dimension and therefore improves substantially over single-task RL when $O(N)$7 (Chai et al., 1 Feb 2025). Relative to earlier empirical transfer-Q formulations, this version of TQL is explicitly a theory-driven construction for exploiting shared low-rank dynamics while adapting along sparse dimensions.

4. Transformer Q-Learning and the scaling of value functions

“Transformer Q-Learning (TQL)” addresses a distinct problem: the failure of naively scaled transformer critics in offline RL (Dong et al., 1 Feb 2026). The paper identifies attention-score collapse as the critical failure mode. As model capacity increases, attention matrices become increasingly peaked, value-function landscapes become non-smooth, and policy performance degrades. The proposed solution is to control the entropy of attention scores in each transformer layer, thereby preventing collapse and stabilizing training (Dong et al., 1 Feb 2026).

The architecture represents state and action using one token per scalar feature and prepends a special [VALUE] token whose final embedding predicts the Q-value. TQL adds entropy regularization and entropy-target control to the critic loss, with learnable per-layer temperature coefficients and a special treatment for the [VALUE] token. The stated implementation does not require architectural changes and is compatible with pre-trained transformer models (Dong et al., 1 Feb 2026).

The empirical evaluation is conducted on OGBench, comprising 25 robotic and puzzle-solving tasks, with transformer critics ranging from 0.4M to 26M parameters. Without TQL, scaling from the smallest to the largest model leads to performance degradation of up to 10.6% on average; with TQL, scaling yields up to a 43% average improvement. The paper further reports that TQL achieves the highest scores on 4 out of 5 domains and on the mean across all 25 tasks, outperforming MLP, flow, and transformer baselines as well as methods such as FQL, floq, Q-Transformer, and PAC (Dong et al., 1 Feb 2026). In this usage, TQL is not a tabular method at all, but a scaling law intervention for transformer-based value learning.

5. TQL as a type-driven query language for data discovery

In data discovery, “TQL” denotes a Type-driven Query Language intended to re-prioritize the client through a language-driven approach rather than a system-driven or model-driven one (Kang et al., 11 Aug 2025). The language is designed to unify search and transformation: unlike SRQL, which is discovery-oriented, and SQL, which is retrieval-oriented, TQL can express both discovery constraints and downstream transformation context. Its core mechanism is a type-like system in which signatures act analogously to types and constrain collections of candidate datasets (Kang et al., 11 Aug 2025).

The syntax is SQL-like and declarative, with support for signatures such as SRC[str], COL[str], COL*[str], SIML[id], PFKEY[id], FORALL[pd], and EXISTS[pd]. Queries can combine discovery predicates with relational operations such as SELECT, JOIN, FILTER, UNION, DIFF, and PROD. The accompanying formal model is type-collected relational algebra, which generalizes relational algebra from explicit tables to collections of possible datasets. The paper states a relational-completeness theorem: type-collected relational algebra can express all operations of classic relational algebra, but over sets of as-yet undetermined datasets (Kang et al., 11 Aug 2025).

A complementary systems paper presents TQL through the algebraic core “Imperative Relational Algebra with Types” (ImpRAT) and a modular proof-of-concept implementation (Kang et al., 11 Aug 2025). ImpRAT extends relational algebra with types and imperative statements such as assignments, declarations of free variables, sequencing, and return. The prototype is implemented in Python, uses PLY for parsing, and employs a backtracking choice function with feedback from failed attempts to search over candidate dataset assignments (Kang et al., 11 Aug 2025). This systems view makes explicit that TQL is not merely a notation for querying known tables; it is also a search language over repositories of partially known or unknown datasets.

The practical expressiveness of TQL can be summarized by the task archetypes explicitly compared in the paper:

Task archetype	SQL	SRQL	TQL
Constraint Search	✗	✓	✓
Data Composition	✓	✗	✓
Combined Queries	✗	✗	✓

The papers also highlight ongoing work on type inference. The goal is to infer maximally-restrictive signatures for input collections from the types of composed collections, drawing inspiration from the Curry-Howard correspondence, though the mapping from output signatures to input constraints is not always unique (Kang et al., 11 Aug 2025). A plausible implication is that future TQL systems will rely increasingly on static inference to prune discovery search spaces before execution.

6. TQL as temporal description logic for ontology-based data access

A separate use of TQL appears in knowledge representation, where it denotes a Temporal Description Logic designed for ontology-based data access over temporal data with validity time (Artale et al., 2013). This TQL extends OWL 2 QL with temporal operators while preserving first-order rewritability of conjunctive queries for suitably defined temporal data instances. The syntax introduces temporalized concepts and roles using past and future modalities, with grammar of the form

$O(N)$8

$O(N)$9

$O(K2^N)$0

$O(K2^N)$1

Only the left-hand sides of TBox inclusions may contain temporal operators, a restriction introduced specifically to ensure FO-rewritability (Artale et al., 2013).

Semantically, the logic is interpreted over sequences of standard DL structures indexed by time points in $O(K2^N)$2, with a common domain and stable interpretations of individual constants. ABox facts are time-stamped assertions such as $O(K2^N)$3 and $O(K2^N)$4, thereby encoding validity time directly in the data (Artale et al., 2013). The language supports temporal conceptual modeling patterns such as rigidity, persistence, instantaneity, and convexity. It is therefore a modeling language as well as a query-rewriting formalism.

The main formal result states that for a TQL TBox $O(K2^N)$5 and a conjunctive query $O(K2^N)$6, one can construct a union of conjunctive queries $O(K2^N)$7 such that query answering over $O(K2^N)$8 is equivalent to evaluating $O(K2^N)$9 over an explicit temporal data instance $\Delta$0 containing both the ABox atoms and the time-order relation. Query answering is thus FO-rewritable and in $\Delta$1 with respect to data complexity (Artale et al., 2013). The paper also marks a boundary of tractability: adding right-hand temporal operators, exemplified by a TBox containing $\Delta$2, destroys FO-rewritability and makes query answering coNP-hard (Artale et al., 2013). In this sense, TQL is presented as a maximal temporal extension of OWL 2 QL that still retains database-oriented rewritability guarantees.

7. TQL as topological quantum liquid

In condensed-matter physics, TQL abbreviates “topological quantum liquid” or “topological quantum liquid phase” rather than a learning or query formalism (Verresen et al., 2020). A prominent instance is the proposal that toric-code $\Delta$3 topological order can be realized in a two-dimensional array of Rydberg atoms on the ruby lattice at specific blockade radii. The effective Hamiltonian is

$\Delta$4

with the blockade model mapping to a monomer-dimer model on the kagome lattice and admitting an interpretation as an odd $\Delta$5 gauge theory whose dynamics is generated by monomer fluctuations (Verresen et al., 2020).

The phase diagram was obtained using DMRG on cylinder geometries, and the intermediate phase was identified as a TQL through multiple diagnostics. The evidence listed in the paper includes a continuous transition between two featureless phases, topological entanglement entropy $\Delta$6, degenerate topological ground states, and the expected modular matrix from ground-state overlap (Verresen et al., 2020). The topological phase is also reported to persist when idealized blockade interactions are replaced by realistic algebraically decaying van der Waals interactions $\Delta$7 (Verresen et al., 2020).

The work further constructs topological loop operators, including the Fredenhagen-Marcu order parameter, and proposes an experimental dynamic protocol to measure them. It also describes trapping an emergent anyon, realizing different topological boundary conditions, and discusses implications for fault-tolerant quantum memories (Verresen et al., 2020). Here, TQL is a many-body phase characterized by deconfinement and nontrivial topological response, not an algorithmic abbreviation. The contrast with the RL and query-language senses of TQL is complete: the shared acronym masks entirely different mathematical objects, observables, and research agendas.