Stratifying Reinforcement Learning with Signal Temporal Logic

Abstract: In this paper, we develop a stratification-based semantics for Signal Temporal Logic (STL) in which each atomic predicate is interpreted as a membership test in a stratified space. This perspective reveals a novel correspondence principle between stratification theory and STL, showing that most STL formulas can be viewed as inducing a stratification of space-time. The significance of this interpretation is twofold. First, it offers a fresh theoretical framework for analyzing the structure of the embedding space generated by deep reinforcement learning (DRL) and relates it to the geometry of the ambient decision space. Second, it provides a principled framework that both enables the reuse of existing high-dimensional analysis tools and motivates the creation of novel computational techniques. To ground the theory, we (1) illustrate the role of stratification theory in Minigrid games and (2) apply numerical techniques to the latent embeddings of a DRL agent playing such a game where the robustness of STL formulas is used as the reward. In the process, we propose computationally efficient signatures that, based on preliminary evidence, appear promising for uncovering the stratification structure of such embedding spaces.

• The paper introduces a stratified framework that maps STL semantics to poset-indexed state spaces in DRL, enhancing reward design.
• It employs novel numerical techniques (VGT and VGT-dot) to detect non-manifold and singular structures in high-dimensional neural representations.
• Empirical validation in Minigrid environments highlights improved policy generalization and inherent hourglass stratification in Transformer-based agents.

Stratification, Poset Semantics, and Signal Temporal Logic in Deep Reinforcement Learning

Introduction and Theoretical Constructs

This work develops a formal stratification-based framework for Signal Temporal Logic (STL) in the context of deep reinforcement learning (DRL), making significant theoretical and computational connections between poset-indexed stratified spaces, STL semantics, and the geometric properties of neural representations. The central claim is that STL formulas naturally induce stratifications of space-time, with atomic predicates interpreted as membership in semi-algebraic strata and STL robustness measures serving as task rewards. This stratified perspective both refines the ambient decision process and suggests new approaches for analyzing neural embedding spaces.

The authors argue that neural network latent spaces in DRL do not satisfy the manifold hypothesis but exhibit stratified structure, often inherited from the ambient task geometry and reward function. This thesis is systematically formalized by constructing poset-indexed stratification maps and linking them to canonical RL settings, particularly Minigrid environments. The paper defines stratified spaces as those equipped with a continuous mapping into a poset endowed with the Alexandrov topology, generalizing classical manifold theory to combinatorially rich, singular spaces.

A disk with an attached line is utilized to illustrate how face-posets can capture boundary relationships beyond dimension-indexed filtrations.

Figure 1: A disk with a line attached, stratified via the face-relation poset, which encodes detailed boundary relations refining a simple dimension ordering.

Stratification in Minigrid and State Spaces

The paper applies the stratification framework to Minigrid RL environments. States in a 3x3 grid can be mapped to cell complexes, where cells (2-cells, 1-edges, 0-vertices) induce a poset structure linked to policy-induced trees or more intricate posets, reflecting the potential of the policy and transition structure to stratify digital spaces.

Figure 2: Examples of the Minigrid environments used: empty grid and door-key variant.

Figure 3: Translating a $3 \times 3$ Minigrid into a 49-cell complex, visualized as a finite topologized space and mapped to a poset reflecting the policy tree.

This formalism extends to more complex tasks such as a “coin collection” game, demonstrating how event ordering and temporal constraints induce stratifications over trajectory spaces indexed by order complexes of event posets.

Figure 4: Stratification constructed for a space-time “coin collection” task, indicating policy-induced topological complexity across trajectory sets.

The theoretical results prove that for tasks with well-defined target sets, the trajectory space itself is naturally stratified by closed-set posets ordered by reverse inclusion, establishing a rigorous backbone for associating RL trajectory classes to specific strata (Theorem in Section 2.3).

Numerical Stratification Detection in Embedding Spaces

To operationalize stratification detection in high-dimensional data (especially neural embeddings), the authors introduce and employ the Volume Growth Transform (VGT) and its derivative VGT-dot. These quantify the scaling of neighborhood volumes around points, with the slope of the log-log curve corresponding to local dimension—a key diagnostic for detecting manifold versus singular/stratified structure.

A synthetic “room with a corridor” environment is used to benchmark intrinsic dimension estimators. While standard estimators struggle, the VGT-dot provides local-to-global clustering signatures that reveal both manifold regions and singular points.

Figure 5: A two-region environment (room and winding corridor) yielding non-uniform intrinsic dimensionality.

Figure 6: Local dimensionality estimation using several algorithms, with VGT-dot capturing dimension transitions across regions.

Quantitative analysis Figure 7 demonstrates that the VGT-dot signature accurately traces changes in scaling behavior, distinguishing generic 2D regions from 1D “pinch points” and correctly identifying singular junctions. HADES singularity detection [lim2025hades] is also benchmarked and shown to be more accurate but less scalable than the proposed VGT-dot, while dimension-induced clustering provides interpretable but less fine-grained stratification.

Figure 7: VGT curves and their derivatives for probe points—showcasing the diagnostic power for distinguishing strata and junctions.

Figure 8: HADES-based singularity detection and signature clustering with DIC and VGT-dot, reinforcing the correspondence between clustering patterns and stratification.

STL Semantics and Stratified Neighborhoods

The authors formalize the semantics of STL predicates as membership queries in maximal tubular neighborhoods about semi-algebraic strata within the task’s state space. This geometric connection is made rigorous through results on tubular neighborhoods for semi-algebraic sets (Lemma in Section 3.1), relating the robustness of STL formulas to structured stratification. This generates a novel, practical definition of atomic predicates (e.g., “close to $S$”) whose robustness score drives the reward function in RL settings.

Figure 9: Tubular neighborhoods around a closed interval, demonstrating volume growth laws key to stratification analysis.

The main theorem in this section demonstrates that the regions of satisfaction for these predicates inherit poset structures isomorphic to the stratification poset, and that membership in a specific stratum can be determined by suitable logical combinations of neighborhood predicates. This bridges symbolic reasoning and geometric stratification, opening new avenues for programmatically generating STL specifications aligned to task stratifications.

Empirical Analysis in Transformer-based DRL

The theoretical and numerical formalism is validated in transformer-based DRL agents trained on Minigrid variants with STL-robustness rewards. The agent observes raw pixel space; tasks employ “eventually in green” or “eventually in green and always not in red” STL objectives. Observations are encoded in high-dimensional token embeddings, learned via a Transformer-XL architecture trained by PPO.

Visualization and clustering of the 256D token embedding space, using UMAP and agglomerative clustering on VGT-dot features, reveal a consistent hourglass-type stratification. Distinct clusters correspond to task-relevant temporal and symbolic structure—for example, the activation period of an STL “eventually” operator is mapped to a topologically segregated cluster.

Figure 10: Example input observations: the temporal operator status is visually encoded and must be interpreted by the agent.

Figure 11: UMAP embedding of token vectors clustered by VGT-dot, capturing an hourglass geometry between temporal phases of the STL task.

Dimensionality reduction (DCT to 100D, UMAP to 3D) followed by HADES singularity detection confirms that non-manifold (singular) points are concentrated at policy-critical “neck” regions—mirroring theoretical predictions Figure 12. The correspondence between VGT-dot, DIC, and HADES supports the thesis of stratified structure imposed by the reward/formula geometry.

Figure 12: 3D UMAP after DCT reduction—points failing the manifold test are concentrated at the neck in line with the stratification hypothesis.

Analysis of the two-formula (“eventually” and “always not”) scenario delivers analogous stratification signatures, even in the absence of statistical non-manifold confirmation due to memory constraints Figure 13.

Figure 13: Left: UMAP and right: ISOMAP embeddings, clustered by VGT-dot; both demonstrate robustness of hourglass-type stratification under additional logical constraints.

Implications and Future Directions

The poset-based stratification framework introduced here provides a systematic embedding of STL semantics and RL dynamics within the apparatus of stratification theory and topological data analysis. Practically, it enables (i) new analyses of neural representations with respect to the underlying task geometry and (ii) principled synthesis of control specifications directly from task stratifications. The work demonstrates that task design via STL formulas directly enforces geometric structure in neural representations, facilitating robust policy generalization and explainability.

A key empirical result is the reproducible stratified structure—especially the hourglass signatures—in token embeddings trained solely on task/reward geometry, validated across clustering, local dimension, and singularity detection metrics.

The proposed VGT-dot is numerically efficient, scalable to high dimensions, and captures fine-grained local-to-global stratification, promising as a future diagnostic or architectural constraint.

Future research should address automated extraction of strata in more general high-dimensional and dynamic settings, construction of stratification-aware reward design for compositional tasks, and deeper study of the interaction between poset-indexed stratifications and policy optimality or sample efficiency in RL. There is also a clear path for integrating persistent homology and more advanced topological machine learning tools to further formalize the observed geometry.

Conclusion

The paper establishes a rigorous and computationally explicit bridge between stratification theory, STL semantics, and DRL policy learning. By formulating both state and neural embedding spaces as stratified or poset-indexed, and by leveraging robust numerical stratification detection, it sets a foundation for future work in interpretable, geometrically motivated, and logic-driven RL and AI system design (2604.04923).