Spatiotemporal Perception Module

• The Spatiotemporal Perception Module is a framework that extracts and integrates spatial and temporal features, defining agency and perception in dynamic systems.
• It employs mathematical constructs like branch-morphs and counterfactual reasoning to measure non-determinacy and quantify autonomy.
• Its applications include cellular automata, sensor networks, and agent-based simulations, offering robust tools for modeling complex spatiotemporal dynamics.

A Spatiotemporal Perception Module is a fundamental architectural and conceptual construct in both biological and artificial intelligence for extracting, integrating, and reasoning over features distributed in both space and time. Such modules arise across diverse theoretical, algorithmic, and applied domains—from formal agent models in multivariate Markov chains, to neural and deep learning architectures for video, point cloud, event stream, and sensor network data. At its core, a spatiotemporal perception module provides mechanisms to represent, segregate, and aggregate information about entities, objects, or patterns as they evolve, interact, and act within spatial and temporal contexts.

1. Formalization and Theoretical Foundations

In formal agent modeling within multivariate Markov chains, entities are not treated as fixed stochastic processes but as elements of “entity-sets” of spatiotemporal patterns (STPs) (Biehl et al., 2017). An STP is defined as a specific value $x_A$ of the random variable $X_A$ over the index set $A \subset V$, where $V = J \times T$ spans spatial ($J$) and temporal ($T$) indices. The entity-set $\mathcal{E}(X_i)$ is a subset of all possible STPs, often chosen to correspond with operational closure, specific integration, or other criteria that delineate agent-like structures within the system—e.g., gliders in cellular automata.

A central theoretical construct is the definition of actions and perceptions for these entities within a Markov chain. Despite the system’s dynamics being fully determined by its transition matrix $p_{j,t}(x_{j,t} \mid x_{\text{pa}(j,t)})$, agency emerges by identifying, for a given STP, the existence of counterfactual alternatives (co-action entities) with identical environmental context but divergent future states. Perception, analogously, arises from counterfactual alternatives whose histories and environmental conditions diverge only at the partition imposed by the agent’s sensory resolution.

The standard perception-action loop (PA-loop) is encapsulated as a special case, with the agent process $M_t$ and environment process $E_t$, and equivalence classes on $E_t$ defined by:

$$\hat{e}_t \equiv_{\varepsilon_t} \bar{e}_t \iff \forall m_t, m_{t+1}: p_{M_{t+1}}(m_{t+1} \mid m_t, \hat{e}_t) = p_{M_{t+1}}(m_{t+1} \mid m_t, \bar{e}_t).$$

Entity-based actions are identified via the existence of “co-action” partners that share identical context at $t$ but yield diverging states at $t+1$; non-determinacy under fixed environment (non-heteronomy) is quantified by conditional entropy $H(M_{t+1} \mid E_t) > 0$.

2. Mechanistic and Mathematical Structure

The dynamics of spatiotemporal perception modules are governed by the underlying transition mechanisms. For Markovian systems, this is encoded in the conditional transition probabilities over spaces and times. The formalism introduces mathematical constructs such as branch-morphs, which partition the space of co-perception entities according to their immediate post-fork futures and define a conditioned branch distribution:

$$p_\eta(b \mid \hat{x}_{V_t \setminus A_t}, x_{A_{\leq t}}) = \frac{p(b \mid \hat{x}_{V_t \setminus A_t}, x_{A_{\leq t}})}{\sum_{c \in \eta(x_A, t)} p(c \mid \hat{x}_{V_t \setminus A_t}, x_{A_{\leq t}})}$$

where $b$ indexes equivalence classes in the branching structure of the co-perception entity set.

A non-interpenetration requirement is imposed to guarantee that mutually exclusive future realizations cannot coexist in a trajectory, ensuring rigorous probabilistic semantics for counterfactual branching in the presence of stochastic dynamics.

3. Conceptual Implications: Non-Heteronomy, Autonomy, and Perceptual Partitioning

A key implication is that spatiotemporally-defined actions guarantee a form of agency or partial autonomy: the entity’s future is not wholly determined by the environment, but depends on which counterfactual realization is “chosen” (as observed in the system’s evolution). This property, “non-heteronomy,” is a precise mathematical requirement for autonomy discussed in the literature (see Bertschinger et al., 2008). Formally, the residual conditional entropy under fixed environment reflects the agent’s action capacity.

On the perceptual side, the module’s capacity to “partition” environmental inputs is formalized via equivalence relations that generalize classical PA-loop notions. These partitions are physically meaningful in systems such as reaction-diffusion media or cellular automata, where STPs (such as gliders or spots) can be classified according to the environmental distinctions they resolve.

4. Applications and Instantiations in Artificial Systems

The entity-set perspective is instantiated in a variety of artificial systems:

• In cellular automata, entity-sets capturing gliders can be equipped with both an “action” definition (existence of divergent but compatible trajectories under equal environmental context) and a “perceptual” structure via equivalence partitions on environmental configurations.
• Analogous ideas are used in reaction-diffusion systems, particle-based models, or agent-based simulations, where individuation and pattern formation must be rigorously tied to the underlying mechanism rather than imposed externally.

This abstraction also enables a reinterpretation of notions from other work, such as Beer’s cognitive domain (for gliders), or the use of counterfactual trajectories to measure uncertainty in self-organizing systems, within a unified mathematical framework.

5. Mathematical Formulations in Comparative Context

The mathematical definitions provided in this framework generalize, and in specific restrictions (such as when the entity-set consists of entire trajectories of an agent memory process), reduce to familiar formulas from the PA-loop and classical agent-environment models: conditional probabilities, perception-induced equivalence relations, and entropy-based quantifications of autonomy all fall neatly under the broader spatiotemporal pattern perspective.

For example:

• When specialized to the PA-loop entity-set, entity actions correspond to value actions, and the branch-morph formalism reduces to the standard conditional probability $p_{M_{t+1}}(\cdot \mid e_t, m_t)$.
• The existence of counterfactual partners is essential for “action” in this sense, as only unpredictable variation (from the viewpoint of the environment alone) meets the agency criterion.
• Perception is a process of grouping histories and environments that are indistinguishable by the agent up to the current time, as subspaces of probabilistically-equivalent environmental influence on the next state.

6. Generalization, Comparison, and Robustness

Relative to alternative theories (such as those treating agents as stochastic processes, or as dynamically individuated domains), the formalism of spatiotemporal perception modules based on STPs offers:

• Direct rigorous formulation without recourse to ergodic or process-level constraints.
• A mechanism to tie autonomy and perception/action loop conceptualizations explicitly to underlying system dynamics.
• Robustness under model choice: the definitions are flexible in terms of the entity-set, supporting both phenomenological and operational closure criteria.

Alternative frameworks that forgo a strict branching structure or do not require non-interpenetration may lack precise semantics on the independence or mutual exclusivity of agent alternatives, which in this formalism are strictly enforced.

7. Summary and Research Significance

The spatiotemporal perception module formalism underlies a unified theoretical framework for defining agency, action, and perception in multivariate Markov chain systems by identifying entities with spatiotemporal patterns selected from an entity-set. Actions are characterized by the existence of counterfactual alternatives under fixed environment, formalized via the structure of the transition mechanism, and lead to measurable autonomy (residual entropy). Perceptions emerge from environment-induced equivalence partitions, mathematically grounded in branch-morph statistics. These formulations both generalize the classical perception-action loop and provide concrete mathematical tools for rigorous modeling of agent-like phenomena in dynamical and computational systems (Biehl et al., 2017).