---
title: 'Deflexformer: Decomposable Energy-Based Model'
url: https://www.emergentmind.com/topics/deflexformer
type: topic
---

# Deflexformer: Decomposable Energy-Based Model

Deflexformer is the neural network subsystem of the Deflex framework for automatic discovery of multiscale mathematical formulas in complex systems. In "Discovering Multiscale Deep Formulas in Complex Systems via Neural-Guided Lambda Calculus" [2606.07426], it is defined as a **decomposable deep energy-based model (EBM)** designed to learn unified representations of data and underlying rules across multiple scales, from element-wise or local structure to global or coarse-grained patterns. Within the overall Deflex pipeline, Deflexformer operates in synergy with Deflexpressor, a lambda-calculus symbolic regression subsystem: synthetic expressions and data generated by Deflexpressor are used to pre-train Deflexformer blocks, and the resulting intermediate representations then guide symbolic recovery of explicit formulas by decoupling multiscale latent relationships.

## 1. Definition and role within Deflex

Deflex consists of two subsystems named **Deflexformer** and **Deflexpressor** [2606.07426]. Deflexpressor is a lambda-calculus symbolic regression model for higher-order formulas, whereas Deflexformer is a decomposable deep energy model for learning unified representations across scales. The division of labor is explicit: Deflexformer is responsible for representation learning and decomposition of complex multiscale structure, while Deflexpressor performs symbolic regression over those learned representations to recover interpretable, executable formulas.

The high-level workflow is specified as follows. First, Deflexpressor generates synthetic mathematical expressions and samples corresponding data. Second, the synthetic dataset is used to pre-train a Deflexformer block. Third, Deflexformer, composed of multiple repeated or pipelined blocks, is further trained on real observation data from the target complex system. Fourth, the outputs of all Deflexformer blocks on both real and additional synthetic samples are collected. Finally, Deflexpressor performs symbolic regression on these representations to recover explicit formulas for each decomposed relationship [2606.07426].

In this construction, Deflexformer is not merely a generic feature extractor. Its blocks are intended to correspond to elementary mathematical transformations, so that symbolic regression need not fit the entire system-level mapping in a single step. This decomposition is central to the tractability claim made for the framework.

## 2. Energy-based formulation and unified representation

Deflexformer is formulated as a parameterized energy function model [2606.07426]. The governing idea is that all formulas, whether deterministic or stochastic and whether expressed as equations or distributions, are viewed as probability distributions over system states. In this framing, equations or invariants become sharply peaked distributions, whereas stochastic or distributional formulas correspond to broader probability distributions.

The probabilistic model is expressed through an energy function:
$$
p(\mathbf{x}) = \frac{1}{Z} \exp(-E(\mathbf{x})).
$$
Here, \(E(\mathbf{x})\) is the energy function and \(Z\) is the normalization constant, or partition function. Deflexformer approximates the optimal energy function for system patterns via a neural network \(\mathcal{E}_{\boldsymbol{\theta}}(\mathbf{x})\), parameterized by \(\boldsymbol{\theta}\), and learned from data [2606.07426].

The training objective is maximum log-likelihood of observed data under the model. The paper states the negative log-likelihood objective as
$$
L_{\boldsymbol{\theta}} = \frac{1}{N} \sum_{i=1}^N \left( \mathcal{E}_{\boldsymbol{\theta}}(\mathbf{x}_i) + \log Z_{\boldsymbol{\theta}} \right).
$$
Because \(\log Z_{\boldsymbol{\theta}}\) is intractable to compute directly, its estimation is performed via MCMC using Langevin dynamics, thereby circumventing explicit normalization [2606.07426].

This formulation is significant because it places invariants and distributions inside a single representational scheme. A plausible implication is that multiscale scientific laws can be treated uniformly even when some scales are best described by deterministic relations and others by distributional structure.

## 3. Architectural organization

Deflexformer is described as a **decomposable Transformer-like neural network** built from an input layer, stacked blocks, and an output layer [2606.07426]. Raw data, including element-level states and possible global features, is first mapped to high-dimensional embeddings using multiscale Fourier embeddings.

The core network consists of multiple sequential, identical blocks, each designed to be decomposable and to represent a base mathematical transformation. Within each block there are two sub-modules. The first is a point-wise feedforward network, implemented as a layered fully connected ReLU network, which captures element-specific or local evolution. The second is a multi-head cross-attention network that models inter-element and temporal interactions using self-attention. Spatial mixing applies attention over the element dimension to model interactions among particles or entities, while temporal mixing applies causal attention over the time dimension to capture inter-frame relationships. Both submodules output updated representations with residual and normalization connections [2606.07426].

After each block, representations are split into element-level and global-level forms:
- \(P_i \in \mathbb{R}^{n \times t \times D}\)
- \(G_i \in \mathbb{R}^{c \times D}\)

These are then propagated and transformed through subsequent blocks. The output layer maps the final global representation back to a scalar energy value via an inverse embedding [2606.07426].

Two architectural properties are emphasized. First, the model is fully permutation invariant or equivariant, making it suitable for set and sequence data of arbitrary sizes. Second, multi-scale handling arises from the combination of attention mechanisms and aggregation at both element and global levels, which captures both local and global system behaviors and enables smooth transitions and unified representations across scales [2606.07426].

## 4. Decomposability and hierarchical symbolic regression

Deflexformer’s defining structural assumption is that a complex energy function can be composed as a sequence of simpler base functions:
$$
E = E^1 \circ E^2 \circ \ldots \circ E^k.
$$
Each Deflexformer block approximates one such \(E^i\), so symbolic regression can operate on lower-complexity subcomponents rather than on the full mapping at once [2606.07426].

After training, intermediate representations at each layer are accessible. Deflexpressor then performs symbolic regression to identify explicit formulas that map between input-output pairs for each block, or for combinations of blocks. The paper characterizes this as **hierarchical extraction** and further specifies that hierarchical symbolic regression is scheduled by a binary tree procedure: results for small block groups are used to seed larger, composed block arrays, improving tractability [2606.07426].

This decomposition has an interpretability function as well as a computational one. Each block’s output corresponds to an intermediate, potentially interpretable mathematical transformation in the overall learned pattern. The resulting pipeline therefore attempts to build complex rules inductively from simple, interpretable building blocks rather than by a single monolithic regression pass.

A common misconception would be to treat Deflexformer itself as the symbolic regression engine. The framework separates these roles: Deflexformer learns decomposable energy-based representations, while Deflexpressor performs the lambda-calculus symbolic regression over those representations [2606.07426].

## 5. Synthetic pretraining and integration with Deflexpressor

The interaction between Deflexformer and Deflexpressor is a tightly coupled pipeline. Deflexpressor first generates a large pool of mathematical expressions, including higher-order, mapping, and aggregation types, and samples synthetic data from these formulas, producing pairs \((x, y=f(x))\) [2606.07426].

A single Deflexformer block is then pre-trained as a function approximator on this synthetic dataset. The stated objective of this stage is to encode mathematical priors and base transformation capabilities into the block before exposure to real data. The pre-trained block is replicated or cascaded multiple times to construct the full Deflexformer, with the number of blocks determined by desired complexity or saturation. The full network is subsequently trained on observation data from the complex system using the EBM negative log-likelihood loss [2606.07426].

At inference time, observation data, and optionally synthetic or random samples, are passed through the trained Deflexformer. Intermediate representations from each block are collected, and Deflexpressor receives input-output pairs from each block or composite of blocks. It then performs hierarchical symbolic regression using lambda calculus with types, lists, and higher-order logic to recover interpretable formulas at each decomposition level [2606.07426].

The paper explicitly states that this approach decouples multiscale latent relationships: each block targets a distinct mathematical relationship at a specific scale or transformation layer. This suggests that the architecture is intended not only to improve optimization, but also to align representational stages with the scale structure of the target system.

## 6. Empirical scope and reported behavior

The reported empirical scope covers six representative complex systems with diverse behaviors, and the framework is stated to uncover both classic and undocumented formulas in settings including Navier–Stokes, Langevin, power laws, and Maxwell-Boltzmann, at both fine-grained and coarse-grained scales [2606.07426]. The paper further states that negative log-likelihood and Earth Mover’s Distance across scales are optimal or near-optimal.

On efficiency, the abstract reports that Deflex achieves **up to 7-fold higher efficiency than the state-of-the-art methods while enabling automated multiscale discovery** [2606.07426]. The detailed description also characterizes the method as orders of magnitude faster than prior symbolic regression methods for complex, high-dimensional rules due to pretraining and decomposability, while noting that baseline symbolic regression methods without this architecture fail at these tasks or require impracticable computation.

The stated range of applications includes fluid dynamics, particles, and collective animal and human movements. The framework is also described as handling scale transitions and renormalization extremely well [2606.07426]. Ablation studies reportedly show that pretraining Deflexformer blocks with synthetic data from Deflexpressor dramatically accelerates formula recovery and improves success rates, and that increasing block depth or embedding size improves performance up to a point of saturation.

These empirical claims position Deflexformer as the mechanism by which multiscale latent relationships are organized into learnable, decomposable representations. A plausible implication is that its contribution is strongest where symbolic recovery must bridge heterogeneous scales rather than fit a single-scale closed-form law.

## 7. Conceptual significance

Within Deflex, Deflexformer supplies a unified representational layer that connects energy-based learning, decomposable neural architectures, and hierarchical symbolic regression [2606.07426]. Its key conceptual contribution is the treatment of equations, invariants, and distributions as instances of a common energy-based formalism, combined with a blockwise decomposition intended to make symbolic recovery tractable.

The model therefore occupies an intermediate position between black-box sequence or set modeling and explicit formula extraction. It is trained as an energy-based neural network on observation data, but its outputs are designed to be consumed by a symbolic system. The resulting division allows the framework to preserve expressive neural modeling while directing the learned structure toward interpretable sub-formulas.

The paper’s concluding characterization is that Deflexformer is a modular, energy-based deep neural architecture that learns energy-based representations unifying invariant and distributional cross-scale patterns, breaks down complex relationships into discrete, symbolic, human-interpretable sub-formulas, and synergizes with Deflexpressor through synthetic-data pretraining and decoupling of multiscale latent relationships [2606.07426]. Within that formulation, Deflexformer is the representation-learning component that makes automated multiscale discovery operational.

Source: https://www.emergentmind.com/topics/deflexformer