---
title: Pre-structural Embeddings Overview
url: https://www.emergentmind.com/topics/pre-structural-embeddings
type: topic
---

# Pre-structural Embeddings Overview

Searching arXiv for recent papers and terminology around “pre-structural embeddings.”
“Pre-structural embeddings” is a cross-domain research term for representations extracted or constructed before a downstream structural decoding or task-specific reasoning stage, but after some prior source of signal has already been encoded. The phrase is used in several distinct senses in recent literature: as structure-aware tabular embeddings obtained by contextualizing frozen PLM representations with row-level graph structure in schema matching [2605.30729]; as intermediate latent states taken before structure decoding in biomolecular foundation models [2603.23583]; as deterministic, non-semantic token embeddings that encode only orthographic or glyph structure in language models [2507.04886]; and, more broadly, as explicit mappings from structured objects into vector spaces in representation learning for structured data [1905.06147]. Across these usages, a common theme is that embeddings are treated not as final semantic objects but as an interface layer in which structural priors, relational constraints, or latent geometric organization are introduced before downstream inference.

## 1. Terminological scope and conceptual variants

The term does not denote a single standardized formalism. In schema matching, “pre-structural embeddings” refers to semantic embeddings that have been contextualized by table structure after a PLM encodes text but before the embeddings are used for downstream matching [2605.30729]. In biomolecular modeling, “pre-structural embeddings” are the intermediate representations produced by the trunk of Boltz-2 just before the structure prediction head is applied, namely per-position single embeddings and pair embeddings extracted from the final trunk layer with 3 recycling steps [2603.23583]. In transformer language modeling, the term is naturally extended to frozen visual Unicode embeddings that are precomputed from rendered glyph structure, are non-trainable, and carry no learned semantics, forcing meaning to emerge in later layers [2507.04886].

A broader, survey-style sense appears in representation learning for structured data, where an embedding is any map $\phi : X \to \mathbb{R}^n$ from non-vectorial structured objects such as sequences, trees, and graphs into a vector space [1905.06147]. This suggests that “pre-structural” can function as an umbrella label for vector representations that preserve or expose structure prior to the final predictive stage. A plausible implication is that the phrase is best understood as a family resemblance concept rather than a settled taxonomy.

The literature also distinguishes adjacent but non-identical ideas. Some works treat pre-trained text-only embeddings as “pre-structural” in the sense that they are ungrounded and must be reshaped by external constraints, such as robot sensory-motor experience [2104.08521]. Others describe sentence embeddings as already containing overlapping structural layers that can be separated by probing architectures, even when the base model was not explicitly trained to encode structure in the sentence-level vector [2406.16563]. These usages share the intuition that structure can be latent, added, or extracted prior to downstream decision-making.

## 2. Structure-aware contextualization in tabular data

A concrete instantiation of the concept is given by SemStruct for schema matching, where a table is represented as a heterogeneous graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ with column nodes, row nodes, and value nodes [2605.30729]. Structural information is defined there as row-level co-occurrence, relational context between columns and values, and table topology. The method addresses a specific limitation of PLM-based table encoders: serialization into flat text discards the explicit signal that “this value in column A co-occurs with that value in column B in row $r$,” so models largely treat column content as an independent bag of tokens rather than as relational structure.

SemStruct initializes column and value nodes with a frozen sentence encoder, E5-small by default, and then applies a GNN over the heterogeneous graph to propagate context. The graph includes bidirectional column–value and row–value edges, numerical values are discretized via quantile-based binning with $k = 8$, and value nodes whose PLM embeddings have cosine similarity above $\tau = 0.9$ are merged to reduce node count [2605.30729]. The default structural encoder is a 3-layer GraphSAGE with hidden dimension 384, and final column embeddings are formed through a residual combination of the original PLM embedding and an MLP projection of the GNN output.

In this formulation, the resulting column vector is a structure-aware column embedding: a semantic prior refined by explicit structural message passing before downstream matching. The paper reports that on Valentine and SOTAB-SM, SemStruct improves MRR and Recall@GT over frozen PLM baselines and evaluable fully fine-tuned baselines, with particularly strong gains on semantically-joinable cases and on SOTAB-SM where headers are deleted or replaced by numbers [2605.30729]. An important ablation result is that row nodes function primarily as topological conduits rather than semantic entities, with zero-initialized row nodes often performing best. This supports a specific interpretation of pre-structural embeddings: what matters is not that every intermediate node carry rich semantics, but that the graph connectivity induces the right contextual transformation before task-specific comparison.

The same paper frames this design as a general pattern: start with frozen PLM semantics, build an explicit structural graph, run a relational encoder such as a GNN, and fuse semantic and structural information, often via residuals [2605.30729]. This suggests a reusable recipe for table-centric tasks beyond schema matching, although such extensions are presented there as implications rather than evaluated claims.

## 3. Intermediate latent states before structure decoding in biomolecular modeling

ZeroFold uses the term in a more literal architectural sense. Here, pre-structural embeddings are intermediate representations taken from Boltz-2 immediately before the structure module converts them into 3D coordinates [2603.23583]. For each protein or RNA chain, ZeroFold extracts a per-position single representation $s_i^{\text{msa}} \in \mathbb{R}^{384}$ and a pair representation $z_{ij}^{\text{pair}} \in \mathbb{R}^{128}$ from the final trunk layer. These representations are then processed by chain-specific encoders and a cross-modal attention module to predict protein–RNA binding affinity directly from sequence.

The motivation is tied to RNA flexibility. The paper states that RNA often behaves like intrinsically disordered proteins, so a single three-dimensional conformation cannot adequately represent the bound state [2603.23583]. Standard structure-based methods commit to one predicted or experimental structure and thereby discard ensemble information relevant to binding. ZeroFold instead uses the latent states before structure decoding, arguing that these embeddings implicitly encode conformational ensemble information without requiring predicted structures.

Empirically, ZeroFold is trained on PRADB, a curated dataset of 2,621 unique protein–RNA pairs with experimentally measured affinities drawn from four complementary databases, and evaluated under a held-out split with 40% sequence identity thresholds for both protein and RNA [2603.23583]. On the held-out test set it reports MAE $= 1.14$, RMSE $= 1.47$, PCC $= 0.63$, and SCC $= 0.65$, while the empirical noise ceiling estimated from replicate measurements is approximately Pearson $r \approx 0.664$ and Spearman $\approx 0.616$ [2603.23583]. The paper also reports inference time of about 18 ms per pair for ZeroFold versus about 17.1 s for a full Boltz-2 structure plus downstream affinity pipeline.

In this context, pre-structural embeddings are neither handcrafted structural features nor post hoc graph refinements of textual embeddings. They are latent states from a structure-prediction model captured before collapse into a single structure. A plausible implication is that the term can denote a “before-decoding” representation that preserves uncertainty and interaction-relevant geometry better than final coordinates, especially for conformationally heterogeneous systems.

## 4. Structure as primitive form rather than semantics

A different but related use appears in work on transformer language models with frozen visual Unicode representations. There, token embeddings are entirely frozen, constructed deterministically from the rendered visual structure of Unicode glyphs, and never updated during training [2507.04886]. For each token, glyph bitmaps are rendered, resized to fixed resolution, flattened, projected by PCA, L2-normalized, and used as the embedding vector in a decoder-only Transformer with $d_{\text{model}} = 1024$.

These embeddings are explicitly described as structural rather than semantic. They encode orthographic and visual properties such as length, script, punctuation shape, and character composition, but not learned semantic similarity [2507.04886]. The central claim is that high-level semantics are not inherent to input embeddings but emerge from the Transformer’s compositional architecture and data scale. In this framing, embeddings cease to be “meaning vectors” and become structural primitives.

The reported evidence includes stable convergence with frozen visual embeddings, nearly identical loss curves between frozen and trainable-embedding variants in 0.5B models, and stronger MMLU performance for frozen models. For EN+RU 0.5B models, `best_bvv_ru` with frozen embeddings achieves MMLU $= 22.29$, whereas `best_bvv_unfrozen_ru` with trainable embeddings achieves MMLU $= 11.37$ [2507.04886]. The paper attributes this to “representational interference,” namely the burden placed on a trainable embedding layer when it must jointly encode low-level structural and high-level semantic information.

Under this interpretation, pre-structural embeddings are not “partially semantic” at all. They are deliberately non-semantic, fixed, and precomputed, with semantics delegated to later layers. This usage broadens the concept substantially: the “pre-” in pre-structural can refer not merely to temporal order before a downstream task, but to a separation-of-concerns principle in model architecture.

## 5. Grounding, probing, and transformation of latent structure

Several additional papers clarify how pre-structural embeddings can be transformed or dissected once a base representation already exists. In robotics, pre-trained 300-dimensional Word2Vec vectors are treated as purely linguistic and ungrounded representations learned under the distributional hypothesis [2104.08521]. The rPRAE model inserts a three-layer tanh retrofit layer before a bidirectional action–language recurrent autoencoder, and trains this layer alternately with the rest of the model so that pre-trained word vectors are transformed into embodied ones aligned with robot sensory-motor experience. The resulting grounded embeddings cluster synonyms according to shared action and environment usage, separate antonyms such as “slowly” and “fast,” and support action generation from unseen words [2104.08521].

In sentence representation analysis, raw ELECTRA sentence embeddings from the final $\langle s \rangle$ token are treated as containing overlapping layers of information that can be separated [2406.16563]. A VAE-like architecture with a 5-dimensional latent bottleneck and a max-margin objective is used to recover chunk structure, grammatical number, and semantic roles from raw sentence embeddings. On a controlled sentence-structure task, the resulting system reaches F1 scores of $0.9992$ for French and $0.997$ for English [2406.16563]. The paper interprets this as evidence that sentence embeddings are not monolithic holistic vectors but contain identifiable structural parts, even without explicit structural supervision at pretraining time.

At the architectural level, transformer contextual embeddings have also been decomposed into a sum of vector factors: an input-related term, a cumulative multi-head attention contribution, a cumulative feed-forward contribution, and a bias/layer-norm term [2206.03529]. The analysis shows that the bias term lies in at most a $2\Lambda$-dimensional subspace, which for BERT-base corresponds to 48 dimensions out of 768, and that feed-forward components dominate masked language modeling while attention components are more useful for tasks such as word sense disambiguation and named entity recognition [2206.03529]. This line of work does not use the exact phrase “pre-structural embeddings,” but it reinforces the view that embedding spaces possess internal structure that can be analyzed before downstream prediction.

Together, these studies show that pre-structural embeddings may be added by retrofitting, extracted by probes, or algebraically decomposed into subcomponents. This suggests that the concept spans both construction and analysis.

## 6. General frameworks, graph structure, and interpretable structural feature maps

A general formal background is provided by the survey on representation learning for structured data, which defines an embedding simply as a map $\phi : X \to \mathbb{R}^n$ from structured objects into a vector space [1905.06147]. The survey organizes methods into kernel or feature-map approaches, distance-based methods with dimensionality reduction, and neural encoders such as RNNs, recursive neural networks, and GCNs. In all three families, a structured object is turned into a vector through explicit substructure counts, structural distances, or message passing. This is the broadest sense in which “pre-structural embeddings” can be understood.

A more explicit graph-theoretic instantiation is given by structural node embeddings based on homomorphism counts [2308.15283]. For a family of rooted pattern graphs $K$ and a graph $G$, the embedding of node $v$ is defined as
\[
emb^G_K(v) \coloneqq (Hom(H, r, G, v))_{(H,r)\in K},
\]
that is, each coordinate counts rooted homomorphisms from a pattern graph into $G$ with the root mapped to $v$ [2308.15283]. These embeddings are isomorphism-invariant, interpretable, and theoretically connected to 1-WL, message-passing GNNs, and spectral structure through trees and cycles. They are also explicitly presented as features for downstream models such as Random Forests and SVMs rather than as learned end-to-end neural states.

A complementary perspective comes from “Structure Inducing Pre-Training,” which formalizes pre-training around a pre-training graph $G_{\mathrm{PT}}$ over samples and a structure-inducing loss $SI$ that ensures graph recoverability from the learned per-sample latent space [2103.10334]. The full objective is
\[
\mathcal L_{\mathrm{PT}} = (1 - \lambda_{\mathrm{SI}}) M + \lambda_{\mathrm{SI}} SI,
\]
where $M$ is an intra-sample objective and $SI$ is a structure-inducing objective, such as contrastive or multi-similarity metric learning over graph edges [2103.10334]. The theoretical analysis shows that nearest-neighbor accuracy for a downstream task is lower-bounded by the local consistency of that task on the pre-training graph, and it distinguishes explicit versus implicit and deep versus shallow structural constraints. This provides a general account of how pre-structural embeddings can be engineered through relational inductive bias.

Finally, an unsupervised framework for evaluating structural node embeddings defines a quality score $\psi = 1 - r^2$, where $r$ is the Pearson correlation between pairwise distances in feature space and weighted pairwise distances in embedding space [2306.10770]. Although this is an evaluation framework rather than an embedding method, it operationalizes the claim that structural embeddings should at least reconstruct selected structural node features such as degree, centrality, or clustering. This evaluation perspective is compatible with the broader idea that pre-structural embeddings are useful insofar as their geometry preserves task-relevant structure before any supervised downstream model is trained.

A common misconception is that such embeddings must always be learned end-to-end or must always be semantic. The cited literature shows otherwise: they may be deterministic and frozen [2507.04886], graph-count-based and non-neural [2308.15283], latent trunk states before decoding [2603.23583], or semantic vectors structurally refined by a lightweight encoder [2605.30729].

## 7. Synthesis and research significance

Across domains, pre-structural embeddings serve as an intermediate representational layer that changes what downstream learning must do. In tabular data, they reduce ambiguity left unresolved by column headers or flat serialization by injecting row-level relational context [2605.30729]. In biomolecular modeling, they preserve ensemble-aware information that would otherwise be collapsed by commitment to a single structure [2603.23583]. In language modeling, they can encode only form and let semantics emerge later, challenging the assumption that meaning must reside in the embedding matrix itself [2507.04886]. In robotics and probing studies, they reveal that pretrained vectors can be grounded or decomposed into more structured subspaces without discarding their original usefulness [2104.08521; 2406.16563].

This suggests that the main scientific function of pre-structural embeddings is architectural mediation. They locate structure neither wholly in raw input symbols nor wholly in the final prediction head, but in an intermediate space where geometry, topology, and prior relational constraints can be made explicit. A plausible implication is that the term will continue to be used in multiple domain-specific ways rather than converging on a single universal definition. Even so, the unifying principle is stable: pre-structural embeddings are representations intentionally positioned before downstream inference and organized so that structure is available as a first-class property of the embedding space itself.

Source: https://www.emergentmind.com/topics/pre-structural-embeddings