Pre-structural Embeddings Overview

• Pre-structural embeddings are intermediate representations extracted before downstream decoding, integrating encoded signals with structural priors.
• They are applied across domains such as schema matching, biomolecular modeling, and transformer LM design to refine semantics with explicit structural context.
• Techniques often employ frozen embeddings, graph neural networks, or retrofit layers to combine latent geometric organization with task-specific inference.

Searching arXiv for papers and terminology around “pre-structural embeddings.” “Pre-structural embeddings” is a cross-domain research term for representations extracted or constructed before a downstream structural decoding or task-specific reasoning stage, but after some prior source of signal has already been encoded. The phrase is used in several distinct senses in recent literature: as structure-aware tabular embeddings obtained by contextualizing frozen PLM representations with row-level graph structure in schema matching (Kang et al., 29 May 2026); as intermediate latent states taken before structure decoding in biomolecular foundation models (Hanke et al., 24 Mar 2026); as deterministic, non-semantic token embeddings that encode only orthographic or glyph structure in LLMs (Bochkov, 7 Jul 2025); and, more broadly, as explicit mappings from structured objects into vector spaces in representation learning for structured data (Paaßen et al., 2019). Across these usages, a common theme is that embeddings are treated not as final semantic objects but as an interface layer in which structural priors, relational constraints, or latent geometric organization are introduced before downstream inference.

1. Terminological scope and conceptual variants

The term does not denote a single standardized formalism. In schema matching, “pre-structural embeddings” refers to semantic embeddings that have been contextualized by table structure after a PLM encodes text but before the embeddings are used for downstream matching (Kang et al., 29 May 2026). In biomolecular modeling, “pre-structural embeddings” are the intermediate representations produced by the trunk of Boltz-2 just before the structure prediction head is applied, namely per-position single embeddings and pair embeddings extracted from the final trunk layer with 3 recycling steps (Hanke et al., 24 Mar 2026). In transformer language modeling, the term is naturally extended to frozen visual Unicode embeddings that are precomputed from rendered glyph structure, are non-trainable, and carry no learned semantics, forcing meaning to emerge in later layers (Bochkov, 7 Jul 2025).

A broader, survey-style sense appears in representation learning for structured data, where an embedding is any map $\phi : X \to \mathbb{R}^n$ from non-vectorial structured objects such as sequences, trees, and graphs into a vector space (Paaßen et al., 2019). This suggests that “pre-structural” can function as an umbrella label for vector representations that preserve or expose structure prior to the final predictive stage. A plausible implication is that the phrase is best understood as a family resemblance concept rather than a settled taxonomy.

The literature also distinguishes adjacent but non-identical ideas. Some works treat pre-trained text-only embeddings as “pre-structural” in the sense that they are ungrounded and must be reshaped by external constraints, such as robot sensory-motor experience (Toyoda et al., 2021). Others describe sentence embeddings as already containing overlapping structural layers that can be separated by probing architectures, even when the base model was not explicitly trained to encode structure in the sentence-level vector (Nastase et al., 2024). These usages share the intuition that structure can be latent, added, or extracted prior to downstream decision-making.

2. Structure-aware contextualization in tabular data

A concrete instantiation of the concept is given by SemStruct for schema matching, where a table is represented as a heterogeneous graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ with column nodes, row nodes, and value nodes (Kang et al., 29 May 2026). Structural information is defined there as row-level co-occurrence, relational context between columns and values, and table topology. The method addresses a specific limitation of PLM-based table encoders: serialization into flat text discards the explicit signal that “this value in column A co-occurs with that value in column B in row $r$,” so models largely treat column content as an independent bag of tokens rather than as relational structure.

SemStruct initializes column and value nodes with a frozen sentence encoder, E5-small by default, and then applies a GNN over the heterogeneous graph to propagate context. The graph includes bidirectional column–value and row–value edges, numerical values are discretized via quantile-based binning with $k = 8$, and value nodes whose PLM embeddings have cosine similarity above $\tau = 0.9$ are merged to reduce node count (Kang et al., 29 May 2026). The default structural encoder is a 3-layer GraphSAGE with hidden dimension 384, and final column embeddings are formed through a residual combination of the original PLM embedding and an MLP projection of the GNN output.

In this formulation, the resulting column vector is a structure-aware column embedding: a semantic prior refined by explicit structural message passing before downstream matching. The paper reports that on Valentine and SOTAB-SM, SemStruct improves MRR and Recall@GT over frozen PLM baselines and evaluable fully fine-tuned baselines, with particularly strong gains on semantically-joinable cases and on SOTAB-SM where headers are deleted or replaced by numbers (Kang et al., 29 May 2026). An important ablation result is that row nodes function primarily as topological conduits rather than semantic entities, with zero-initialized row nodes often performing best. This supports a specific interpretation of pre-structural embeddings: what matters is not that every intermediate node carry rich semantics, but that the graph connectivity induces the right contextual transformation before task-specific comparison.

The same paper frames this design as a general pattern: start with frozen PLM semantics, build an explicit structural graph, run a relational encoder such as a GNN, and fuse semantic and structural information, often via residuals (Kang et al., 29 May 2026). This suggests a reusable recipe for table-centric tasks beyond schema matching, although such extensions are presented there as implications rather than evaluated claims.

3. Intermediate latent states before structure decoding in biomolecular modeling

ZeroFold uses the term in a more literal architectural sense. Here, pre-structural embeddings are intermediate representations taken from Boltz-2 immediately before the structure module converts them into 3D coordinates (Hanke et al., 24 Mar 2026). For each protein or RNA chain, ZeroFold extracts a per-position single representation $s_i^{\text{msa}} \in \mathbb{R}^{384}$ and a pair representation $z_{ij}^{\text{pair}} \in \mathbb{R}^{128}$ from the final trunk layer. These representations are then processed by chain-specific encoders and a cross-modal attention module to predict protein–RNA binding affinity directly from sequence.

The motivation is tied to RNA flexibility. The paper states that RNA often behaves like intrinsically disordered proteins, so a single three-dimensional conformation cannot adequately represent the bound state (Hanke et al., 24 Mar 2026). Standard structure-based methods commit to one predicted or experimental structure and thereby discard ensemble information relevant to binding. ZeroFold instead uses the latent states before structure decoding, arguing that these embeddings implicitly encode conformational ensemble information without requiring predicted structures.

Empirically, ZeroFold is trained on PRADB, a curated dataset of 2,621 unique protein–RNA pairs with experimentally measured affinities drawn from four complementary databases, and evaluated under a held-out split with 40% sequence identity thresholds for both protein and RNA (Hanke et al., 24 Mar 2026). On the held-out test set it reports MAE $= 1.14$, RMSE $= 1.47$, PCC $= 0.63$, and SCC $\mathcal{G} = (\mathcal{V}, \mathcal{E})$0, while the empirical noise ceiling estimated from replicate measurements is approximately Pearson $\mathcal{G} = (\mathcal{V}, \mathcal{E})$1 and Spearman $\mathcal{G} = (\mathcal{V}, \mathcal{E})$2 (Hanke et al., 24 Mar 2026). The paper also reports inference time of about 18 ms per pair for ZeroFold versus about 17.1 s for a full Boltz-2 structure plus downstream affinity pipeline.

In this context, pre-structural embeddings are neither handcrafted structural features nor post hoc graph refinements of textual embeddings. They are latent states from a structure-prediction model captured before collapse into a single structure. A plausible implication is that the term can denote a “before-decoding” representation that preserves uncertainty and interaction-relevant geometry better than final coordinates, especially for conformationally heterogeneous systems.

4. Structure as primitive form rather than semantics

A different but related use appears in work on transformer LLMs with frozen visual Unicode representations. There, token embeddings are entirely frozen, constructed deterministically from the rendered visual structure of Unicode glyphs, and never updated during training (Bochkov, 7 Jul 2025). For each token, glyph bitmaps are rendered, resized to fixed resolution, flattened, projected by PCA, L2-normalized, and used as the embedding vector in a decoder-only Transformer with $\mathcal{G} = (\mathcal{V}, \mathcal{E})$3.

These embeddings are explicitly described as structural rather than semantic. They encode orthographic and visual properties such as length, script, punctuation shape, and character composition, but not learned semantic similarity (Bochkov, 7 Jul 2025). The central claim is that high-level semantics are not inherent to input embeddings but emerge from the Transformer’s compositional architecture and data scale. In this framing, embeddings cease to be “meaning vectors” and become structural primitives.

The reported evidence includes stable convergence with frozen visual embeddings, nearly identical loss curves between frozen and trainable-embedding variants in 0.5B models, and stronger MMLU performance for frozen models. For EN+RU 0.5B models, best_bvv_ru with frozen embeddings achieves MMLU $\mathcal{G} = (\mathcal{V}, \mathcal{E})$4, whereas best_bvv_unfrozen_ru with trainable embeddings achieves MMLU $\mathcal{G} = (\mathcal{V}, \mathcal{E})$5 (Bochkov, 7 Jul 2025). The paper attributes this to “representational interference,” namely the burden placed on a trainable embedding layer when it must jointly encode low-level structural and high-level semantic information.

Under this interpretation, pre-structural embeddings are not “partially semantic” at all. They are deliberately non-semantic, fixed, and precomputed, with semantics delegated to later layers. This usage broadens the concept substantially: the “pre-” in pre-structural can refer not merely to temporal order before a downstream task, but to a separation-of-concerns principle in model architecture.

5. Grounding, probing, and transformation of latent structure

Several additional papers clarify how pre-structural embeddings can be transformed or dissected once a base representation already exists. In robotics, pre-trained 300-dimensional Word2Vec vectors are treated as purely linguistic and ungrounded representations learned under the distributional hypothesis (Toyoda et al., 2021). The rPRAE model inserts a three-layer tanh retrofit layer before a bidirectional action–language recurrent autoencoder, and trains this layer alternately with the rest of the model so that pre-trained word vectors are transformed into embodied ones aligned with robot sensory-motor experience. The resulting grounded embeddings cluster synonyms according to shared action and environment usage, separate antonyms such as “slowly” and “fast,” and support action generation from unseen words (Toyoda et al., 2021).

In sentence representation analysis, raw ELECTRA sentence embeddings from the final $\mathcal{G} = (\mathcal{V}, \mathcal{E})$6 token are treated as containing overlapping layers of information that can be separated (Nastase et al., 2024). A VAE-like architecture with a 5-dimensional latent bottleneck and a max-margin objective is used to recover chunk structure, grammatical number, and semantic roles from raw sentence embeddings. On a controlled sentence-structure task, the resulting system reaches F1 scores of $\mathcal{G} = (\mathcal{V}, \mathcal{E})$7 for French and $\mathcal{G} = (\mathcal{V}, \mathcal{E})$8 for English (Nastase et al., 2024). The paper interprets this as evidence that sentence embeddings are not monolithic holistic vectors but contain identifiable structural parts, even without explicit structural supervision at pretraining time.

At the architectural level, transformer contextual embeddings have also been decomposed into a sum of vector factors: an input-related term, a cumulative multi-head attention contribution, a cumulative feed-forward contribution, and a bias/layer-norm term (Mickus et al., 2022). The analysis shows that the bias term lies in at most a $\mathcal{G} = (\mathcal{V}, \mathcal{E})$9-dimensional subspace, which for BERT-base corresponds to 48 dimensions out of 768, and that feed-forward components dominate masked language modeling while attention components are more useful for tasks such as word sense disambiguation and named entity recognition (Mickus et al., 2022). This line of work does not use the exact phrase “pre-structural embeddings,” but it reinforces the view that embedding spaces possess internal structure that can be analyzed before downstream prediction.

Together, these studies show that pre-structural embeddings may be added by retrofitting, extracted by probes, or algebraically decomposed into subcomponents. This suggests that the concept spans both construction and analysis.

6. General frameworks, graph structure, and interpretable structural feature maps

A general formal background is provided by the survey on representation learning for structured data, which defines an embedding simply as a map $r$0 from structured objects into a vector space (Paaßen et al., 2019). The survey organizes methods into kernel or feature-map approaches, distance-based methods with dimensionality reduction, and neural encoders such as RNNs, recursive neural networks, and GCNs. In all three families, a structured object is turned into a vector through explicit substructure counts, structural distances, or message passing. This is the broadest sense in which “pre-structural embeddings” can be understood.

A more explicit graph-theoretic instantiation is given by structural node embeddings based on homomorphism counts (Wolf et al., 2023). For a family of rooted pattern graphs $r$1 and a graph $r$2, the embedding of node $r$3 is defined as

$r$4

that is, each coordinate counts rooted homomorphisms from a pattern graph into $r$5 with the root mapped to $r$6 (Wolf et al., 2023). These embeddings are isomorphism-invariant, interpretable, and theoretically connected to 1-WL, message-passing GNNs, and spectral structure through trees and cycles. They are also explicitly presented as features for downstream models such as Random Forests and SVMs rather than as learned end-to-end neural states.

A complementary perspective comes from “Structure Inducing Pre-Training,” which formalizes pre-training around a pre-training graph $r$7 over samples and a structure-inducing loss $r$8 that ensures graph recoverability from the learned per-sample latent space (McDermott et al., 2021). The full objective is

$r$9

where $k = 8$0 is an intra-sample objective and $k = 8$1 is a structure-inducing objective, such as contrastive or multi-similarity metric learning over graph edges (McDermott et al., 2021). The theoretical analysis shows that nearest-neighbor accuracy for a downstream task is lower-bounded by the local consistency of that task on the pre-training graph, and it distinguishes explicit versus implicit and deep versus shallow structural constraints. This provides a general account of how pre-structural embeddings can be engineered through relational inductive bias.

Finally, an unsupervised framework for evaluating structural node embeddings defines a quality score $k = 8$2, where $k = 8$3 is the Pearson correlation between pairwise distances in feature space and weighted pairwise distances in embedding space (Dehghan et al., 2023). Although this is an evaluation framework rather than an embedding method, it operationalizes the claim that structural embeddings should at least reconstruct selected structural node features such as degree, centrality, or clustering. This evaluation perspective is compatible with the broader idea that pre-structural embeddings are useful insofar as their geometry preserves task-relevant structure before any supervised downstream model is trained.

A common misconception is that such embeddings must always be learned end-to-end or must always be semantic. The cited literature shows otherwise: they may be deterministic and frozen (Bochkov, 7 Jul 2025), graph-count-based and non-neural (Wolf et al., 2023), latent trunk states before decoding (Hanke et al., 24 Mar 2026), or semantic vectors structurally refined by a lightweight encoder (Kang et al., 29 May 2026).

7. Synthesis and research significance

Across domains, pre-structural embeddings serve as an intermediate representational layer that changes what downstream learning must do. In tabular data, they reduce ambiguity left unresolved by column headers or flat serialization by injecting row-level relational context (Kang et al., 29 May 2026). In biomolecular modeling, they preserve ensemble-aware information that would otherwise be collapsed by commitment to a single structure (Hanke et al., 24 Mar 2026). In language modeling, they can encode only form and let semantics emerge later, challenging the assumption that meaning must reside in the embedding matrix itself (Bochkov, 7 Jul 2025). In robotics and probing studies, they reveal that pretrained vectors can be grounded or decomposed into more structured subspaces without discarding their original usefulness (Toyoda et al., 2021, Nastase et al., 2024).

This suggests that the main scientific function of pre-structural embeddings is architectural mediation. They locate structure neither wholly in raw input symbols nor wholly in the final prediction head, but in an intermediate space where geometry, topology, and prior relational constraints can be made explicit. A plausible implication is that the term will continue to be used in multiple domain-specific ways rather than converging on a single universal definition. Even so, the unifying principle is stable: pre-structural embeddings are representations intentionally positioned before downstream inference and organized so that structure is available as a first-class property of the embedding space itself.